From 9a87197e668bc9589ca55b90fa4792b8cb36f602 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:03:57 -0300 Subject: [PATCH 001/123] store: machine-wide content-addressed store of blobs and refs Blobs addressed by BLAKE2b over fixed 64 KiB pieces, put from bytes or files, received piece by piece with resume and range reads of partial blobs, refs per namespace, usage and sweep. Registered as a base engine package so the allocation lint applies; owning skill mirai-store. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-store/SKILL.md | 63 +++++++ README.md | 1 + cook/build/engine.odin | 2 +- docs/ARCHITECTURE.md | 1 + store/address.odin | 177 +++++++++++++++++ store/blobs.odin | 169 +++++++++++++++++ store/concurrent_test.odin | 85 +++++++++ store/hostile_test.odin | 144 ++++++++++++++ store/receive.odin | 282 ++++++++++++++++++++++++++++ store/refs.odin | 118 ++++++++++++ store/refs_test.odin | 39 ++++ store/space.odin | 91 +++++++++ store/store.odin | 165 ++++++++++++++++ store/store_test.odin | 211 +++++++++++++++++++++ tools/mirai/package.odin | 2 +- 15 files changed, 1548 insertions(+), 2 deletions(-) create mode 100644 .agents/skills/mirai-store/SKILL.md create mode 100644 store/address.odin create mode 100644 store/blobs.odin create mode 100644 store/concurrent_test.odin create mode 100644 store/hostile_test.odin create mode 100644 store/receive.odin create mode 100644 store/refs.odin create mode 100644 store/refs_test.odin create mode 100644 store/space.odin create mode 100644 store/store.odin create mode 100644 store/store_test.odin diff --git a/.agents/skills/mirai-store/SKILL.md b/.agents/skills/mirai-store/SKILL.md new file mode 100644 index 000000000..a01dc4abe --- /dev/null +++ b/.agents/skills/mirai-store/SKILL.md @@ -0,0 +1,63 @@ +--- +name: mirai-store +description: Mirai machine-wide content-addressed store (the `store` package) - blobs named by their + BLAKE2b address and cut in pieces, receiving a blob piece by piece with resume, range reads of + partial blobs, refs that name an address, usage and sweep, and the default root. Use when + storing or sharing heavy content between projects or peers, building a blob transport, an asset + library or a derived data cache on top of it, or diagnosing a corrupt or missing blob. +--- + +# Mirai store + +`store` (layer 1) imports only `base`, `core` and `std`: a plain library a game binary may link. +It knows two concepts, blobs and refs, and interprets neither: the transport, the library and the +derived data cache that use it decide what a ref means and what is live. + +## Map + +- Address, pieces and hex: `store/address.odin`. +- Root, layout version, errors and atomic writes: `store/store.odin`. +- Complete blobs (put, read, table, verify, remove): `store/blobs.odin`. +- Receiving a blob piece by piece and range reads of a partial one: `store/receive.odin`. +- Refs and their names: `store/refs.odin`. +- Usage and sweep: `store/space.odin`. + +## Workflow + +1. A sender serves the piece table of a complete blob and its pieces; it reads ranges by path or + with the range read. +2. A receiver begins from the address and the table, gives each piece as it arrives in any order, + and asks which pieces are missing to request them again, also after a restart. +3. A layer that names content sets a ref in its own namespace; it sweeps with the set of addresses + its refs and open work reach. + +## Invariants + +- The address is a pure function of the bytes: the length and the hash of every fixed-size piece, + one rule for every size. Piece hashes and addresses are keyed apart so none is taken for another. +- Only the two private procedures of `address.odin` (the value of a piece, the address of a table) + touch the hash primitive; everything else, tests included, goes through them or the API, so the + primitive can be replaced in one place. +- State lives in a `Store` value; several values and processes share one root without locks. + Every file appears whole through a rename from the root's tmp folder; a receiver writes only its + own piece and its own byte of the piece marks, so concurrent receivers of one blob agree. +- A table is checked against the address before anything is kept, a piece against the table + before it is written, and the whole blob against its address before it becomes complete; a + piece that fails then is marked missing again. +- A complete blob is a plain file of its raw bytes, never written in place: readers open it by + path. +- The root carries its layout version; a root of another layout is refused, never read. +- The store never decides what is live: sweep removes the complete blobs outside the set it is + given, and partial blobs only go through remove. +- Ref namespaces and names are lowercase ASCII, digits, `.`, `_` and `-`, so a ref is the same + file on every system. +- No input terminates the process: damaged or truncated files of the root, random tables and + pieces, out-of-range indices and memory that runs out return an `Error`. + +## Verification + +- `mirai check store` (it runs the allocation lint) and `mirai test store`. +- Focal tests: `address_is_the_same_through_put_bytes_put_file_and_receiving` (sizes around the + piece boundaries), `receiving_resumes_after_reopening_and_reads_the_pieces_it_has`, + `two_stores_on_one_root_put_and_receive_from_several_threads` and + `hostile_inputs_return_errors`. diff --git a/README.md b/README.md index 15e329c3a..9fbf6de8d 100644 --- a/README.md +++ b/README.md @@ -162,6 +162,7 @@ with a map from tasks to references. | Projects and build | [mirai-projects](.agents/skills/mirai-projects/SKILL.md) | | Standard library | [mirai-std](.agents/skills/mirai-std/SKILL.md) | | Platform and audio | [mirai-platform](.agents/skills/mirai-platform/SKILL.md) | +| Content-addressed store | [mirai-store](.agents/skills/mirai-store/SKILL.md) | The layers and contracts are in [ARCHITECTURE.md](docs/ARCHITECTURE.md) and the project rules in [AGENTS.md](AGENTS.md). diff --git a/cook/build/engine.odin b/cook/build/engine.odin index 5045ae857..31f71c658 100644 --- a/cook/build/engine.odin +++ b/cook/build/engine.odin @@ -79,7 +79,7 @@ ENGINE_PACKAGES :: [?]Engine_Package{ {.Base,"std/spatial","std"},{.Base,"std/jobs","std"},{.Base,"std/anim","std"},{.Base,"std/ai","std"},{.Base,"std/input","std"}, {.Base,"std/ui_format","std"}, {.Base,"audio","system"},{.Base,"platform","system"},{.Base,"rhi","system"}, - {.Base,"project","system"},{.Base,"core",""}, + {.Base,"project","system"},{.Base,"store","system"},{.Base,"core",""}, {.Engine,"asset",""},{.Engine,"net",""},{.Engine,"render/abi","render"},{.Engine,"render","render"},{.Engine,"third_party/freetype","freetype"}, {.Engine,"ui","ui"},{.Engine,"host",""},{.Engine,"engine/input","engine"},{.Engine,"engine/scene","engine"},{.Engine,"engine/volume","engine"}, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fcdb32964..683563cd6 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -330,6 +330,7 @@ Numbered layers; a package only imports lower layers. `std/ui_format` is what the structure of a `.ui` is judged by (node kinds, override fields, limits), which `ui` builds and `asset` checks without importing each other. - **1** `project`: `project.mirai` and `plugin.mirai` manifests, the layout of a project. +- **1** `store`: the machine-wide content-addressed store of blobs and refs under one root. - **2** `net`, `asset`, `ui`, `audio`, `rhi`: network transport, asset formats, UI, mixer and GPU; independent of each other; `asset` reads from `project` what a project's assets compress with. `rhi/shared_device` is the device the GPU tests share; only tests import it, also the GPU tests of diff --git a/store/address.odin b/store/address.odin new file mode 100644 index 000000000..b9ebff019 --- /dev/null +++ b/store/address.odin @@ -0,0 +1,177 @@ +package store + +import "core:crypto/blake2b" +import "core:os" + +// An address is a pure function of a blob's bytes. Only piece_value and +// address_value know the hash primitive, so it can be replaced in one place. +Hash :: distinct [32]u8 + +// A blob is cut in pieces of PIECE_BYTES, the last one shorter; an empty blob has none. +PIECE_BYTES :: 64<<10 + +// BLAKE2b-256 keyed apart for pieces and addresses, so a piece hash, a piece +// table and an address can never be taken for one another. +@(private) +KEY_PIECE :: "mirai store piece" +@(private) +KEY_ADDRESS :: "mirai store address" + +// Gives the piece hash at index of a table, called for every index in order. +@(private) +Piece_Source :: proc(user:rawptr,index:int) -> (Hash,Error) + +piece_count :: proc "contextless" (size:i64) -> int { + if size<=0 {return 0} + return int((size-1)/PIECE_BYTES+1) +} + +// The bytes of piece index of a blob of size; 0 for an index out of range. +piece_size :: proc "contextless" (size:i64,index:int) -> int { + if index<0 || index>=piece_count(size) {return 0} + return int(min(size-i64(index)*PIECE_BYTES,PIECE_BYTES)) +} + +address_of_bytes :: proc(data:[]u8) -> Hash { + data:=data + hash,_:=address_value(i64(len(data)),proc(user:rawptr,index:int) -> (Hash,Error) { + data:=(^[]u8)(user)^ + return piece_value(data[index*PIECE_BYTES:][:piece_size(i64(len(data)),index)],index),.None + },&data) + return hash +} + +// The address of the file at path, read piece by piece. +address_of_file :: proc(path:string) -> (hash:Hash,size:i64,error:Error) { + file,open_error:=os.open(path) + if open_error!=nil {return {},0,os_error(open_error)} + defer os.close(file) + file_size,size_error:=os.file_size(file) + if size_error!=nil {return {},0,os_error(size_error)} + hash,error=file_hash(file,file_size,nil,nil) + if error==.Corrupt {error=.Io} + return hash,file_size,error +} + +// Lowercase hexadecimal of hash, in buffer. +hex :: proc(hash:Hash,buffer:^[64]u8) -> string { + digits:="0123456789abcdef" + for b,i in hash { + buffer[2*i]=digits[b>>4] + buffer[2*i+1]=digits[b&15] + } + return string(buffer[:]) +} + +// The hash whose hex is text: exactly 64 lowercase hexadecimal digits. +parse_hex :: proc(text:string) -> (hash:Hash,ok:bool) { + if len(text)!=64 {return} + for i in 0..<32 { + high,high_ok:=hex_digit(text[2*i]) + low,low_ok:=hex_digit(text[2*i+1]) + if !high_ok || !low_ok {return {},false} + hash[i]=high<<4|low + } + return hash,true +} + +@(private) +hex_digit :: proc "contextless" (c:u8) -> (u8,bool) { + switch c { + case '0'..='9': return c-'0',true + case 'a'..='f': return c-'a'+10,true + } + return 0,false +} + +// The hash of the piece at index of a blob, from its bytes. +@(private) +piece_value :: proc(piece:[]u8,index:int) -> Hash { + ctx:blake2b.Context + key:=KEY_PIECE + blake2b.init_mac(&ctx,transmute([]u8)key,size_of(Hash)) + blake2b.update(&ctx,piece) + hash:Hash + blake2b.final(&ctx,hash[:]) + return hash +} + +// The address of a blob of size bytes, from its piece table, which pieces +// gives one entry at a time so a table is never held whole. +@(private) +address_value :: proc(size:i64,pieces:Piece_Source,user:rawptr) -> (Hash,Error) { + ctx:blake2b.Context + key:=KEY_ADDRESS + blake2b.init_mac(&ctx,transmute([]u8)key,size_of(Hash)) + length:[8]u8 + for i in 0..<8 {length[i]=u8(u64(size)>>(8*uint(i)))} + blake2b.update(&ctx,length[:]) + for index in 0.. Hash { + pieces:=pieces + hash,_:=address_value(size,proc(user:rawptr,index:int) -> (Hash,Error) { + return (^[]Hash)(user)[index],.None + },&pieces) + return hash +} + +// Reads size bytes of file piece by piece: their address. Each piece is also +// written to copy_to and its hash to table, when given. A file shorter than +// size, or longer, is Corrupt. +@(private) +file_hash :: proc(file:^os.File,size:i64,copy_to:^os.File,table:[]Hash) -> (hash:Hash,error:Error) { + if table!=nil && len(table)!=piece_count(size) {return {},.Invalid_Argument} + Reading :: struct { + buffer:[PIECE_BYTES]u8, + file,copy_to:^os.File, + size:i64, + table:[]Hash, + } + reading:=Reading{file=file,copy_to=copy_to,size=size,table=table} + hash=address_value(size,proc(user:rawptr,index:int) -> (Hash,Error) { + r:=(^Reading)(user) + offset:=i64(index)*PIECE_BYTES + piece:=r.buffer[:piece_size(r.size,index)] + if error:=read_exact(r.file,piece,offset);error!=.None {return {},error} + if r.copy_to!=nil { + if _,write_error:=os.write_at(r.copy_to,piece,offset);write_error!=nil {return {},os_error(write_error)} + } + hash:=piece_value(piece,index) + if r.table!=nil {r.table[index]=hash} + return hash,.None + },&reading) or_return + after:[1]u8 + if n,_:=os.read_at(file,after[:],size);n>0 {return {},.Corrupt} + return hash,.None +} + +// Fills buffer from file at offset; Corrupt when the file ends first. +@(private) +read_exact :: proc(file:^os.File,buffer:[]u8,offset:i64) -> Error { + done:=0 + for done=len(buffer) {break} + if n<=0 { + if read_error==nil || read_error==.EOF {return .Corrupt} + return os_error(read_error) + } + } + return .None +} diff --git a/store/blobs.odin b/store/blobs.odin new file mode 100644 index 000000000..61e5130c1 --- /dev/null +++ b/store/blobs.odin @@ -0,0 +1,169 @@ +package store + +import "base:runtime" +import "core:os" + +// Complete blobs are plain files of their bytes at blobs//
, +// both in hex, and are never written in place. + +put_bytes :: proc(s:^Store,data:[]u8) -> (hash:Hash,error:Error) { + hash=address_of_bytes(data) + if has(s,hash) {return hash,.None} + temp_buffer:[PATH_BYTES]u8 + temp,file:=temp_create(s.root,&temp_buffer) or_return + _,write_error:=os.write(file,data) + sync_error:=os.sync(file) + os.close(file) + if write_error!=nil || sync_error!=nil { + os.remove(temp) + return {},os_error(write_error) if write_error!=nil else .Io + } + return hash,blob_place(s,temp,hash) +} + +// Puts the file at path, read once piece by piece. +put_file :: proc(s:^Store,path:string) -> (hash:Hash,error:Error) { + source,open_error:=os.open(path) + if open_error!=nil {return {},os_error(open_error)} + defer os.close(source) + size,size_error:=os.file_size(source) + if size_error!=nil {return {},os_error(size_error)} + temp_buffer:[PATH_BYTES]u8 + temp,file:=temp_create(s.root,&temp_buffer) or_return + hash,error=file_hash(source,size,file,nil) + sync_error:=os.sync(file) if error==.None else nil + os.close(file) + if error==.Corrupt {error=.Io} + if error==.None && sync_error!=nil {error=.Io} + if error!=.None || has(s,hash) { + os.remove(temp) + return hash if error==.None else {},error + } + return hash,blob_place(s,temp,hash) +} + +// Whether the blob is complete here. +has :: proc(s:^Store,hash:Hash) -> bool { + buffer:[PATH_BYTES]u8 + path,error:=blob_file(s,hash,&buffer) + return error==.None && os.is_file(path) +} + +// The bytes of a complete blob, or of one being received. +blob_size :: proc(s:^Store,hash:Hash) -> (total:i64,error:Error) { + buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + file,open_error:=os.open(path) + if open_error==nil { + defer os.close(file) + size,size_error:=os.file_size(file) + return size,os_error(size_error) + } + if open_error!=.Not_Exist {return 0,os_error(open_error)} + p:=partial_open(s,hash,{}) or_return + defer partial_close(&p) + return p.size,.None +} + +// The path of a complete blob, for readers that open it themselves. +blob_path :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (cloned_path:string,error:Error) { + buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + if !os.is_file(path) {return "",.Missing} + cloned,clone_error:=make([]u8,len(path),allocator) + if clone_error!=nil {return "",.Out_Of_Memory} + copy(cloned,path) + return string(cloned),.None +} + +// Fills buffer with the bytes of the blob from offset: from a complete blob, +// or from the pieces a partial one has; Not_Here_Yet when one of the pieces +// of the range has not arrived. +read :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { + if offset<0 {return .Invalid_Argument} + for attempt in 0..<2 { + error:=read_complete(s,hash,offset,buffer) + if error!=.Missing {return error} + error=read_partial(s,hash,offset,buffer) + if error!=.Missing || attempt==1 {return error} + } + return .Missing +} + +// The piece hashes of a complete blob, checked against its address on the way. +blob_table :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (pieces:[]Hash,error:Error) { + buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + file,open_error:=os.open(path) + if open_error!=nil {return nil,os_error(open_error)} + defer os.close(file) + size,size_error:=os.file_size(file) + if size_error!=nil {return nil,os_error(size_error)} + make_error:runtime.Allocator_Error + pieces,make_error=make([]Hash,piece_count(size),allocator) + if make_error!=nil {return nil,.Out_Of_Memory} + address:Hash + address,error=file_hash(file,size,nil,pieces) + if error==.None && address!=hash {error=.Corrupt} + if error!=.None { + delete(pieces,allocator) + return nil,error + } + return pieces,.None +} + +// Corrupt when the file of a complete blob no longer has the bytes of its address. +verify :: proc(s:^Store,hash:Hash) -> Error { + buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + address,_,error:=address_of_file(path) + if error==.Io {return .Corrupt} + if error!=.None {return error} + return .None if address==hash else .Corrupt +} + +// Removes the blob, complete or partial. +remove :: proc(s:^Store,hash:Hash) -> Error { + buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + removed:=os.remove(path) + if removed!=nil && removed!=.Not_Exist {return os_error(removed)} + discarded:=partial_discard(s,hash) + if discarded!=.None && discarded!=.Missing {return discarded} + return .Missing if removed!=nil && discarded==.Missing else .None +} + +@(private) +blob_file :: proc(s:^Store,hash:Hash,buffer:^[PATH_BYTES]u8) -> (string,Error) { + name:[64]u8 + text:=hex(hash,&name) + return join(buffer,s.root,"blobs",text[:2],text) +} + +// Renames a finished file of tmp into the blob's place; a blob already there stays. +@(private) +blob_place :: proc(s:^Store,temp:string,hash:Hash) -> Error { + buffer,folder_buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + name:[64]u8 + text:=hex(hash,&name) + folder_make(join(&folder_buffer,s.root,"blobs",text[:2]) or_return) or_return + if os.is_file(path) || os.rename(temp,path)!=nil { + os.remove(temp) + return .None if os.is_file(path) else .Io + } + return .None +} + +@(private) +read_complete :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { + path_buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&path_buffer) or_return + file,open_error:=os.open(path) + if open_error!=nil {return os_error(open_error)} + defer os.close(file) + size,size_error:=os.file_size(file) + if size_error!=nil {return os_error(size_error)} + if offset>size || i64(len(buffer))>size-offset {return .Invalid_Argument} + return read_exact(file,buffer,offset) +} diff --git a/store/concurrent_test.odin b/store/concurrent_test.odin new file mode 100644 index 000000000..b22665c9d --- /dev/null +++ b/store/concurrent_test.odin @@ -0,0 +1,85 @@ +package store + +import "core:os" +import "core:strings" +import "core:testing" +import "core:thread" +import "mirai:std/base" + +@(private="file") +Worker :: struct { + s:^Store, + shared:[][]u8, + tables:[][]Hash, + own:[]u8, + own_table:[]Hash, + seed:int, + error:Error, +} + +@(private="file") +worker_receive :: proc(w:^Worker,data:[]u8,pieces:[]Hash) { + hash:=address_of_bytes(data) + if error:=receive_begin(w.s,hash,i64(len(data)),pieces);error!=.None {w.error=error;return} + for i in 0.. runtime.Allocator { + return {procedure=proc(data:rawptr,mode:runtime.Allocator_Mode,size,alignment:int,old_memory:rawptr,old_size:int,loc:=#caller_location) -> ([]byte,runtime.Allocator_Error) { + #partial switch mode { + case .Free,.Free_All: return nil,.None + } + return nil,.Out_Of_Memory + }} +} + +@(private="file") +path_in :: proc(folder:string,parts:..string) -> string { + all:=make([dynamic]string,context.temp_allocator) + append(&all,folder) + append(&all,..parts) + return strings.join(all[:],"/",context.temp_allocator) +} + +@(test) +hostile_inputs_return_errors :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-hostile");if folder=="" {return} + memory:base.Hostile_Memory + allocator:=base.hostile_memory_begin(&memory) + s,error:=open(folder,allocator) + if !testing.expect_value(t,error,Error.None) {return} + h:=base.Hostile{state=42} + data:=test_bytes(3*PIECE_BYTES+9,5) + hash:=address_of_bytes(data) + pieces:=test_table(data) + random:=make([]u8,PIECE_BYTES+3,context.temp_allocator) + for _ in 0..<64 { + size:=i64(base.hostile_int(&h,6*PIECE_BYTES))-PIECE_BYTES + table:=make([]Hash,base.hostile_int(&h,8),context.temp_allocator) + for &entry in table { + for &b in entry {b=u8(base.hostile_next(&h))} + } + target:Hash + for &b in target {b=u8(base.hostile_next(&h))} + e:=receive_begin(&s,target,size,table) + testing.expectf(t,e==.Corrupt || e==.Invalid_Argument,"a random table: %v",e) + } + testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),pieces),Error.None) + for _ in 0..<64 { + index:=base.hostile_int(&h,10)-3 + piece:=random[:base.hostile_int(&h,len(random))] + for &b in piece {b=u8(base.hostile_next(&h))} + e:=receive_piece(&s,hash,index,piece) + testing.expectf(t,e==.Corrupt || e==.Invalid_Argument,"a random piece %d: %v",index,e) + } + unknown:Hash + unknown[5]=9 + testing.expect_value(t,receive_piece(&s,unknown,0,data[:10]),Error.Missing) + testing.expect_value(t,read(&s,unknown,0,random[:1]),Error.Missing) + testing.expect_value(t,verify(&s,unknown),Error.Missing) + _,size_error:=blob_size(&s,unknown) + testing.expect_value(t,size_error,Error.Missing) + testing.expect_value(t,read(&s,hash,-1,random[:1]),Error.Invalid_Argument) + testing.expect_value(t,read(&s,hash,i64(len(data))+1,nil),Error.Invalid_Argument) + testing.expect_value(t,read(&s,hash,max(i64),random[:1]),Error.Invalid_Argument) + + // Files of the root damaged on disk. + stored,_:=put_bytes(&s,data[:PIECE_BYTES+5]) + path,_:=blob_path(&s,stored,context.temp_allocator) + damaged:=make([]u8,PIECE_BYTES+5,context.temp_allocator) + copy(damaged,data) + damaged[PIECE_BYTES+1]~=1 + Damage :: struct {bytes:[]u8} + for d in ([]Damage{{damaged},{damaged[:PIECE_BYTES]},{nil},{data[:PIECE_BYTES+6]}}) { + testing.expect(t,os.write_entire_file(path,d.bytes)==nil) + testing.expect_value(t,verify(&s,stored),Error.Corrupt) + _,table_error:=blob_table(&s,stored,context.temp_allocator) + testing.expect_value(t,table_error,Error.Corrupt) + } + testing.expect(t,os.write_entire_file(path_in(folder,"partial",hex_of(hash),"table"),[]u8{1,2,3})==nil) + testing.expect_value(t,receive_piece(&s,hash,0,test_piece(data,0)),Error.Corrupt) + _,missing_error:=receive_missing(&s,hash,context.temp_allocator) + testing.expect_value(t,missing_error,Error.Corrupt) + testing.expect_value(t,read(&s,hash,0,random[:1]),Error.Corrupt) + testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),pieces),Error.None) + _,missing_error=receive_missing(&s,hash,context.temp_allocator) + testing.expect_value(t,missing_error,Error.None) + + // Refs. + for name in ([]string{"","."," ..","..","a/b","a\\b","A","con","com1.txt","x.","a b",strings.repeat("x",NAME_MAX+1,context.temp_allocator)}) { + testing.expectf(t,ref_set(&s,"ns",name,hash)==.Invalid_Argument,"name %q",name) + testing.expectf(t,ref_set(&s,name,"n",hash)==.Invalid_Argument,"namespace %q",name) + _,get_error:=ref_get(&s,"ns",name) + testing.expect_value(t,get_error,Error.Invalid_Argument) + _,list_error:=ref_list(&s,name,context.temp_allocator) + testing.expect_value(t,list_error,Error.Invalid_Argument) + } + testing.expect_value(t,ref_set(&s,"ns","n",hash),Error.None) + testing.expect(t,os.write_entire_file(path_in(folder,"refs","ns","n"),transmute([]u8)string("not a hash"))==nil) + _,get_error:=ref_get(&s,"ns","n") + testing.expect_value(t,get_error,Error.Corrupt) + + // Memory that runs out. + out:=s + out.allocator=failing_allocator() + _,table_error:=blob_table(&s,address_of_bytes(data[:PIECE_BYTES]),out.allocator) + testing.expect_value(t,table_error,Error.Missing) + good,_:=put_bytes(&s,data[:10]) + _,table_error=blob_table(&s,good,out.allocator) + testing.expect_value(t,table_error,Error.Out_Of_Memory) + _,missing_error=receive_missing(&s,hash,out.allocator) + testing.expect_value(t,missing_error,Error.Out_Of_Memory) + _,path_error:=blob_path(&s,good,out.allocator) + testing.expect_value(t,path_error,Error.Out_Of_Memory) + _,list_error:=ref_list(&out,"ns",context.temp_allocator) + testing.expect_value(t,list_error,Error.Out_Of_Memory) + _,sweep_error:=sweep(&out,{good}) + testing.expect_value(t,sweep_error,Error.Out_Of_Memory) + other,_:=put_bytes(&s,data[:20]) + testing.expect_value(t,receive_begin(&out,address_of_bytes(data[:PIECE_BYTES+1]),PIECE_BYTES+1,test_table(data[:PIECE_BYTES+1])),Error.Out_Of_Memory) + testing.expect(t,has(&s,other)) + _,open_error:=open(path_in(folder,"other"),failing_allocator()) + testing.expect_value(t,open_error,Error.Out_Of_Memory) + close(&s) + leaked,bad_frees:=base.hostile_memory_end(&memory) + testing.expect(t,leaked==0 && bad_frees==0,"the store keeps no memory past close") + + // Roots it refuses. + testing.expect(t,os.write_entire_file(path_in(folder,"layout"),transmute([]u8)string("mirai store 0\n"))==nil) + _,layout_error:=open(folder) + testing.expect_value(t,layout_error,Error.Unsupported_Layout) + _,long_error:=open(strings.repeat("x",ROOT_MAX+1,context.temp_allocator)) + testing.expect_value(t,long_error,Error.Invalid_Argument) + _,empty_error:=open("") + testing.expect_value(t,empty_error,Error.Invalid_Argument) +} + +@(private="file") +hex_of :: proc(hash:Hash) -> string { + buffer:=new([64]u8,context.temp_allocator) + return hex(hash,buffer) +} diff --git a/store/receive.odin b/store/receive.odin new file mode 100644 index 000000000..1a931556f --- /dev/null +++ b/store/receive.odin @@ -0,0 +1,282 @@ +package store + +import "core:mem" +import "core:os" +import "mirai:std/base" + +// A blob being received lives in partial/
/: `table` (its size as +// u64 little endian and its piece hashes), `data` (the pieces at their +// offsets) and `have` (one byte per piece, 1 once it is in data). A folder +// appears there whole through a rename; each writer writes only its own +// piece and its own byte, so receivers in several processes need no lock. + +@(private) +TABLE_HEADER :: 8 + +@(private) +Partial :: struct { + table,data,have:^os.File, + size:i64, + count:int, +} + +@(private) +Partial_File :: enum {Data,Have} + +// Starts receiving a blob from its address and piece table; the table is +// checked against the address before anything is kept. A blob already +// complete or being received is left as it is. +receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { + if size<0 || len(pieces)!=piece_count(size) {return .Invalid_Argument} + if table_address(size,pieces)!=hash {return .Corrupt} + if has(s,hash) { + partial_discard(s,hash) + return .None + } + if size==0 { + _,error:=put_bytes(s,nil) + return error + } + existing,existing_error:=partial_open(s,hash,{.Data,.Have}) + partial_close(&existing) + if existing_error==.None {return .None} + if existing_error==.Corrupt {partial_discard(s,hash)} + folder_buffer,tmp_buffer:[PATH_BYTES]u8 + folder:=partial_folder(s,hash,&folder_buffer) or_return + tmp:=join(&tmp_buffer,s.root,"tmp") or_return + made,made_error:=os.make_directory_temp(tmp,"receive-*",s.allocator) + if made_error!=nil {return os_error(made_error)} + defer delete(made,s.allocator) + error:=partial_write(made,size,pieces) + if error==.None && os.rename(made,folder)!=nil { + error=.None if os.is_directory(folder) else .Io + } + base.remove_all(made) + return error +} + +// Keeps one received piece: checked against the table and written at its +// offset; a duplicate is harmless. The last missing piece completes the blob. +receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { + p,error:=partial_open(s,hash,{.Data,.Have}) + if error==.Missing {return complete_piece(s,hash,index,piece)} + if error!=.None {return error} + defer partial_close(&p) + if index<0 || index>=p.count || len(piece)!=piece_size(p.size,index) {return .Invalid_Argument} + expected:Hash + read_exact(p.table,expected[:],TABLE_HEADER+i64(index)*size_of(Hash)) or_return + if piece_value(piece,index)!=expected {return .Corrupt} + if _,write_error:=os.write_at(p.data,piece,i64(index)*PIECE_BYTES);write_error!=nil {return os_error(write_error)} + mark:=[1]u8{1} + if _,write_error:=os.write_at(p.have,mark[:],i64(index));write_error!=nil {return os_error(write_error)} + full:=partial_full(&p) or_return + if !full {return .None} + return partial_complete(s,hash,&p) +} + +// The indices of the pieces a blob being received still lacks; none for a complete blob. +receive_missing :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (missing:[]int,error:Error) { + if has(s,hash) {return nil,.None} + p:Partial + p,error=partial_open(s,hash,{.Have}) + if error==.Missing && has(s,hash) {return nil,.None} + if error!=.None {return nil,error} + defer partial_close(&p) + absent:=0 + marks:[4096]u8 + for at:=0;at (string,Error) { + name:[64]u8 + return join(buffer,s.root,"partial",hex(hash,&name)) +} + +// Opens the table of a blob being received, and the files asked for; the +// sizes of the table and of have are checked. +@(private) +partial_open :: proc(s:^Store,hash:Hash,files:bit_set[Partial_File]) -> (p:Partial,error:Error) { + folder_buffer,buffer:[PATH_BYTES]u8 + folder:=partial_folder(s,hash,&folder_buffer) or_return + open_error:os.Error + p.table,open_error=os.open(join(&buffer,folder,"table") or_return) + if open_error!=nil {return {},os_error(open_error)} + defer if error!=.None {partial_close(&p)} + header:[TABLE_HEADER]u8 + read_exact(p.table,header[:],0) or_return + for i in 0.. Error { + buffer:[PATH_BYTES]u8 + header:[TABLE_HEADER]u8 + for i in 0..>(8*uint(i)))} + table,table_error:=os.open(join(&buffer,folder,"table") or_return,{.Write,.Create,.Trunc}) + if table_error!=nil {return os_error(table_error)} + _,header_error:=os.write(table,header[:]) + _,pieces_error:=os.write(table,mem.slice_to_bytes(pieces)) + os.close(table) + if header_error!=nil || pieces_error!=nil {return .Io} + data,data_error:=os.open(join(&buffer,folder,"data") or_return,{.Write,.Create,.Trunc}) + if data_error!=nil {return os_error(data_error)} + os.close(data) + have,have_error:=os.open(join(&buffer,folder,"have") or_return,{.Write,.Create,.Trunc}) + if have_error!=nil {return os_error(have_error)} + truncate_error:=os.truncate(have,i64(len(pieces))) + os.close(have) + return os_error(truncate_error) +} + +@(private) +partial_full :: proc(p:^Partial) -> (full:bool,error:Error) { + marks:[4096]u8 + for at:=0;at Error { + Checking :: struct { + buffer:[PIECE_BYTES]u8, + p:^Partial, + damaged:bool, + } + checking:=Checking{p=p} + address:=address_value(p.size,proc(user:rawptr,index:int) -> (Hash,Error) { + c:=(^Checking)(user) + entry:Hash + if error:=read_exact(c.p.table,entry[:],TABLE_HEADER+i64(index)*size_of(Hash));error!=.None {return {},error} + piece:=c.buffer[:piece_size(c.p.size,index)] + got:=read_exact(c.p.data,piece,i64(index)*PIECE_BYTES) + if got!=.None && got!=.Corrupt {return {},got} + if got==.Corrupt || piece_value(piece,index)!=entry { + c.damaged=true + unmark:=[1]u8{0} + if _,write_error:=os.write_at(c.p.have,unmark[:],i64(index));write_error!=nil {return {},os_error(write_error)} + } + return entry,.None + },&checking) or_return + damaged:=checking.damaged + data_size,size_error:=os.file_size(p.data) + if size_error!=nil {return os_error(size_error)} + if address!=hash || data_size>p.size { + partial_close(p) + partial_discard(s,hash) + return .Corrupt + } + if damaged {return .None} + partial_close(p) + folder_buffer,data_buffer:[PATH_BYTES]u8 + folder:=partial_folder(s,hash,&folder_buffer) or_return + data:=join(&data_buffer,folder,"data") or_return + if !os.is_file(data) {return .None if has(s,hash) else .Missing} + blob_place(s,data,hash) or_return + partial_discard(s,hash) + return .None +} + +// A piece given for a blob with no partial folder: harmless once the blob is +// complete, when index and length fit it. +@(private) +complete_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { + size,error:=blob_size(s,hash) + if error!=.None {return error} + if index<0 || index>=piece_count(size) || len(piece)!=piece_size(size,index) {return .Invalid_Argument} + return .None +} + +// Moves the partial folder of a blob into tmp and removes it there. +@(private) +partial_discard :: proc(s:^Store,hash:Hash) -> Error { + folder_buffer,tmp_buffer,moved_buffer:[PATH_BYTES]u8 + folder:=partial_folder(s,hash,&folder_buffer) or_return + if !os.is_directory(folder) {return .Missing} + tmp:=join(&tmp_buffer,s.root,"tmp") or_return + made,made_error:=os.make_directory_temp(tmp,"discard-*",s.allocator) + if made_error!=nil {return os_error(made_error)} + defer delete(made,s.allocator) + moved:=join(&moved_buffer,made,"blob") or_return + renamed:=os.rename(folder,moved) + base.remove_all(made) + if renamed!=nil {return .Missing if !os.is_directory(folder) else .Io} + return .None +} + +// The range of a blob being received, when all its pieces are here. +@(private) +read_partial :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { + p:=partial_open(s,hash,{.Data,.Have}) or_return + defer partial_close(&p) + if offset>p.size || i64(len(buffer))>p.size-offset {return .Invalid_Argument} + if len(buffer)==0 {return .None} + first:=int(offset/PIECE_BYTES) + last:=int((offset+i64(len(buffer))-1)/PIECE_BYTES) + marks:[4096]u8 + for at:=first;at<=last;at+=len(marks) { + chunk:=marks[:min(len(marks),last+1-at)] + read_exact(p.have,chunk,i64(at)) or_return + for mark in chunk { + if mark==0 {return .Not_Here_Yet} + } + } + return read_exact(p.data,buffer,offset) +} diff --git a/store/refs.odin b/store/refs.odin new file mode 100644 index 000000000..b7ff7ef36 --- /dev/null +++ b/store/refs.odin @@ -0,0 +1,118 @@ +package store + +import "core:os" + +// A ref names an address: refs//, holding its hex. The store +// does not interpret refs. Namespaces and names are lowercase ASCII letters, +// digits, '.', '_' and '-', so they are the same file on every system. + +Ref :: struct { + name:string, + hash:Hash, +} + +ref_get :: proc(s:^Store,namespace,name:string) -> (hash:Hash,error:Error) { + buffer:[PATH_BYTES]u8 + path:=ref_file(s,namespace,name,&buffer) or_return + file,open_error:=os.open(path) + if open_error!=nil {return {},os_error(open_error)} + defer os.close(file) + return ref_parse(file) +} + +// Points the ref at hash, replacing what it held at once. +ref_set :: proc(s:^Store,namespace,name:string,hash:Hash) -> Error { + buffer,folder_buffer:[PATH_BYTES]u8 + path:=ref_file(s,namespace,name,&buffer) or_return + folder_make(join(&folder_buffer,s.root,"refs",namespace) or_return) or_return + text:[64]u8 + return file_write(s.root,path,transmute([]u8)hex(hash,&text)) +} + +ref_remove :: proc(s:^Store,namespace,name:string) -> Error { + buffer:[PATH_BYTES]u8 + path:=ref_file(s,namespace,name,&buffer) or_return + return os_error(os.remove(path)) +} + +// The refs of a namespace; an unknown namespace has none. ref_list_delete frees them. +ref_list :: proc(s:^Store,namespace:string,allocator:=context.allocator) -> (refs:[]Ref,error:Error) { + if !name_valid(namespace) {return nil,.Invalid_Argument} + buffer:[PATH_BYTES]u8 + folder:=join(&buffer,s.root,"refs",namespace) or_return + entries,read_error:=os.read_all_directory_by_path(folder,s.allocator) + if read_error==.Not_Exist {return nil,.None} + if read_error!=nil {return nil,os_error(read_error)} + defer os.file_info_slice_delete(entries,s.allocator) + list,make_error:=make([]Ref,len(entries),allocator) + if make_error!=nil {return nil,.Out_Of_Memory} + n:=0 + for entry in entries { + if entry.type!=.Regular || !name_valid(entry.name) {continue} + file,open_error:=os.open(entry.fullpath) + if open_error==.Not_Exist {continue} + error=os_error(open_error) + hash:Hash + if error==.None { + hash,error=ref_parse(file) + os.close(file) + } + name,name_error:=make([]u8,len(entry.name),allocator) + if error==.None && name_error!=nil {error=.Out_Of_Memory} + if error!=.None { + delete(name,allocator) + ref_list_delete(list[:n],allocator) + return nil,error + } + copy(name,entry.name) + list[n]={name=string(name),hash=hash} + n+=1 + } + return list[:n],.None +} + +ref_list_delete :: proc(list:[]Ref,allocator:=context.allocator) { + for ref in list {delete(ref.name,allocator)} + delete(list,allocator) +} + +// 1 to NAME_MAX of lowercase letters, digits, '.', '_' and '-'; neither only +// dots, nor ending in a dot, nor a name Windows keeps for a device. +name_valid :: proc(name:string) -> bool { + if len(name)==0 || len(name)>NAME_MAX || name[len(name)-1]=='.' {return false} + dots:=0 + stem:=len(name) + for c,i in transmute([]u8)name { + switch c { + case 'a'..='z','0'..='9','_','-': + case '.': + dots+=1 + stem=min(stem,i) + case: + return false + } + } + if dots==len(name) {return false} + switch name[:stem] { + case "con","prn","aux","nul": return false + } + if stem==4 && (name[:3]=="com" || name[:3]=="lpt") && name[3]>='0' && name[3]<='9' {return false} + return true +} + +@(private) +ref_file :: proc(s:^Store,namespace,name:string,buffer:^[PATH_BYTES]u8) -> (string,Error) { + if !name_valid(namespace) || !name_valid(name) {return "",.Invalid_Argument} + return join(buffer,s.root,"refs",namespace,name) +} + +@(private) +ref_parse :: proc(file:^os.File) -> (Hash,Error) { + text:[65]u8 + n,read_error:=os.read_at(file,text[:],0) + if n (total:Usage,error:Error) { + error=blobs_walk(s,proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { + total:=(^Usage)(user) + total.bytes+=size + total.blobs+=1 + return .None + },&total) + return +} + +// Removes every complete blob whose address is not in live, and the stale +// files of tmp: what it freed. The caller decides what is live. +sweep :: proc(s:^Store,live:[]Hash) -> (freed:Usage,error:Error) { + Sweep :: struct {live:[]Hash,freed:Usage} + sorted,make_error:=make([]Hash,len(live),s.allocator) + if make_error!=nil {return {},.Out_Of_Memory} + defer delete(sorted,s.allocator) + copy(sorted,live) + slice.sort_by(sorted,proc(a,b:Hash) -> bool { + a,b:=a,b + return mem.compare(a[:],b[:])<0 + }) + state:=Sweep{live=sorted} + error=blobs_walk(s,proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { + state:=(^Sweep)(user) + _,found:=slice.binary_search_by(state.live,hash,proc(a,b:Hash) -> slice.Ordering { + a,b:=a,b + return slice.Ordering(mem.compare(a[:],b[:])) + }) + if found {return .None} + removed:=os.remove(path) + if removed==.Not_Exist {return .None} + if removed!=nil {return os_error(removed)} + state.freed.bytes+=size + state.freed.blobs+=1 + return .None + },&state) + if error!=.None {return state.freed,error} + buffer:[PATH_BYTES]u8 + tmp:=join(&buffer,s.root,"tmp") or_return + entries,read_error:=os.read_all_directory_by_path(tmp,s.allocator) + if read_error!=nil {return state.freed,os_error(read_error)} + defer os.file_info_slice_delete(entries,s.allocator) + now:=time.now() + for entry in entries { + if time.diff(entry.modification_time,now)>TMP_STALE {base.remove_all(entry.fullpath)} + } + return state.freed,.None +} + +// Visits every complete blob: a file of blobs// named by its address. +@(private) +blobs_walk :: proc(s:^Store,visit:proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error,user:rawptr) -> Error { + buffer:[PATH_BYTES]u8 + blobs:=join(&buffer,s.root,"blobs") or_return + folders,folders_error:=os.read_all_directory_by_path(blobs,s.allocator) + if folders_error!=nil {return os_error(folders_error)} + defer os.file_info_slice_delete(folders,s.allocator) + for folder in folders { + if folder.type!=.Directory || len(folder.name)!=2 {continue} + files,files_error:=os.read_all_directory_by_path(folder.fullpath,s.allocator) + if files_error==.Not_Exist {continue} + if files_error!=nil {return os_error(files_error)} + defer os.file_info_slice_delete(files,s.allocator) + for file in files { + hash,ok:=parse_hex(file.name) + if !ok || file.type!=.Regular || file.name[:2]!=folder.name {continue} + visit(user,file.fullpath,hash,file.size) or_return + } + } + return .None +} diff --git a/store/store.odin b/store/store.odin new file mode 100644 index 000000000..2d59c2f2b --- /dev/null +++ b/store/store.odin @@ -0,0 +1,165 @@ +package store + +import "base:runtime" +import "core:os" + +// A content-addressed store of the machine: blobs named by their address and +// refs, mutable names of an address. Several Store values and processes share +// one root without locks: every file appears whole through a rename. + +// The first line of the root's layout file; another one is refused. +LAYOUT :: "mirai store 1\n" +ROOT_MAX :: 1024 // bytes of a root path +NAME_MAX :: 128 // bytes of a ref namespace or name + +@(private) +PATH_BYTES :: ROOT_MAX+256 + +Error :: enum u8 { + None, + Missing, // no blob, partial blob or ref with that name + Not_Here_Yet, // a range of a partial blob whose pieces have not arrived + Corrupt, // bytes that do not match their address or table, or a damaged file of the root + Invalid_Argument, + Out_Of_Memory, + Io, + Unsupported_Layout, // a root written by another layout of the store +} + +Store :: struct { + root:string, // owned, without a trailing separator + allocator:runtime.Allocator, +} + +// Opens the store at root, making its folders the first time. +open :: proc(root:string,allocator:=context.allocator) -> (s:Store,error:Error) { + trimmed:=root + for len(trimmed)>1 && (trimmed[len(trimmed)-1]=='/' || trimmed[len(trimmed)-1]=='\\') {trimmed=trimmed[:len(trimmed)-1]} + if trimmed=="" || len(trimmed)>ROOT_MAX {return {},.Invalid_Argument} + for c in transmute([]u8)trimmed { + if c==0 {return {},.Invalid_Argument} + } + buffer:[PATH_BYTES]u8 + folder_make(trimmed) or_return + for folder in ([4]string{"blobs","partial","refs","tmp"}) { + folder_make(join(&buffer,trimmed,folder) or_return) or_return + } + layout_check(trimmed) or_return + cloned,clone_error:=make([]u8,len(trimmed),allocator) + if clone_error!=nil {return {},.Out_Of_Memory} + copy(cloned,trimmed) + return {root=string(cloned),allocator=allocator},.None +} + +close :: proc(s:^Store) { + delete(s.root,s.allocator) + s^={} +} + +// The root every program of this machine shares: /Mirai/store, with +// forward slashes; "" when the system has no user data folder. +default_root :: proc(allocator:=context.allocator) -> string { + data,error:=os.user_data_dir(context.temp_allocator) + if error!=nil || data=="" {return ""} + suffix:="/Mirai/store" + joined,join_error:=make([]u8,len(data)+len(suffix),allocator) + if join_error!=nil {return ""} + copy(joined,data) + copy(joined[len(data):],suffix) + for &c in joined { + if c=='\\' {c='/'} + } + return string(joined) +} + +@(private) +layout_check :: proc(root:string) -> Error { + buffer:[PATH_BYTES]u8 + path:=join(&buffer,root,"layout") or_return + file,open_error:=os.open(path) + if open_error==nil { + defer os.close(file) + read:[len(LAYOUT)+1]u8 + n,_:=os.read_at(file,read[:],0) + size,size_error:=os.file_size(file) + if size_error!=nil {return .Io} + if size!=len(LAYOUT) || n!=len(LAYOUT) || string(read[:n])!=LAYOUT {return .Unsupported_Layout} + return .None + } + if open_error!=.Not_Exist {return .Io} + return file_write(root,path,transmute([]u8)string(LAYOUT)) +} + +// Copies parts joined by '/' into buffer. +@(private) +join :: proc(buffer:^[PATH_BYTES]u8,parts:..string) -> (string,Error) { + n:=0 + for part,i in parts { + if i>0 { + if n>=len(buffer) {return "",.Invalid_Argument} + buffer[n]='/' + n+=1 + } + if len(part)>len(buffer)-n {return "",.Invalid_Argument} + copy(buffer[n:],part) + n+=len(part) + } + return string(buffer[:n]),.None +} + +@(private) +folder_make :: proc(path:string) -> Error { + if os.is_directory(path) {return .None} + made:=os.make_directory_all(path) + if made!=nil && !os.is_directory(path) {return os_error(made)} + return .None +} + +@(private) +os_error :: proc(error:os.Error) -> Error { + if error==nil {return .None} + #partial switch e in error { + case runtime.Allocator_Error: + return .Out_Of_Memory + case os.General_Error: + if e==.Not_Exist {return .Missing} + } + return .Io +} + +// Writes data to a file of the root's tmp folder and renames it to path, +// which then holds all of it or what it held before. +@(private) +file_write :: proc(root:string,path:string,data:[]u8) -> Error { + temp_buffer:[PATH_BYTES]u8 + temp,file:=temp_create(root,&temp_buffer) or_return + _,write_error:=os.write(file,data) + sync_error:=os.sync(file) + os.close(file) + if write_error!=nil || sync_error!=nil { + os.remove(temp) + return os_error(write_error) if write_error!=nil else .Io + } + if os.rename(temp,path)!=nil { + os.remove(temp) + return .Io + } + return .None +} + +// A new file of the root's tmp folder, open to write; its path is in buffer. +@(private) +temp_create :: proc(root:string,buffer:^[PATH_BYTES]u8) -> (path:string,file:^os.File,error:Error) { + folder_buffer:[PATH_BYTES]u8 + folder:=join(&folder_buffer,root,"tmp") or_return + created,create_error:=os.create_temp_file(folder,"write-*") + if create_error!=nil {return "",nil,os_error(create_error)} + name:=os.name(created) + if len(name)>len(buffer) { + os.close(created) + os.remove(name) + return "",nil,.Invalid_Argument + } + copy(buffer[:],name) + return string(buffer[:len(name)]),created,.None +} diff --git a/store/store_test.odin b/store/store_test.odin new file mode 100644 index 000000000..80fbe046a --- /dev/null +++ b/store/store_test.odin @@ -0,0 +1,211 @@ +package store + +import "core:os" +import "core:strings" +import "core:testing" +import "mirai:std/base" + +@(private) +test_bytes :: proc(size:int,seed:u64) -> []u8 { + data:=make([]u8,size,context.temp_allocator) + h:=base.Hostile{state=seed} + for &b in data {b=u8(base.hostile_next(&h))} + return data +} + +@(private) +test_piece :: proc(data:[]u8,index:int) -> []u8 { + return data[index*PIECE_BYTES:min((index+1)*PIECE_BYTES,len(data))] +} + +@(private) +test_table :: proc(data:[]u8) -> []Hash { + pieces:=make([]Hash,piece_count(i64(len(data))),context.temp_allocator) + for &piece,i in pieces {piece=piece_value(test_piece(data,i),i)} + return pieces +} + +@(private) +test_store :: proc(t:^testing.T,root:string,loc:=#caller_location) -> (Store,bool) { + s,error:=open(root) + return s,testing.expect_value(t,error,Error.None,loc=loc) +} + +@(test) +address_is_the_same_through_put_bytes_put_file_and_receiving :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-address");if folder=="" {return} + Case :: struct {size:int} + cases:=[]Case{{0},{1},{PIECE_BYTES-1},{PIECE_BYTES},{PIECE_BYTES+1},{3*PIECE_BYTES+17}} + for c,n in cases { + data:=test_bytes(c.size,u64(n)+1) + root:=folder + a,ok:=test_store(t,join_test(root,"a",n));if !ok {return} + defer close(&a) + b,_:=test_store(t,join_test(root,"b",n)) + defer close(&b) + r,_:=test_store(t,join_test(root,"r",n)) + defer close(&r) + put,put_error:=put_bytes(&a,data) + testing.expect_value(t,put_error,Error.None) + testing.expect(t,put==address_of_bytes(data),"put bytes gives the address of the bytes") + testing.expect(t,put==table_address(i64(c.size),test_table(data)),"the address is that of the table") + source:=join_test(root,"source",n) + testing.expect(t,os.write_entire_file(source,data)==nil) + from_file,file_size,file_error:=address_of_file(source) + testing.expect(t,file_error==.None && from_file==put && file_size==i64(c.size),"the address of the file is the address of its bytes") + filed,filed_error:=put_file(&b,source) + testing.expect(t,filed_error==.None && filed==put,"put file gives the same address") + pieces,table_error:=blob_table(&a,put,context.temp_allocator) + testing.expect_value(t,table_error,Error.None) + testing.expect_value(t,receive_begin(&r,put,i64(c.size),pieces),Error.None) + for index:=len(pieces)-1;index>=0;index-=1 { + testing.expect_value(t,receive_piece(&r,put,index,test_piece(data,index)),Error.None) + } + for s in ([]^Store{&a,&b,&r}) { + testing.expect(t,has(s,put),"every path ends with the blob") + testing.expect_value(t,verify(s,put),Error.None) + size,size_error:=blob_size(s,put) + testing.expect(t,size_error==.None && size==i64(c.size)) + back:=make([]u8,c.size,context.temp_allocator) + testing.expect_value(t,read(s,put,0,back),Error.None) + testing.expect(t,string(back)==string(data),"the blob reads back its bytes") + path,path_error:=blob_path(s,put,context.temp_allocator) + testing.expect_value(t,path_error,Error.None) + on_disk,_:=os.read_entire_file(path,context.temp_allocator) + testing.expect(t,string(on_disk)==string(data),"the blob file holds the raw bytes") + } + again,again_error:=put_bytes(&a,data) + testing.expect(t,again_error==.None && again==put) + total,_:=usage(&a) + testing.expect_value(t,total,Usage{bytes=i64(c.size),blobs=1}) + } +} + +@(private) +join_test :: proc(root,name:string,n:int) -> string { + buffer:=make([]u8,len(root)+len(name)+8,context.temp_allocator) + copy(buffer,root) + copy(buffer[len(root):],"/") + copy(buffer[len(root)+1:],name) + buffer[len(root)+1+len(name)]='-' + buffer[len(root)+2+len(name)]=u8('0'+n%10) + return string(buffer[:len(root)+3+len(name)]) +} + +@(test) +a_wrong_table_or_piece_is_refused_and_leaves_no_blob :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-wrong");if folder=="" {return} + s,ok:=test_store(t,folder);if !ok {return} + defer close(&s) + data:=test_bytes(2*PIECE_BYTES+5,7) + hash:=address_of_bytes(data) + pieces:=test_table(data) + wrong:=make([]Hash,len(pieces),context.temp_allocator) + copy(wrong,pieces) + wrong[1][0]~=1 + testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),wrong),Error.Corrupt) + testing.expect_value(t,receive_begin(&s,hash,i64(len(data))+1,pieces),Error.Corrupt) + testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),pieces[:2]),Error.Invalid_Argument) + testing.expect_value(t,receive_begin(&s,hash,-1,nil),Error.Invalid_Argument) + _,missing_error:=receive_missing(&s,hash,context.temp_allocator) + testing.expect_value(t,missing_error,Error.Missing) + testing.expect(t,!has(&s,hash),"a refused table keeps nothing") + testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),pieces),Error.None) + bad:=make([]u8,PIECE_BYTES,context.temp_allocator) + copy(bad,test_piece(data,0)) + bad[100]~=0xff + testing.expect_value(t,receive_piece(&s,hash,0,bad),Error.Corrupt) + testing.expect_value(t,receive_piece(&s,hash,1,test_piece(data,0)),Error.Corrupt) + testing.expect_value(t,receive_piece(&s,hash,2,test_piece(data,1)),Error.Invalid_Argument) + testing.expect_value(t,receive_piece(&s,hash,3,test_piece(data,2)),Error.Invalid_Argument) + testing.expect_value(t,receive_piece(&s,hash,-1,test_piece(data,0)),Error.Invalid_Argument) + missing,_:=receive_missing(&s,hash,context.temp_allocator) + testing.expect_value(t,len(missing),3) + testing.expect(t,!has(&s,hash),"refused pieces complete nothing") + total,_:=usage(&s) + testing.expect_value(t,total.blobs,0) +} + +@(test) +receiving_resumes_after_reopening_and_reads_the_pieces_it_has :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-resume");if folder=="" {return} + data:=test_bytes(3*PIECE_BYTES+100,11) + hash:=address_of_bytes(data) + { + s,ok:=test_store(t,folder);if !ok {return} + defer close(&s) + testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),test_table(data)),Error.None) + testing.expect_value(t,receive_piece(&s,hash,0,test_piece(data,0)),Error.None) + testing.expect_value(t,receive_piece(&s,hash,2,test_piece(data,2)),Error.None) + testing.expect_value(t,receive_piece(&s,hash,2,test_piece(data,2)),Error.None) + } + s,ok:=test_store(t,folder);if !ok {return} + defer close(&s) + missing,missing_error:=receive_missing(&s,hash,context.temp_allocator) + testing.expect_value(t,missing_error,Error.None) + testing.expect(t,len(missing)==2 && missing[0]==1 && missing[1]==3,"the pieces it had survive a reopen") + size,_:=blob_size(&s,hash) + testing.expect_value(t,size,i64(len(data))) + Range :: struct {offset,count:int,error:Error} + ranges:=[]Range{ + {10,500,.None}, + {2*PIECE_BYTES,PIECE_BYTES,.None}, + {PIECE_BYTES-10,20,.Not_Here_Yet}, + {3*PIECE_BYTES,100,.Not_Here_Yet}, + {0,len(data),.Not_Here_Yet}, + {len(data)-10,20,.Invalid_Argument}, + } + for r in ranges { + buffer:=make([]u8,r.count,context.temp_allocator) + error:=read(&s,hash,i64(r.offset),buffer) + testing.expectf(t,error==r.error,"range %d+%d: %v",r.offset,r.count,error) + if r.error==.None {testing.expect(t,string(buffer)==string(data[r.offset:r.offset+r.count]))} + } + testing.expect(t,!has(&s,hash)) + testing.expect_value(t,receive_piece(&s,hash,3,test_piece(data,3)),Error.None) + testing.expect_value(t,receive_piece(&s,hash,1,test_piece(data,1)),Error.None) + testing.expect(t,has(&s,hash),"the last piece completes the blob") + testing.expect_value(t,verify(&s,hash),Error.None) + testing.expect_value(t,receive_piece(&s,hash,1,test_piece(data,1)),Error.None) + partials,_:=os.read_all_directory_by_path(strings.concatenate({folder,"/partial"},context.temp_allocator),context.temp_allocator) + testing.expect_value(t,len(partials),0) +} + +@(test) +sweep_keeps_exactly_the_live_blobs :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-sweep");if folder=="" {return} + s,ok:=test_store(t,folder);if !ok {return} + defer close(&s) + hashes:[5]Hash + bytes:i64 + for &hash,i in hashes { + data:=test_bytes(1000*(i+1),u64(i)+20) + hash,_=put_bytes(&s,data) + bytes+=i64(len(data)) + } + partial:=test_bytes(PIECE_BYTES+1,30) + partial_hash:=address_of_bytes(partial) + testing.expect_value(t,receive_begin(&s,partial_hash,i64(len(partial)),test_table(partial)),Error.None) + total,_:=usage(&s) + testing.expect_value(t,total,Usage{bytes=bytes,blobs=5}) + stale,fresh:=join_test(folder,"tmp/stale",0),join_test(folder,"tmp/fresh",0) + testing.expect(t,os.write_entire_file(stale,[]u8{1})==nil && os.write_entire_file(fresh,[]u8{1})==nil) + old:=os.modification_time_by_path(stale) or_else {} + testing.expect(t,os.change_times(stale,old,{_nsec=old._nsec-2*i64(TMP_STALE)})==nil) + unknown:Hash + unknown[0]=0xee + freed,error:=sweep(&s,{hashes[3],unknown,hashes[1]}) + testing.expect_value(t,error,Error.None) + testing.expect_value(t,freed,Usage{bytes=1000+3000+5000,blobs=3}) + for hash,i in hashes {testing.expectf(t,has(&s,hash)==(i==1 || i==3),"blob %d",i)} + total,_=usage(&s) + testing.expect_value(t,total,Usage{bytes=2000+4000,blobs=2}) + _,missing_error:=receive_missing(&s,partial_hash,context.temp_allocator) + testing.expect_value(t,missing_error,Error.None) + testing.expect(t,!os.exists(stale) && os.exists(fresh),"sweep removes only stale temporary files") + testing.expect_value(t,remove(&s,hashes[1]),Error.None) + testing.expect_value(t,remove(&s,hashes[1]),Error.Missing) + testing.expect_value(t,remove(&s,partial_hash),Error.None) + _,missing_error=receive_missing(&s,partial_hash,context.temp_allocator) + testing.expect_value(t,missing_error,Error.Missing) +} diff --git a/tools/mirai/package.odin b/tools/mirai/package.odin index c7820e3f4..76401594c 100644 --- a/tools/mirai/package.odin +++ b/tools/mirai/package.odin @@ -21,7 +21,7 @@ PACKAGE_USAGE :: "package engine --out " // in its binaries; their sources stay behind, as do the tests. @(private="file") PACKAGE_TRACKED :: [?]string{ - "asset","audio","core","editor","engine","host","net","platform","plugins","project","render","rhi","std","ui", + "asset","audio","core","editor","engine","host","net","platform","plugins","project","render","rhi","std","store","ui", "shaders","templates","recipes","assets/fonts",".agents/skills","docs/game","LICENSE-MIT","LICENSE-APACHE","THIRD_PARTY_NOTICES.md", } // Generated from those sources, and kept by git in no checkout. From c186798db1f41b6c3bd983b53039bb8fc9372c3f Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:07:47 -0300 Subject: [PATCH 002/123] std/blake3: BLAKE3 in portable Odin with group values Hash of bytes, an incremental hasher, the chaining value of each 64 KiB group and the root from those values, checked against the official vectors. Written with Codex. Co-Authored-By: Claude Fable 5.2 --- .agents/skills/mirai-std/SKILL.md | 2 + THIRD_PARTY_NOTICES.md | 4 + docs/ARCHITECTURE.md | 2 +- std/blake3/blake3.odin | 269 ++++++++++++++++++++++++++++++ std/blake3/blake3_test.odin | 163 ++++++++++++++++++ 5 files changed, 439 insertions(+), 1 deletion(-) create mode 100644 std/blake3/blake3.odin create mode 100644 std/blake3/blake3_test.odin diff --git a/.agents/skills/mirai-std/SKILL.md b/.agents/skills/mirai-std/SKILL.md index 8dac26832..c4d67a889 100644 --- a/.agents/skills/mirai-std/SKILL.md +++ b/.agents/skills/mirai-std/SKILL.md @@ -103,6 +103,8 @@ description: Mirai standard library for games, with the packages `std/base`, `st [octree](references/spatial.md#octree-loose-octree) and [which to use](references/spatial.md#which-to-use). The AABB tree and the BVH are in [physics](../mirai-physics/references/physics.md#stdspatial). +- BLAKE3 of bytes, and the values of its 64 KiB groups a blob is checked by piece by piece + (`std/blake3`): the code; the official vectors are its test. - Streams, schemas, hashes, import and migration of values (`std/serial`): [std/serial](../mirai-assets/references/serial.md#stdserial). - What the structure of a `.ui` is judged by (`std/ui_format`): diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index fe0097dfe..8bafb4e55 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -42,6 +42,10 @@ folders, which include the notices of the work they carry. Parts of Mirai are rewrites in Odin or Slang of the algorithms of these projects. Their notices are kept here and, where the license asks for it, in the header of the file. +- **BLAKE3** (CC0 1.0, Apache-2.0, or Apache-2.0 with LLVM exceptions; + © 2019 Jack O'Connor and Samuel Neves), . + `std/blake3`, translated from the reference implementation under CC0 1.0, with test vectors. + Game. - **Clay v0.14** (zlib; © 2024 Nic Barker), . `ui/layout_tree.odin`, altered. Game. - **MikkTSpace** (zlib; © 2011 Morten S. Mikkelsen), . diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fcdb32964..e0ceb6874 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -322,7 +322,7 @@ project's world through Numbered layers; a package only imports lower layers. -- **0** `std/*`: standard library (base, math, serial, jobs, spatial, geom, anim, ai, input, +- **0** `std/*`: standard library (base, blake3, math, serial, jobs, spatial, geom, anim, ai, input, subprocess, ui_format). It does not depend on the engine. `std/compressor` is not imported as `mirai:`: it is the open implementation of the collection `compressor` (`data`, `texture`, `blocks`, the last engine-time), which every compile names and which the private diff --git a/std/blake3/blake3.odin b/std/blake3/blake3.odin new file mode 100644 index 000000000..c3765eb2b --- /dev/null +++ b/std/blake3/blake3.odin @@ -0,0 +1,269 @@ +package blake3 + +Hash :: [32]u8 +GROUP_BYTES :: 64 << 10 + +Hasher :: struct { + length: u64, + block: [BLOCK_BYTES]u8, + chunk_value: Chaining_Value, + tree: Tree, +} + +init :: proc "contextless" (hasher: ^Hasher) { + if hasher != nil { + hasher^ = {} + } +} + +update :: proc "contextless" (hasher: ^Hasher, data: []u8) -> bool { + if hasher == nil || u64(len(data)) > max(u64) - hasher.length { + return false + } + update_at(hasher, data, 0) + return true +} + +final :: proc "contextless" (hasher: ^Hasher) -> Hash { + if hasher == nil { + return {} + } + return root_hash(final_node(hasher, 0)) +} + +hash :: proc "contextless" (data: []u8) -> Hash { + hasher: Hasher + update_at(&hasher, data, 0) + return final(&hasher) +} + +group_value :: proc "contextless" (data: []u8, index: u64) -> (value: Hash, ok: bool) { + if len(data) == 0 || len(data) > GROUP_BYTES || index > max(u64) / GROUP_CHUNKS { + return {}, false + } + first_chunk := index * GROUP_CHUNKS + hasher: Hasher + update_at(&hasher, data, first_chunk) + return encode(compress(final_node(&hasher, first_chunk))), true +} + +root :: proc "contextless" (values: []Hash, length: u64) -> (hash: Hash, ok: bool) { + if length <= GROUP_BYTES || u64(len(values)) != 1 + (length - 1) / GROUP_BYTES { + return {}, false + } + + tree: Tree + right := len(values) - 1 + for value, index in values[:right] { + tree_push(&tree, u64(index), decode(transmute([8]u32le) value)) + } + node := tree_fold(&tree, u64(right), decode(transmute([8]u32le) values[right])) + return root_hash(node), true +} + +@(private="file") +BLOCK_BYTES :: 64 +@(private="file") +CHUNK_BYTES :: 1024 +@(private="file") +GROUP_CHUNKS :: GROUP_BYTES / CHUNK_BYTES + +@(private="file") +Chaining_Value :: [8]u32 +@(private="file") +Tree :: [54]Chaining_Value // A u64 byte length has at most 54 bits of chunk indices. + +@(private="file") +Node :: struct { + value: Chaining_Value, + words: [16]u32, + counter: u64, + block_length, flags: u32, +} + +@(private="file") +IV :: Chaining_Value{ + 0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, + 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19, +} +@(private="file") +PERMUTATION :: [16]int{2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8} +@(private="file") +CHUNK_START :: 1 << 0 +@(private="file") +CHUNK_END :: 1 << 1 +@(private="file") +PARENT :: 1 << 2 +@(private="file") +ROOT :: 1 << 3 + +@(private="file") +update_at :: proc "contextless" (hasher: ^Hasher, data: []u8, first_chunk: u64) { + remaining := data + for len(remaining) > 0 { + used := int(hasher.length % BLOCK_BYTES) + // Hold a full block until another byte establishes that it is not the last block. + if hasher.length != 0 && used == 0 { + node := block_node(hasher, first_chunk) + if hasher.length % CHUNK_BYTES == 0 { + node.flags |= CHUNK_END + tree_push(&hasher.tree, (hasher.length - 1) / CHUNK_BYTES, compress(node)) + } else { + hasher.chunk_value = compress(node) + } + hasher.block = {} + } + + take := min(BLOCK_BYTES - used, len(remaining)) + copy(hasher.block[used:used + take], remaining[:take]) + hasher.length += u64(take) + remaining = remaining[take:] + } +} + +@(private="file") +block_node :: proc "contextless" (hasher: ^Hasher, first_chunk: u64) -> Node { + last := max(hasher.length, 1) - 1 + node := Node{ + words = decode(transmute([16]u32le) hasher.block), + counter = first_chunk + last / CHUNK_BYTES, + block_length = u32(hasher.length - last / BLOCK_BYTES * BLOCK_BYTES), + } + if last % CHUNK_BYTES < BLOCK_BYTES { + node.value = IV + node.flags = CHUNK_START + } else { + node.value = hasher.chunk_value + } + return node +} + +@(private="file") +final_node :: proc "contextless" (hasher: ^Hasher, first_chunk: u64) -> Node { + node := block_node(hasher, first_chunk) + node.flags |= CHUNK_END + preceding := (max(hasher.length, 1) - 1) / CHUNK_BYTES + if preceding == 0 { + return node + } + return tree_fold(&hasher.tree, preceding, compress(node)) +} + +@(private="file") +tree_push :: proc "contextless" (tree: ^Tree, preceding: u64, value: Chaining_Value) { + // Each set bit in the number of preceding leaves has one subtree at that height. + pending := preceding + merged := value + level := 0 + for pending & 1 != 0 { + merged = compress(parent(tree[level], merged)) + pending >>= 1 + level += 1 + } + tree[level] = merged +} + +@(private="file") +tree_fold :: proc "contextless" (tree: ^Tree, preceding: u64, right: Chaining_Value) -> Node { + node: Node + value := right + pending := preceding + for level := 0; pending != 0; level += 1 { + if pending & 1 != 0 { + node = parent(tree[level], value) + if pending != 1 { + value = compress(node) + } + } + pending >>= 1 + } + return node +} + +@(private="file") +parent :: proc "contextless" (left, right: Chaining_Value) -> Node { + node := Node{value = IV, block_length = BLOCK_BYTES, flags = PARENT} + for word, index in left { + node.words[index] = word + node.words[index + 8] = right[index] + } + return node +} + +@(private="file") +root_hash :: proc "contextless" (node: Node) -> Hash { + output := node + output.counter = 0 + output.flags |= ROOT + return encode(compress(output)) +} + +@(private="file") +decode :: proc "contextless" (bytes: [$N]u32le) -> (words: [N]u32) { + for word, index in bytes { + words[index] = u32(word) + } + return +} + +@(private="file") +encode :: proc "contextless" (value: Chaining_Value) -> Hash { + words: [8]u32le + for word, index in value { + words[index] = u32le(word) + } + return transmute(Hash) words +} + +@(private="file") +compress :: proc "contextless" (node: Node) -> (value: Chaining_Value) { + state := [16]u32{ + node.value[0], node.value[1], node.value[2], node.value[3], + node.value[4], node.value[5], node.value[6], node.value[7], + IV[0], IV[1], IV[2], IV[3], + u32(node.counter), u32(node.counter >> 32), node.block_length, node.flags, + } + words := node.words + for _ in 0..<6 { + mix_round(&state, words) + permuted: [16]u32 + for source, index in PERMUTATION { + permuted[index] = words[source] + } + words = permuted + } + mix_round(&state, words) + for _, index in value { + value[index] = state[index] ~ state[index + 8] + } + return +} + +@(private="file") +mix_round :: proc "contextless" (state: ^[16]u32, words: [16]u32) { + mix(state, 0, 4, 8, 12, words[0], words[1]) + mix(state, 1, 5, 9, 13, words[2], words[3]) + mix(state, 2, 6, 10, 14, words[4], words[5]) + mix(state, 3, 7, 11, 15, words[6], words[7]) + mix(state, 0, 5, 10, 15, words[8], words[9]) + mix(state, 1, 6, 11, 12, words[10], words[11]) + mix(state, 2, 7, 8, 13, words[12], words[13]) + mix(state, 3, 4, 9, 14, words[14], words[15]) +} + +@(private="file") +mix :: proc "contextless" (state: ^[16]u32, a, b, c, d: int, x, y: u32) { + state[a] += state[b] + x + state[d] = rotate_right(state[d] ~ state[a], 16) + state[c] += state[d] + state[b] = rotate_right(state[b] ~ state[c], 12) + state[a] += state[b] + y + state[d] = rotate_right(state[d] ~ state[a], 8) + state[c] += state[d] + state[b] = rotate_right(state[b] ~ state[c], 7) +} + +@(private="file") +rotate_right :: proc "contextless" (word: u32, shift: uint) -> u32 { + return (word >> shift) | (word << (32 - shift)) +} diff --git a/std/blake3/blake3_test.odin b/std/blake3/blake3_test.odin new file mode 100644 index 000000000..37d7b8cca --- /dev/null +++ b/std/blake3/blake3_test.odin @@ -0,0 +1,163 @@ +package blake3 + +import "core:encoding/hex" +import "core:testing" + +@(test) +hash_matches_the_official_vectors :: proc(t: ^testing.T) { + vectors := [?]struct{length: int, expected: string}{ + {0, "af1349b9f5f9a1a6a0404dea36dcc9499bcb25c9adc112b7cc9a93cae41f3262"}, + {1, "2d3adedff11b61f14c886e35afa036736dcd87a74d27b5c1510225d0f592e213"}, + {2, "7b7015bb92cf0b318037702a6cdd81dee41224f734684c2c122cd6359cb1ee63"}, + {3, "e1be4d7a8ab5560aa4199eea339849ba8e293d55ca0a81006726d184519e647f"}, + {4, "f30f5ab28fe047904037f77b6da4fea1e27241c5d132638d8bedce9d40494f32"}, + {5, "b40b44dfd97e7a84a996a91af8b85188c66c126940ba7aad2e7ae6b385402aa2"}, + {6, "06c4e8ffb6872fad96f9aaca5eee1553eb62aed0ad7198cef42e87f6a616c844"}, + {7, "3f8770f387faad08faa9d8414e9f449ac68e6ff0417f673f602a646a891419fe"}, + {8, "2351207d04fc16ade43ccab08600939c7c1fa70a5c0aaca76063d04c3228eaeb"}, + {63, "e9bc37a594daad83be9470df7f7b3798297c3d834ce80ba85d6e207627b7db7b"}, + {64, "4eed7141ea4a5cd4b788606bd23f46e212af9cacebacdc7d1f4c6dc7f2511b98"}, + {65, "de1e5fa0be70df6d2be8fffd0e99ceaa8eb6e8c93a63f2d8d1c30ecb6b263dee"}, + {127, "d81293fda863f008c09e92fc382a81f5a0b4a1251cba1634016a0f86a6bd640d"}, + {128, "f17e570564b26578c33bb7f44643f539624b05df1a76c81f30acd548c44b45ef"}, + {129, "683aaae9f3c5ba37eaaf072aed0f9e30bac0865137bae68b1fde4ca2aebdcb12"}, + {1023, "10108970eeda3eb932baac1428c7a2163b0e924c9a9e25b35bba72b28f70bd11"}, + {1024, "42214739f095a406f3fc83deb889744ac00df831c10daa55189b5d121c855af7"}, + {1025, "d00278ae47eb27b34faecf67b4fe263f82d5412916c1ffd97c8cb7fb814b8444"}, + {2048, "e776b6028c7cd22a4d0ba182a8bf62205d2ef576467e838ed6f2529b85fba24a"}, + {2049, "5f4d72f40d7a5f82b15ca2b2e44b1de3c2ef86c426c95c1af0b6879522563030"}, + {3072, "b98cb0ff3623be03326b373de6b9095218513e64f1ee2edd2525c7ad1e5cffd2"}, + {3073, "7124b49501012f81cc7f11ca069ec9226cecb8a2c850cfe644e327d22d3e1cd3"}, + {4096, "015094013f57a5277b59d8475c0501042c0b642e531b0a1c8f58d2163229e969"}, + {4097, "9b4052b38f1c5fc8b1f9ff7ac7b27cd242487b3d890d15c96a1c25b8aa0fb995"}, + {5120, "9cadc15fed8b5d854562b26a9536d9707cadeda9b143978f319ab34230535833"}, + {5121, "628bd2cb2004694adaab7bbd778a25df25c47b9d4155a55f8fbd79f2fe154cff"}, + {6144, "3e2e5b74e048f3add6d21faab3f83aa44d3b2278afb83b80b3c35164ebeca205"}, + {6145, "f1323a8631446cc50536a9f705ee5cb619424d46887f3c376c695b70e0f0507f"}, + {7168, "61da957ec2499a95d6b8023e2b0e604ec7f6b50e80a9678b89d2628e99ada77a"}, + {7169, "a003fc7a51754a9b3c7fae0367ab3d782dccf28855a03d435f8cfe74605e7817"}, + {8192, "aae792484c8efe4f19e2ca7d371d8c467ffb10748d8a5a1ae579948f718a2a63"}, + {8193, "bab6c09cb8ce8cf459261398d2e7aef35700bf488116ceb94a36d0f5f1b7bc3b"}, + {16384, "f875d6646de28985646f34ee13be9a576fd515f76b5b0a26bb324735041ddde4"}, + {31744, "62b6960e1a44bcc1eb1a611a8d6235b6b4b78f32e7abc4fb4c6cdcce94895c47"}, + {102400, "bc3e3d41a1146b069abffad3c0d44860cf664390afce4d9661f7902e7943e085"}, + } + input: [102400]u8 + fill_pattern(input[:]) + for vector in vectors { + expected: Hash + _, ok := hex.decode_into_buffer(transmute([]u8) vector.expected, expected[:]) + testing.expectf(t, ok && hash(input[:vector.length]) == expected, + "Official vector for %d bytes", vector.length) + } +} + +@(test) +incremental_and_group_hashes_match_the_whole_blob :: proc(t: ^testing.T) { + sizes := [?]int{ + 0, 1, 1023, 1024, 1025, + GROUP_BYTES - 1, GROUP_BYTES, GROUP_BYTES + 1, + 2*GROUP_BYTES - 1, 2*GROUP_BYTES, 2*GROUP_BYTES + 5, + 3*GROUP_BYTES - 1, 3*GROUP_BYTES, 3*GROUP_BYTES + 5, + 5*GROUP_BYTES - 1, 5*GROUP_BYTES, 5*GROUP_BYTES + 5, + 8*GROUP_BYTES - 1, 8*GROUP_BYTES, 8*GROUP_BYTES + 5, + 9*GROUP_BYTES - 1, 9*GROUP_BYTES, 9*GROUP_BYTES + 5, + } + splits := [?]int{1, 63, 0, 3, 1024, 17, 65529, 131} + sample :: "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%^&*()-_" + sample_8 :: sample + sample + sample + sample + sample + sample + sample + sample + sample_64 :: sample_8 + sample_8 + sample_8 + sample_8 + + sample_8 + sample_8 + sample_8 + sample_8 + sample_512 :: sample_64 + sample_64 + sample_64 + sample_64 + + sample_64 + sample_64 + sample_64 + sample_64 + sample_4096 :: sample_512 + sample_512 + sample_512 + sample_512 + + sample_512 + sample_512 + sample_512 + sample_512 + input := transmute([]u8) string(sample_4096 + sample_4096) + hasher: Hasher + + for size in sizes { + expected := hash(input[:size]) + init(&hasher) + offset := 0 + for step := 0; offset < size; step += 1 { + end := min(offset + splits[step % len(splits)], size) + ok := update(&hasher, input[offset:end]) + testing.expectf(t, ok, "Update at byte %d of %d", offset, size) + offset = end + } + testing.expectf(t, final(&hasher) == expected, "Incremental hash of %d bytes", size) + + if size > GROUP_BYTES { + values: [10]Hash + count := 1 + (size - 1) / GROUP_BYTES + for index in 0.. Date: Wed, 18 Nov 2026 18:15:28 -0300 Subject: [PATCH 003/123] std/blake3: compress chunks from the input, four at once Whole chunks are compressed straight from the caller's bytes with unrolled rounds, four chunks at a time through core:simd where there are that many. Written with Codex. Co-Authored-By: Claude Fable 5.2 --- std/blake3/blake3.odin | 258 +++++++++++++++++------------------- std/blake3/blake3_test.odin | 2 +- std/blake3/compress.odin | 164 +++++++++++++++++++++++ 3 files changed, 284 insertions(+), 140 deletions(-) create mode 100644 std/blake3/compress.odin diff --git a/std/blake3/blake3.odin b/std/blake3/blake3.odin index c3765eb2b..6322f2660 100644 --- a/std/blake3/blake3.odin +++ b/std/blake3/blake3.odin @@ -28,7 +28,7 @@ final :: proc "contextless" (hasher: ^Hasher) -> Hash { if hasher == nil { return {} } - return root_hash(final_node(hasher, 0)) + return finish(hasher, 0, ROOT) } hash :: proc "contextless" (data: []u8) -> Hash { @@ -44,7 +44,7 @@ group_value :: proc "contextless" (data: []u8, index: u64) -> (value: Hash, ok: first_chunk := index * GROUP_CHUNKS hasher: Hasher update_at(&hasher, data, first_chunk) - return encode(compress(final_node(&hasher, first_chunk))), true + return finish(&hasher, first_chunk, 0), true } root :: proc "contextless" (values: []Hash, length: u64) -> (hash: Hash, ok: bool) { @@ -54,43 +54,30 @@ root :: proc "contextless" (values: []Hash, length: u64) -> (hash: Hash, ok: boo tree: Tree right := len(values) - 1 - for value, index in values[:right] { - tree_push(&tree, u64(index), decode(transmute([8]u32le) value)) + for group, index in values[:right] { + value := decode_hash(group) + tree_push(&tree, u64(index), &value) } - node := tree_fold(&tree, u64(right), decode(transmute([8]u32le) values[right])) - return root_hash(node), true + value := decode_hash(values[right]) + tree_fold(&tree, u64(right), &value, ROOT) + return encode(value), true } -@(private="file") +@(private) BLOCK_BYTES :: 64 -@(private="file") +@(private) CHUNK_BYTES :: 1024 @(private="file") GROUP_CHUNKS :: GROUP_BYTES / CHUNK_BYTES -@(private="file") +@(private) Chaining_Value :: [8]u32 @(private="file") Tree :: [54]Chaining_Value // A u64 byte length has at most 54 bits of chunk indices. -@(private="file") -Node :: struct { - value: Chaining_Value, - words: [16]u32, - counter: u64, - block_length, flags: u32, -} - -@(private="file") -IV :: Chaining_Value{ - 0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, - 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19, -} -@(private="file") -PERMUTATION :: [16]int{2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8} -@(private="file") +@(private) CHUNK_START :: 1 << 0 -@(private="file") +@(private) CHUNK_END :: 1 << 1 @(private="file") PARENT :: 1 << 2 @@ -99,109 +86,155 @@ ROOT :: 1 << 3 @(private="file") update_at :: proc "contextless" (hasher: ^Hasher, data: []u8, first_chunk: u64) { + if len(data) == 0 { + return + } + remaining := data - for len(remaining) > 0 { - used := int(hasher.length % BLOCK_BYTES) - // Hold a full block until another byte establishes that it is not the last block. - if hasher.length != 0 && used == 0 { - node := block_node(hasher, first_chunk) - if hasher.length % CHUNK_BYTES == 0 { - node.flags |= CHUNK_END - tree_push(&hasher.tree, (hasher.length - 1) / CHUNK_BYTES, compress(node)) - } else { - hasher.chunk_value = compress(node) - } - hasher.block = {} + if hasher.length != 0 { + preceding := (hasher.length - 1) / CHUNK_BYTES + used := int(1 + (hasher.length - 1) % CHUNK_BYTES) + take := min(CHUNK_BYTES - used, len(remaining)) + append_chunk(hasher, remaining[:take], used, first_chunk + preceding) + if take == len(remaining) { + return } + value := finish_chunk(hasher, CHUNK_BYTES, first_chunk + preceding, 0) + tree_push(&hasher.tree, preceding, &value) + remaining = remaining[take:] + } + + preceding := hasher.length / CHUNK_BYTES + // The final block must remain available for the CHUNK_END and ROOT flags. + for len(remaining) > BATCH_CHUNKS * CHUNK_BYTES { + batch := full_chunks(raw_data(remaining), first_chunk + preceding) + for &value in batch { + tree_push(&hasher.tree, preceding, &value) + preceding += 1 + } + remaining = remaining[BATCH_CHUNKS * CHUNK_BYTES:] + } + for len(remaining) > CHUNK_BYTES { + value := full_chunk(raw_data(remaining), first_chunk + preceding) + tree_push(&hasher.tree, preceding, &value) + preceding += 1 + remaining = remaining[CHUNK_BYTES:] + } + hasher.length = preceding * CHUNK_BYTES + append_chunk(hasher, remaining, 0, first_chunk + preceding) +} + +@(private="file") +append_chunk :: proc "contextless" (hasher: ^Hasher, data: []u8, used: int, counter: u64) { + if len(data) == 0 { + return + } - take := min(BLOCK_BYTES - used, len(remaining)) - copy(hasher.block[used:used + take], remaining[:take]) - hasher.length += u64(take) + hasher.length += u64(len(data)) + remaining := data + flags: u32 = CHUNK_START + if used > 0 { + buffered := 1 + (used - 1) % BLOCK_BYTES + take := min(BLOCK_BYTES - buffered, len(remaining)) + copy(hasher.block[buffered:buffered + take], remaining[:take]) + if take == len(remaining) { + return + } + if used <= BLOCK_BYTES { + hasher.chunk_value = IV + } else { + flags = 0 + } + compress(&hasher.chunk_value, raw_data(hasher.block[:]), counter, BLOCK_BYTES, flags) + flags = 0 remaining = remaining[take:] + } else { + hasher.chunk_value = IV + } + + for len(remaining) > BLOCK_BYTES { + compress(&hasher.chunk_value, raw_data(remaining), counter, BLOCK_BYTES, flags) + flags = 0 + remaining = remaining[BLOCK_BYTES:] } + copy(hasher.block[:], remaining) } @(private="file") -block_node :: proc "contextless" (hasher: ^Hasher, first_chunk: u64) -> Node { - last := max(hasher.length, 1) - 1 - node := Node{ - words = decode(transmute([16]u32le) hasher.block), - counter = first_chunk + last / CHUNK_BYTES, - block_length = u32(hasher.length - last / BLOCK_BYTES * BLOCK_BYTES), +finish_chunk :: proc "contextless" ( + hasher: ^Hasher, used: int, counter: u64, flags: u32, +) -> Chaining_Value { + value := IV + mode := flags | CHUNK_END | CHUNK_START + if used > BLOCK_BYTES { + value = hasher.chunk_value + mode &= ~u32(CHUNK_START) } - if last % CHUNK_BYTES < BLOCK_BYTES { - node.value = IV - node.flags = CHUNK_START + buffered := used - max(used - 1, 0) / BLOCK_BYTES * BLOCK_BYTES + if buffered == BLOCK_BYTES { + compress(&value, raw_data(hasher.block[:]), counter, BLOCK_BYTES, mode) } else { - node.value = hasher.chunk_value + padded: [BLOCK_BYTES]u8 + copy(padded[:], hasher.block[:buffered]) + compress(&value, raw_data(padded[:]), counter, u32(buffered), mode) } - return node + return value } @(private="file") -final_node :: proc "contextless" (hasher: ^Hasher, first_chunk: u64) -> Node { - node := block_node(hasher, first_chunk) - node.flags |= CHUNK_END +finish :: proc "contextless" (hasher: ^Hasher, first_chunk: u64, flags: u32) -> Hash { preceding := (max(hasher.length, 1) - 1) / CHUNK_BYTES + used := int(hasher.length - preceding * CHUNK_BYTES) if preceding == 0 { - return node + return encode(finish_chunk(hasher, used, first_chunk, flags)) } - return tree_fold(&hasher.tree, preceding, compress(node)) + value := finish_chunk(hasher, used, first_chunk + preceding, 0) + tree_fold(&hasher.tree, preceding, &value, flags) + return encode(value) } @(private="file") -tree_push :: proc "contextless" (tree: ^Tree, preceding: u64, value: Chaining_Value) { +tree_push :: proc "contextless" (tree: ^Tree, preceding: u64, value: ^Chaining_Value) { // Each set bit in the number of preceding leaves has one subtree at that height. pending := preceding - merged := value level := 0 for pending & 1 != 0 { - merged = compress(parent(tree[level], merged)) + merge_parent(&tree[level], value, 0) pending >>= 1 level += 1 } - tree[level] = merged + tree[level] = value^ } @(private="file") -tree_fold :: proc "contextless" (tree: ^Tree, preceding: u64, right: Chaining_Value) -> Node { - node: Node - value := right +tree_fold :: proc "contextless" ( + tree: ^Tree, preceding: u64, right: ^Chaining_Value, flags: u32, +) { pending := preceding for level := 0; pending != 0; level += 1 { if pending & 1 != 0 { - node = parent(tree[level], value) - if pending != 1 { - value = compress(node) - } + merge_parent(&tree[level], right, flags if pending == 1 else 0) } pending >>= 1 } - return node } @(private="file") -parent :: proc "contextless" (left, right: Chaining_Value) -> Node { - node := Node{value = IV, block_length = BLOCK_BYTES, flags = PARENT} - for word, index in left { - node.words[index] = word - node.words[index + 8] = right[index] +merge_parent :: proc "contextless" (left, right: ^Chaining_Value, flags: u32) { + words: [16]u32le = --- + #unroll for index in 0..<8 { + words[index] = u32le(left[index]) + words[index + 8] = u32le(right[index]) } - return node -} - -@(private="file") -root_hash :: proc "contextless" (node: Node) -> Hash { - output := node - output.counter = 0 - output.flags |= ROOT - return encode(compress(output)) + right^ = IV + compress(right, ([^]u8)(&words), 0, BLOCK_BYTES, PARENT | flags) } @(private="file") -decode :: proc "contextless" (bytes: [$N]u32le) -> (words: [N]u32) { - for word, index in bytes { - words[index] = u32(word) +decode_hash :: proc "contextless" (hash: Hash) -> (value: Chaining_Value) { + words := transmute([8]u32le) hash + #unroll for index in 0..<8 { + value[index] = u32(words[index]) } return } @@ -214,56 +247,3 @@ encode :: proc "contextless" (value: Chaining_Value) -> Hash { } return transmute(Hash) words } - -@(private="file") -compress :: proc "contextless" (node: Node) -> (value: Chaining_Value) { - state := [16]u32{ - node.value[0], node.value[1], node.value[2], node.value[3], - node.value[4], node.value[5], node.value[6], node.value[7], - IV[0], IV[1], IV[2], IV[3], - u32(node.counter), u32(node.counter >> 32), node.block_length, node.flags, - } - words := node.words - for _ in 0..<6 { - mix_round(&state, words) - permuted: [16]u32 - for source, index in PERMUTATION { - permuted[index] = words[source] - } - words = permuted - } - mix_round(&state, words) - for _, index in value { - value[index] = state[index] ~ state[index + 8] - } - return -} - -@(private="file") -mix_round :: proc "contextless" (state: ^[16]u32, words: [16]u32) { - mix(state, 0, 4, 8, 12, words[0], words[1]) - mix(state, 1, 5, 9, 13, words[2], words[3]) - mix(state, 2, 6, 10, 14, words[4], words[5]) - mix(state, 3, 7, 11, 15, words[6], words[7]) - mix(state, 0, 5, 10, 15, words[8], words[9]) - mix(state, 1, 6, 11, 12, words[10], words[11]) - mix(state, 2, 7, 8, 13, words[12], words[13]) - mix(state, 3, 4, 9, 14, words[14], words[15]) -} - -@(private="file") -mix :: proc "contextless" (state: ^[16]u32, a, b, c, d: int, x, y: u32) { - state[a] += state[b] + x - state[d] = rotate_right(state[d] ~ state[a], 16) - state[c] += state[d] - state[b] = rotate_right(state[b] ~ state[c], 12) - state[a] += state[b] + y - state[d] = rotate_right(state[d] ~ state[a], 8) - state[c] += state[d] - state[b] = rotate_right(state[b] ~ state[c], 7) -} - -@(private="file") -rotate_right :: proc "contextless" (word: u32, shift: uint) -> u32 { - return (word >> shift) | (word << (32 - shift)) -} diff --git a/std/blake3/blake3_test.odin b/std/blake3/blake3_test.odin index 37d7b8cca..4f432c238 100644 --- a/std/blake3/blake3_test.odin +++ b/std/blake3/blake3_test.odin @@ -55,7 +55,7 @@ hash_matches_the_official_vectors :: proc(t: ^testing.T) { @(test) incremental_and_group_hashes_match_the_whole_blob :: proc(t: ^testing.T) { sizes := [?]int{ - 0, 1, 1023, 1024, 1025, + 0, 1, 1023, 1024, 1025, 4095, 4096, 4097, GROUP_BYTES - 1, GROUP_BYTES, GROUP_BYTES + 1, 2*GROUP_BYTES - 1, 2*GROUP_BYTES, 2*GROUP_BYTES + 5, 3*GROUP_BYTES - 1, 3*GROUP_BYTES, 3*GROUP_BYTES + 5, diff --git a/std/blake3/compress.odin b/std/blake3/compress.odin new file mode 100644 index 000000000..549c87a32 --- /dev/null +++ b/std/blake3/compress.odin @@ -0,0 +1,164 @@ +package blake3 + +import "base:intrinsics" +import "core:simd" + +@(private) +IV :: Chaining_Value{ + 0x6a09e667, 0xbb67ae85, 0x3c6ef372, 0xa54ff53a, + 0x510e527f, 0x9b05688c, 0x1f83d9ab, 0x5be0cd19, +} +@(private="file") +MESSAGE_SCHEDULE :: [7][16]int{ + {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15}, + {2, 6, 3, 10, 7, 0, 4, 13, 1, 11, 12, 5, 9, 14, 15, 8}, + {3, 4, 10, 12, 13, 2, 7, 14, 6, 5, 9, 0, 11, 15, 8, 1}, + {10, 7, 12, 9, 14, 3, 13, 15, 4, 0, 11, 2, 5, 8, 1, 6}, + {12, 13, 9, 11, 15, 10, 14, 8, 7, 2, 5, 3, 0, 1, 6, 4}, + {9, 14, 11, 5, 8, 12, 15, 1, 13, 3, 0, 10, 2, 6, 4, 7}, + {11, 15, 5, 0, 1, 9, 8, 6, 14, 10, 2, 12, 3, 4, 7, 13}, +} + +@(private) +compress :: #force_inline proc "contextless" ( + value: ^Chaining_Value, block: [^]u8, counter: u64, length, flags: u32, +) { + compress_lanes(value, block, u32(counter), u32(counter >> 32), length, flags) +} + +@(private) +full_chunk :: proc "contextless" (data: [^]u8, counter: u64) -> Chaining_Value { + value := IV + compress(&value, data, counter, BLOCK_BYTES, CHUNK_START) + for offset := BLOCK_BYTES; offset < CHUNK_BYTES - BLOCK_BYTES; offset += BLOCK_BYTES { + compress(&value, &data[offset], counter, BLOCK_BYTES, 0) + } + compress(&value, &data[CHUNK_BYTES - BLOCK_BYTES], counter, BLOCK_BYTES, CHUNK_END) + return value +} + +@(private) +BATCH_CHUNKS :: 4 + +@(private) +full_chunks :: proc "contextless" ( + data: [^]u8, counter: u64, +) -> (values: [BATCH_CHUNKS]Chaining_Value) { + // A lane holds one chunk, with the same word index in every lane. + iv := IV + chaining: [8]simd.u32x4 = --- + #unroll for index in 0..<8 { + chaining[index] = simd.u32x4(iv[index]) + } + + low, high: [BATCH_CHUNKS]u32 = ---, --- + #unroll for lane in 0..> 32) + } + counter_low := transmute(simd.u32x4) low + counter_high := transmute(simd.u32x4) high + + for offset := 0; offset < CHUNK_BYTES; offset += BLOCK_BYTES { + flags := u32(CHUNK_START) if offset == 0 else + u32(CHUNK_END) if offset == CHUNK_BYTES - BLOCK_BYTES else 0 + compress_lanes(&chaining, &data[offset], counter_low, counter_high, BLOCK_BYTES, flags) + } + #unroll for lane in 0.. u32 --- + + @(link_name="llvm.fshr.v4i32") + funnel_right_vector :: proc "none" (high, low, shift: simd.u32x4) -> simd.u32x4 --- +} From fed60d13652fd46aca03e07ff466a4751b69b7fb Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:15:28 -0300 Subject: [PATCH 004/123] asset: records of imports in the machine store The package manifest, the build manifest and the list of a project (packages.mirai), as versioned texts, with their refs, the import key, the name of the build a program without importers reads (build_select) and packages_resolve, which lists the blobs of what a project uses and reports what is not built. Beside the sidecar: nothing uses them yet. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-assets/SKILL.md | 8 + .../skills/mirai-assets/references/formats.md | 30 + .agents/skills/mirai-imports/SKILL.md | 2 + asset/packages.odin | 960 ++++++++++++++++++ asset/packages_test.odin | 296 ++++++ asset/sidecar.odin | 16 +- docs/ARCHITECTURE.md | 3 +- 7 files changed, 1299 insertions(+), 16 deletions(-) create mode 100644 asset/packages.odin create mode 100644 asset/packages_test.odin diff --git a/.agents/skills/mirai-assets/SKILL.md b/.agents/skills/mirai-assets/SKILL.md index 7d9dd9be8..a73211151 100644 --- a/.agents/skills/mirai-assets/SKILL.md +++ b/.agents/skills/mirai-assets/SKILL.md @@ -27,6 +27,8 @@ types, scenes and their prefab instances, registry). [container](references/formats.md#container), [typed tables](references/formats.md#typed-tables), [asset types](references/formats.md#asset-types), [versions](references/formats.md#versions), [registry](references/formats.md#registry) and [text](references/formats.md#text). +- The records of imports in the machine store (package and build manifests, `packages.mirai`), + their refs and `packages_resolve`: [packages](references/formats.md#packages). - The collection `compressor`, profiles, codecs, streams and decoders: [compression](references/formats.md#compression). - Streams, schemas, migration, paths, text: [serial](references/serial.md#stdserial). @@ -131,6 +133,9 @@ types, scenes and their prefab instances, registry). - **The header says Scene or Prefab.** One `.scene` extension and kind; `Header_Flag.Prefab` is what whoever made the file chose, never guessed from its roots, and a prefab has exactly one root ([prefab](references/scenes.md#prefab)). +- **A project pins the manifest of each package it uses, never a build**; builds are found by + key through refs, and resolving a list never removes nor rebuilds + ([packages](references/formats.md#packages)). - **A guest mount never takes an id the registry has**: what the program shipped wins over what a player adds ([guest mounts](references/formats.md#registry)). @@ -157,6 +162,9 @@ types, scenes and their prefab instances, registry). `a_scene_saved_with_another_layout_of_a_component_loads_it_by_field_names` of `engine/scene`; - the document of a scene or a `.ui` from a file outside the assets (what `dump` and git show): `the_commands_read_write_and_check_the_document_of_an_asset` of `cook/author`; + - records of packages: `package_records_read_back_what_they_write`, + `packages_resolve_lists_the_blobs_of_what_a_project_uses` and + `package_records_refuse_hostile_text` of `asset/packages_test.odin`; - registry: `asset/registry_test.odin` and, in the editor, `editor/content_test.odin`; - layers of an imported material: `asset/kinds_test.odin`, and the overrides shared with prefabs in the editor, `editor/prefabs_test.odin`; diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index f9689523e..cc2f9a366 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -270,6 +270,36 @@ scan opens every file and writes it again. on the creation times of a machine, so a game ships neither until the copy is written with its own id. +## Packages + +The records of the imports kept in the [machine store](../../mirai-store/SKILL.md) +(`asset/packages.odin`), three texts with a version each. They are written beside the +sidecars; the importers, the registry and the build do not use them yet. + +- **Package manifest** (`Package`): a blob, named by `packages/` on the machine + that ingested it. The import that ingests a source writes it. +- **Build manifest** (`Build`): a blob, named by `imports/` (Final, which peers + share) or `imports-fast/` (never shared) by the import that builds it, and by + `builds/` by the refresh of a project that chose it. +- **Project list** (`Packages`): `packages.mirai` beside `project.mirai`, versioned. The editor + and the commands that add or remove a package write it. + +- **A project pins the manifest, never a build.** The manifest holds what the source is (its + `identity`), where it was (`origin`, a hint) and each asset with its id, and nothing that + depends on the engine version: an engine that changes a version leaves the list as it is. +- **Builds are found by name.** `import_key` combines the identity with the versions of what + imports; any machine with the manifest computes it, without the source, and a peer that built + that key gives it. `build_select` combines the manifest with the version of each kind of the + cache it holds, so a program without importers (the dev player) finds the build the refresh + chose. The folders of `cook/cache` name theirs in `cache-`. +- `packages_resolve` lists, for each package of the list, its assets of the cache the project + uses with what they depend on, at `/` as the sidecar lists them, with + the path of their blob. A missing build lists them not built; a manifest or a build the store + lacks or does not read, and an asset the manifest lacks, are problems of the result. It + removes, builds and fetches nothing. +- An asset id is random at the first ingest and the manifest keeps it; an ingest again takes + each id from the manifest before it. + ## Text `document_text_write` produces the JSON that `mirai asset dump` prints and `unpack` saves, of diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index a97cb02e6..145a20858 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -23,6 +23,8 @@ the assets. - Where a source lives (in the assets, beside what it makes), its sidecar (settings, folder, ids), what an import keeps in the imported folder and what in the assets, reimporting, a moved or renamed source and parts with `owner`: [Identity](references/imports.md#identity-and-reimport). +- The records that take the place of the sidecar in the machine store (package and build + manifests, `packages.mirai`): [packages](../mirai-assets/references/formats.md#packages). - A checkout without its imported folder, what a build imports and when a source is imported again: [Sync](references/imports.md#sync). - Adding or changing a format (`Format`, `Reader`, `Source_Space`, version): diff --git a/asset/packages.odin b/asset/packages.odin new file mode 100644 index 000000000..8fae37651 --- /dev/null +++ b/asset/packages.odin @@ -0,0 +1,960 @@ +package asset + +import "base:runtime" +import "core:fmt" +import "core:os" +import "core:strings" +import "mirai:std/base" +import "mirai:std/serial" +import "mirai:store" + +// The records of the imports kept in the machine store, as text: +// +// - Package: what one ingest of a source made, a blob of the store. Its id, +// where the source was, what the source and its settings are (identity) and +// each asset with its id. Nothing in it depends on the version of the +// engine. The import that ingests the source writes it and names it with +// the ref REF_PACKAGES/. +// - Build: the blob of each asset of one build of a package. The import that +// builds it writes it, names it with REF_IMPORTS/ (Final) or +// REF_IMPORTS_FAST/, and the refresh of a project names the one +// it chose with REF_BUILDS/, which a program that has no +// importer finds. +// - Packages: the list of a project (PACKAGES_FILE, beside its manifest), +// versioned: the packages it uses by the hash of their manifest, the folder +// of its assets each is listed in and the assets it uses. The editor and +// the commands that add or remove a package write it. +// +// What a parse gives lives in its allocator, an arena: nothing frees it piece +// by piece. A text that is not a record of its version is an error, never a +// default. + +PACKAGES_FILE :: "packages.mirai" +PACKAGES_VERSION :: 1 +PACKAGE_MANIFEST_VERSION :: 1 +BUILD_MANIFEST_VERSION :: 1 +RECORD_BYTES :: 16<<20 // the most a record takes as text: a larger one is refused unread + +// Namespaces of the refs of the records. +REF_PACKAGES :: "packages" // : its manifest, on a machine that ingested it +REF_IMPORTS :: "imports" // : a build with Final textures, which peers share +REF_IMPORTS_FAST :: "imports-fast" // : a build with Fast textures, never shared +REF_BUILDS :: "builds" // : the build the refresh of this machine chose +REF_CACHE_PREFIX :: "cache-" // and the name of a folder of cook/cache: - +IMPORT_KEY_VERSIONS :: 32 // the most versions an import key combines + +// Where the source of a package was on the machine that ingested it: a hint +// to ingest it again there, not part of what it is. +Package_Origin :: struct { + path:string, + folder:bool, // the source is a folder (an Unreal package), not a file + importer:string, // the format or the importer of a plugin that reads it +} + +Package_Asset :: struct { + kind:Fourcc, + name:string, // in the source: the mesh, the material, the image + index:int, // among those of its kind in the source + id:base.Asset_Id, + part:bool, // a part of the root of the package, decided when it was first ingested + owner:base.Asset_Id, + dependencies:[dynamic]base.Asset_Id, +} + +Package :: struct { + id:base.Asset_Id, + name:string, + origin:Package_Origin, + identity:store.Hash, // of the bytes of the source and its inputs, its settings and the code of the plugin that reads it + settings:string, // the text of the settings object; "" without one + inputs:[dynamic]string, // the files the import read besides the source, from its folder + assets:[dynamic]Package_Asset, +} + +Build_Output :: struct { + id:base.Asset_Id, + kind:Fourcc, + kind_version:u16, + blob:store.Hash, + thumbnail:store.Hash, // zero: none +} + +Build :: struct { + package_id:base.Asset_Id, + key:store.Hash, // import_key of what built it + final:bool, // its textures are Final, else Fast + encoders:u64, // of the block encoders that built its textures + outputs:[dynamic]Build_Output, +} + +Project_Package :: struct { + id:base.Asset_Id, + manifest:store.Hash, + folder:string, // of the assets of the project, where its assets are listed; "": the top + assets:[dynamic]base.Asset_Id, // those the project uses, with what they depend on; none: all +} + +Packages :: struct { + packages:[dynamic]Project_Package, +} + +// What a program reads, as one value: the identity of a package and the +// versions of what imports it and writes its assets. Any machine that has the +// manifest computes it, without the source. ok is false with more than +// IMPORT_KEY_VERSIONS versions. +import_key :: proc(identity:store.Hash,versions:[]u64) -> (key:store.Hash,ok:bool) { + TAG :: "mirai import key" + if len(versions)>IMPORT_KEY_VERSIONS {return} + bytes:[len(TAG)+1+size_of(store.Hash)+8*IMPORT_KEY_VERSIONS]u8 + n:=copy(bytes[:],TAG) + bytes[n]=u8(len(versions)) + n+=1 + identity:=identity + n+=copy(bytes[n:],identity[:]) + for version in versions {n+=word_put(bytes[n:],version,8)} + return store.address_of_bytes(bytes[:n]),true +} + +// The name of the build of the manifest with hash manifest that a program +// reads: the version of each kind of the cache its assets are of, as this +// program reads it. The refresh of a project names its choice so; a program +// without importers finds it by the same name. +build_select :: proc(manifest:store.Hash,p:^Package) -> store.Hash { + TAG :: "mirai build select" + bytes:[len(TAG)+size_of(store.Hash)+6*len(KINDS)]u8 + n:=copy(bytes[:],TAG) + manifest:=manifest + n+=copy(bytes[n:],manifest[:]) + for info in KINDS { + if !import_cached(info.kind) {continue} + for &a in p.assets { + if a.kind!=info.kind {continue} + n+=word_put(bytes[n:],u64(info.kind),4) + n+=word_put(bytes[n:],u64(info.version),2) + break + } + } + return store.address_of_bytes(bytes[:n]) +} + +// The namespace of the refs of a folder of cook/cache, in buffer; "" for a +// name that does not fit. +cache_namespace :: proc(name:string,buffer:^[store.NAME_MAX]u8) -> string { + if len(REF_CACHE_PREFIX)+len(name)>store.NAME_MAX {return ""} + n:=copy(buffer[:],REF_CACHE_PREFIX) + n+=copy(buffer[n:],name) + return string(buffer[:n]) +} + +@(private="file") +word_put :: proc(out:[]u8,value:u64,size:int) -> int { + for i in 0..>uint(8*i))} + return size +} + +// The text of a manifest; .Invalid for one that would not read back as it is. +package_text :: proc(p:^Package,allocator:=context.allocator) -> (text:string,error:Error) { + if !package_valid(p) {return "",.Invalid} + w:serial.Writer + serial.writer_init(&w,1024,allocator) + serial.text_write_raw(&w,"{\n \"version\": ") + serial.text_write_int(&w,PACKAGE_MANIFEST_VERSION) + serial.text_write_raw(&w,",\n \"id\": ") + id_write(&w,p.id) + serial.text_write_raw(&w,",\n \"name\": ") + serial.text_write_string(&w,p.name) + serial.text_write_raw(&w,",\n \"origin\": {\"path\": ") + serial.text_write_string(&w,p.origin.path) + serial.text_write_raw(&w,p.origin.folder?", \"folder\": true, \"importer\": ":", \"folder\": false, \"importer\": ") + serial.text_write_string(&w,p.origin.importer) + serial.text_write_raw(&w,"},\n \"identity\": ") + hash_write(&w,p.identity) + if p.settings!="" { + serial.text_write_raw(&w,",\n \"settings\": ") + serial.text_write_raw(&w,p.settings) + } + serial.text_write_raw(&w,",\n \"inputs\": [") + for input,at in p.inputs { + if at>0 {serial.text_write_raw(&w,", ")} + serial.text_write_string(&w,input) + } + serial.text_write_raw(&w,"],\n \"assets\": [") + for &a,at in p.assets { + serial.text_write_raw(&w,at>0?",\n {\"kind\": ":"\n {\"kind\": ") + kind_write(&w,a.kind) + serial.text_write_raw(&w,", \"name\": ") + serial.text_write_string(&w,a.name) + serial.text_write_raw(&w,", \"index\": ") + serial.text_write_int(&w,i64(a.index)) + serial.text_write_raw(&w,", \"id\": ") + id_write(&w,a.id) + if a.part {serial.text_write_raw(&w,", \"part\": true")} + if a.owner!={} { + serial.text_write_raw(&w,", \"owner\": ") + id_write(&w,a.owner) + } + if len(a.dependencies)>0 { + serial.text_write_raw(&w,", \"dependencies\": [") + for d,i in a.dependencies { + if i>0 {serial.text_write_raw(&w,", ")} + id_write(&w,d) + } + serial.text_write_raw(&w,"]") + } + serial.text_write_raw(&w,"}") + } + serial.text_write_raw(&w,len(p.assets)>0?"\n ]\n}\n":"]\n}\n") + return writer_text(&w) +} + +package_parse :: proc(text:string,allocator:=context.allocator) -> (p:Package,error:Error) { + if len(text)>RECORD_BYTES {return {},.Too_Large} + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + p.inputs=make([dynamic]string,allocator) + p.assets=make([dynamic]Package_Asset,allocator) + Field :: enum {Version,Id,Name,Origin,Identity,Settings,Inputs,Assets} + seen:bit_set[Field] + version:u64 + expect(&r,'{') or_return + for first:=true;!serial.text_accept(&r,'}');first=false { + if !first {expect(&r,',') or_return} + name:=key(&r) or_return + field:Field + switch name { + case "version": field=.Version + case "id": field=.Id + case "name": field=.Name + case "origin": field=.Origin + case "identity": field=.Identity + case "settings": field=.Settings + case "inputs": field=.Inputs + case "assets": field=.Assets + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Version: version=count_read(&r,max(u64)) or_return + case .Id: p.id=id_read(&r) or_return + case .Name: p.name=string_read(&r,allocator) or_return + case .Origin: p.origin=origin_read(&r,allocator) or_return + case .Identity: p.identity=hash_read(&r) or_return + case .Settings: + serial.text_space(&r) + start:=r.offset + if serial.text_peek(&r)!='{' || !serial.text_skip(&r) {return {},.Corrupt} + p.settings=clone(text[start:r.offset],allocator) or_return + case .Inputs: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + input:=string_read(&r,allocator) or_return + if _,no_room:=append(&p.inputs,input);no_room!=nil {return {},.Out_Of_Memory} + } + case .Assets: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + a:=asset_read(&r,allocator) or_return + if _,no_room:=append(&p.assets,a);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + if !serial.text_end(&r) {return {},.Corrupt} + if version!=PACKAGE_MANIFEST_VERSION {return {},.Version} + if seen!={.Version,.Id,.Name,.Origin,.Identity,.Inputs,.Assets} && seen!={.Version,.Id,.Name,.Origin,.Identity,.Settings,.Inputs,.Assets} {return {},.Corrupt} + if !package_valid(&p) {return {},.Corrupt} + return p,.None +} + +// The text of a build manifest; .Invalid for one that would not read back as it is. +build_text :: proc(b:^Build,allocator:=context.allocator) -> (text:string,error:Error) { + if !build_valid(b) {return "",.Invalid} + w:serial.Writer + serial.writer_init(&w,512,allocator) + serial.text_write_raw(&w,"{\n \"version\": ") + serial.text_write_int(&w,BUILD_MANIFEST_VERSION) + serial.text_write_raw(&w,",\n \"package\": ") + id_write(&w,b.package_id) + serial.text_write_raw(&w,",\n \"key\": ") + hash_write(&w,b.key) + serial.text_write_raw(&w,b.final?",\n \"final\": true,\n \"encoders\": \"":",\n \"final\": false,\n \"encoders\": \"") + serial.text_write_hex(&w,b.encoders,16,false) + serial.text_write_raw(&w,"\",\n \"outputs\": [") + for o,at in b.outputs { + serial.text_write_raw(&w,at>0?",\n {\"id\": ":"\n {\"id\": ") + id_write(&w,o.id) + serial.text_write_raw(&w,", \"kind\": ") + kind_write(&w,o.kind) + serial.text_write_raw(&w,", \"kind_version\": ") + serial.text_write_int(&w,i64(o.kind_version)) + serial.text_write_raw(&w,", \"blob\": ") + hash_write(&w,o.blob) + if o.thumbnail!={} { + serial.text_write_raw(&w,", \"thumbnail\": ") + hash_write(&w,o.thumbnail) + } + serial.text_write_raw(&w,"}") + } + serial.text_write_raw(&w,len(b.outputs)>0?"\n ]\n}\n":"]\n}\n") + return writer_text(&w) +} + +build_parse :: proc(text:string,allocator:=context.allocator) -> (b:Build,error:Error) { + if len(text)>RECORD_BYTES {return {},.Too_Large} + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + b.outputs=make([dynamic]Build_Output,allocator) + Field :: enum {Version,Package,Key,Final,Encoders,Outputs} + seen:bit_set[Field] + version:u64 + expect(&r,'{') or_return + for first:=true;!serial.text_accept(&r,'}');first=false { + if !first {expect(&r,',') or_return} + name:=key(&r) or_return + field:Field + switch name { + case "version": field=.Version + case "package": field=.Package + case "key": field=.Key + case "final": field=.Final + case "encoders": field=.Encoders + case "outputs": field=.Outputs + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Version: version=count_read(&r,max(u64)) or_return + case .Package: b.package_id=id_read(&r) or_return + case .Key: b.key=hash_read(&r) or_return + case .Final: b.final=bool_read(&r) or_return + case .Encoders: + digits,ok:=serial.text_string(&r) + if !ok || len(digits)!=16 {return {},.Corrupt} + parsed,parsed_ok:=base.hex_parse_u64(digits) + if !parsed_ok {return {},.Corrupt} + b.encoders=parsed + case .Outputs: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + o:=output_read(&r) or_return + if _,no_room:=append(&b.outputs,o);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + if !serial.text_end(&r) {return {},.Corrupt} + if version!=BUILD_MANIFEST_VERSION {return {},.Version} + if seen!={.Version,.Package,.Key,.Final,.Encoders,.Outputs} {return {},.Corrupt} + if !build_valid(&b) {return {},.Corrupt} + return b,.None +} + +// The text of the list of a project; .Invalid for one that would not read back as it is. +packages_text :: proc(list:^Packages,allocator:=context.allocator) -> (text:string,error:Error) { + if !packages_valid(list) {return "",.Invalid} + w:serial.Writer + serial.writer_init(&w,256,allocator) + serial.text_write_raw(&w,"{\n \"version\": ") + serial.text_write_int(&w,PACKAGES_VERSION) + serial.text_write_raw(&w,",\n \"packages\": [") + for &entry,at in list.packages { + serial.text_write_raw(&w,at>0?",\n {\"id\": ":"\n {\"id\": ") + id_write(&w,entry.id) + serial.text_write_raw(&w,", \"manifest\": ") + hash_write(&w,entry.manifest) + serial.text_write_raw(&w,", \"folder\": ") + serial.text_write_string(&w,entry.folder) + serial.text_write_raw(&w,", \"assets\": [") + for id,i in entry.assets { + if i>0 {serial.text_write_raw(&w,", ")} + id_write(&w,id) + } + serial.text_write_raw(&w,"]}") + } + serial.text_write_raw(&w,len(list.packages)>0?"\n ]\n}\n":"]\n}\n") + return writer_text(&w) +} + +packages_parse :: proc(text:string,allocator:=context.allocator) -> (list:Packages,error:Error) { + if len(text)>RECORD_BYTES {return {},.Too_Large} + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + list.packages=make([dynamic]Project_Package,allocator) + Field :: enum {Version,Packages} + seen:bit_set[Field] + version:u64 + expect(&r,'{') or_return + for first:=true;!serial.text_accept(&r,'}');first=false { + if !first {expect(&r,',') or_return} + name:=key(&r) or_return + field:Field + switch name { + case "version": field=.Version + case "packages": field=.Packages + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Version: version=count_read(&r,max(u64)) or_return + case .Packages: + expect(&r,'[') or_return + for begun:=false;!serial.text_accept(&r,']');begun=true { + if begun {expect(&r,',') or_return} + entry:=entry_read(&r,allocator) or_return + if _,no_room:=append(&list.packages,entry);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + if !serial.text_end(&r) {return {},.Corrupt} + if version!=PACKAGES_VERSION {return {},.Version} + if seen!={.Version,.Packages} || !packages_valid(&list) {return {},.Corrupt} + return list,.None +} + +// The list of the project at root, in allocator. found is false, with an +// empty list, when the project has none. +packages_load :: proc(root:string,allocator:=context.allocator) -> (list:Packages,found:bool,error:Error) { + path:=fmt.tprintf("%s/%s",root,PACKAGES_FILE) + data,read_error:=serial.file_read(path,context.temp_allocator,limit=RECORD_BYTES) + switch read_error { + case .None: + case .Not_Found: + list.packages=make([dynamic]Project_Package,allocator) + return list,false,.None + case .Too_Large: return {},false,.Too_Large + case .Out_Of_Memory: return {},false,.Out_Of_Memory + case .Open,.Read,.Write,.Rename,.Invalid_Alignment: return {},false,.IO + } + list,error=packages_parse(string(data),allocator) + return list,error==.None,error +} + +// Writes the list of the project at root, unless its file already says the same. +packages_save :: proc(root:string,list:^Packages) -> Error { + text:=packages_text(list,context.temp_allocator) or_return + path:=fmt.tprintf("%s/%s",root,PACKAGES_FILE) + if data,read_error:=os.read_entire_file(path,context.temp_allocator);read_error==nil && string(data)==text {return .None} + switch serial.file_write_atomic(path,transmute([]u8)text) { + case .None: return .None + case .Out_Of_Memory: return .Out_Of_Memory + case .Not_Found,.Open,.Read,.Write,.Rename,.Too_Large,.Invalid_Alignment: return .IO + } + return .IO +} + +// Keeps a manifest in the store, by the hash of its text. +package_put :: proc(s:^store.Store,p:^Package) -> (hash:store.Hash,error:Error) { + text:=package_text(p,context.temp_allocator) or_return + put_error:store.Error + hash,put_error=store.put_bytes(s,transmute([]u8)text) + return hash,store_error(put_error) +} + +// The manifest the store keeps under hash, in allocator. .Not_Found without it. +package_get :: proc(s:^store.Store,hash:store.Hash,allocator:=context.allocator) -> (p:Package,error:Error) { + text:=record_get(s,hash) or_return + return package_parse(text,allocator) +} + +build_put :: proc(s:^store.Store,b:^Build) -> (hash:store.Hash,error:Error) { + text:=build_text(b,context.temp_allocator) or_return + put_error:store.Error + hash,put_error=store.put_bytes(s,transmute([]u8)text) + return hash,store_error(put_error) +} + +build_get :: proc(s:^store.Store,hash:store.Hash,allocator:=context.allocator) -> (b:Build,error:Error) { + text:=record_get(s,hash) or_return + return build_parse(text,allocator) +} + +Listed_State :: enum u8 { + Built, + Not_Built, // no build this program reads: a machine with the source, or a peer that built it, gives it +} + +// An asset of the cache a project lists from its packages. +Package_Listed :: struct { + path:string, // in the registry: /, as listed_path makes it + file:string, // the blob that holds it; "" when not built + id:base.Asset_Id, + kind:Fourcc, + package_id:base.Asset_Id, + state:Listed_State, +} + +Package_Problem_Code :: enum u8 { + Manifest_Missing, // the store has no blob of the manifest the project names + Manifest_Unreadable, // it has one that is not a manifest of this version, or of another package + Build_Unreadable, // the build its ref names is missing or does not read: its assets are not built + Asset_Unknown, // the project uses an asset its manifest does not have +} + +Package_Problem :: struct { + package_id:base.Asset_Id, + asset:base.Asset_Id, // Asset_Unknown: the one the project names + code:Package_Problem_Code, +} + +Packages_Resolved :: struct { + listed:[dynamic]Package_Listed, + problems:[dynamic]Package_Problem, +} + +// What a project lists from the packages it uses: each asset of the cache it +// uses, with what they depend on, at its path, and the blob of the build +// REF_BUILDS names for its manifest, or Not_Built. Documents are files of the +// project and are not listed. A manifest or a build the store does not have +// or does not read is a problem of the result, never an error; nothing is +// removed, built or fetched. Two packages that list one path take the name of +// their package after it, " ()". In allocator, an arena; error +// only when memory runs out. +packages_resolve :: proc(s:^store.Store,list:^Packages,allocator:=context.allocator) -> (resolved:Packages_Resolved,error:Error) { + resolved.listed=make([dynamic]Package_Listed,allocator) + resolved.problems=make([dynamic]Package_Problem,allocator) + names:=make([dynamic]string,context.temp_allocator) + for &entry in list.packages { + p,manifest_error:=package_get(s,entry.manifest,context.temp_allocator) + if manifest_error==.Out_Of_Memory {return {},.Out_Of_Memory} + if manifest_error==.None && p.id!=entry.id {manifest_error=.Corrupt} + if manifest_error!=.None { + problem_add(&resolved,{entry.id,{},manifest_error==.Not_Found?.Manifest_Missing:.Manifest_Unreadable}) or_return + continue + } + used,no_used:=make([]bool,len(p.assets),context.temp_allocator) + by_id:=make(map[base.Asset_Id]int,context.temp_allocator) + if no_used!=nil || reserve(&by_id,len(p.assets))!=nil {return {},.Out_Of_Memory} + for &a,at in p.assets {by_id[a.id]=at} + if len(entry.assets)==0 { + for &u in used {u=true} + } else { + pending:=make([dynamic]int,context.temp_allocator) + for id in entry.assets { + at,known:=by_id[id] + if !known { + problem_add(&resolved,{entry.id,id,.Asset_Unknown}) or_return + continue + } + if _,no_room:=append(&pending,at);no_room!=nil {return {},.Out_Of_Memory} + } + for len(pending)>0 { + at:=pop(&pending) + if used[at] {continue} + used[at]=true + for d in p.assets[at].dependencies { + if next,known:=by_id[d];known && !used[next] { + if _,no_room:=append(&pending,next);no_room!=nil {return {},.Out_Of_Memory} + } + } + } + } + outputs:=make(map[base.Asset_Id]Build_Output,context.temp_allocator) + digits:[64]u8 + built,ref_error:=store.ref_get(s,REF_BUILDS,store.hex(build_select(entry.manifest,&p),&digits)) + switch ref_error { + case .None: + b,build_error:=build_get(s,built,context.temp_allocator) + if build_error==.Out_Of_Memory {return {},.Out_Of_Memory} + if build_error==.None && b.package_id!=p.id {build_error=.Corrupt} + if build_error!=.None { + problem_add(&resolved,{entry.id,{},.Build_Unreadable}) or_return + break + } + if reserve(&outputs,len(b.outputs))!=nil {return {},.Out_Of_Memory} + for o in b.outputs {outputs[o.id]=o} + case .Missing: + case .Out_Of_Memory: return {},.Out_Of_Memory + case .Not_Here_Yet,.Corrupt,.Invalid_Argument,.Io,.Unsupported_Layout: + problem_add(&resolved,{entry.id,{},.Build_Unreadable}) or_return + } + folder:=entry.folder!=""?fmt.tprintf("%s/",entry.folder):"" + for &a,at in p.assets { + info,known:=kind_info(a.kind) + if !used[at] || !known || !import_cached(a.kind) {continue} + repeated:=false + for &other in p.assets {repeated||=other.kind==a.kind && other.name==a.name && other.index +// with the name as safe_path gives it, and " ()" before the extension +// when an output of its kind and name has a lower index (repeated). folder: +// "" or ending with '/'. In the temp allocator. +@(private) +listed_path :: proc(folder,name:string,index:int,repeated:bool,extension:string) -> string { + if repeated {return fmt.tprintf("%s%s (%d)%s",folder,safe_path(name),index,extension)} + return fmt.tprintf("%s%s%s",folder,safe_path(name),extension) +} + +// A folder under another one: relative, with forward slashes, never out of it. +@(private) +folder_valid :: proc(folder:string) -> bool { + if folder=="" {return true} + if folder[0]=='/' || strings.contains_any(folder,"\\:") {return false} + rest:=folder + for part in strings.split_iterator(&rest,"/") { + if part=="" || part=="." || part==".." {return false} + } + return true +} + +@(private="file") +package_valid :: proc(p:^Package) -> bool { + if p.id=={} || p.settings!="" && !object_valid(p.settings) {return false} + for &a,at in p.assets { + if a.id=={} || a.id==p.id || !kind_valid(a.kind) || a.index<0 || a.index>int(max(i32)) || a.part && !import_cached(a.kind) {return false} + for &other in p.assets[:at] { + if other.id==a.id || other.kind==a.kind && other.index==a.index && other.name==a.name {return false} + } + for d in a.dependencies { + if d=={} {return false} + } + } + return true +} + +@(private="file") +build_valid :: proc(b:^Build) -> bool { + if b.package_id=={} {return false} + for o,at in b.outputs { + if o.id=={} || !kind_valid(o.kind) {return false} + for other in b.outputs[:at] { + if other.id==o.id {return false} + } + } + return true +} + +@(private="file") +packages_valid :: proc(list:^Packages) -> bool { + for &entry,at in list.packages { + if entry.id=={} || !folder_valid(entry.folder) {return false} + for &other in list.packages[:at] { + if other.id==entry.id {return false} + } + for id,i in entry.assets { + if id=={} {return false} + for other in entry.assets[:i] { + if other==id {return false} + } + } + } + return true +} + +// Four bytes a person can read, not all spaces: what the text of a kind keeps. +@(private="file") +kind_valid :: proc(kind:Fourcc) -> bool { + if kind==fourcc(" ") {return false} + for i in 0..<4 { + c:=u8(u32(kind)>>uint(8*i)) + if c<0x20 || c>0x7e {return false} + } + return true +} + +// One JSON object and nothing after it. +@(private="file") +object_valid :: proc(text:string) -> bool { + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + return serial.text_peek(&r)=='{' && serial.text_skip(&r) && serial.text_end(&r) +} + +@(private="file") +writer_text :: proc(w:^serial.Writer) -> (text:string,error:Error) { + if w.overflow>0 { + serial.writer_destroy(w) + return "",.Out_Of_Memory + } + return string(serial.writer_take(w)),.None +} + +@(private="file") +id_write :: proc(w:^serial.Writer,id:base.Asset_Id) { + serial.text_write_raw(w,"\"") + serial.text_write_asset_id(w,id) + serial.text_write_raw(w,"\"") +} + +@(private="file") +hash_write :: proc(w:^serial.Writer,hash:store.Hash) { + digits:[64]u8 + serial.text_write_raw(w,"\"") + serial.text_write_raw(w,store.hex(hash,&digits)) + serial.text_write_raw(w,"\"") +} + +@(private="file") +kind_write :: proc(w:^serial.Writer,kind:Fourcc) { + buffer:[4]u8 + serial.text_write_string(w,strings.trim_right(fourcc_text(kind,&buffer)," ")) +} + +@(private="file") +expect :: proc(r:^serial.Text_Reader,c:u8) -> Error { + return .None if serial.text_expect(r,c) else .Corrupt +} + +// The name of a field and its colon: a view valid until the next string. +@(private="file") +key :: proc(r:^serial.Text_Reader) -> (name:string,error:Error) { + text,ok:=serial.text_string(r) + if !ok || !serial.text_expect(r,':') {return "",.Corrupt} + return text,.None +} + +@(private="file") +clone :: proc(text:string,allocator:runtime.Allocator) -> (cloned:string,error:Error) { + if text=="" {return "",.None} + copied,no_room:=strings.clone(text,allocator) + if no_room!=nil {return "",.Out_Of_Memory} + return copied,.None +} + +@(private="file") +string_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (text:string,error:Error) { + value,ok:=serial.text_string(r) + if !ok {return "",.Corrupt} + return clone(value,allocator) +} + +@(private="file") +count_read :: proc(r:^serial.Text_Reader,most:u64) -> (value:u64,error:Error) { + negative,magnitude,ok:=serial.text_integer(r) + if !ok || negative || magnitude>most {return 0,.Corrupt} + return magnitude,.None +} + +@(private="file") +bool_read :: proc(r:^serial.Text_Reader) -> (value:bool,error:Error) { + if serial.text_keyword(r,"true") {return true,.None} + if serial.text_keyword(r,"false") {return false,.None} + return false,.Corrupt +} + +@(private="file") +id_read :: proc(r:^serial.Text_Reader) -> (id:base.Asset_Id,error:Error) { + text,ok:=serial.text_string(r) + if !ok {return {},.Corrupt} + parsed,valid:=base.asset_id_parse(text) + if !valid {return {},.Corrupt} + return parsed,.None +} + +@(private="file") +hash_read :: proc(r:^serial.Text_Reader) -> (hash:store.Hash,error:Error) { + text,ok:=serial.text_string(r) + if !ok {return {},.Corrupt} + parsed,valid:=store.parse_hex(text) + if !valid {return {},.Corrupt} + return parsed,.None +} + +@(private="file") +kind_read :: proc(r:^serial.Text_Reader) -> (kind:Fourcc,error:Error) { + text,ok:=serial.text_string(r) + if !ok || len(text)==0 || len(text)>4 {return 0,.Corrupt} + kind=fourcc(text) + if !kind_valid(kind) {return 0,.Corrupt} + return kind,.None +} + +@(private="file") +ids_read :: proc(r:^serial.Text_Reader,ids:^[dynamic]base.Asset_Id) -> Error { + expect(r,'[') or_return + for begun:=false;!serial.text_accept(r,']');begun=true { + if begun {expect(r,',') or_return} + id:=id_read(r) or_return + if _,no_room:=append(ids,id);no_room!=nil {return .Out_Of_Memory} + } + return .None +} + +@(private="file") +origin_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (origin:Package_Origin,error:Error) { + Field :: enum {Path,Folder,Importer} + seen:bit_set[Field] + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "path": field=.Path + case "folder": field=.Folder + case "importer": field=.Importer + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Path: origin.path=string_read(r,allocator) or_return + case .Folder: origin.folder=bool_read(r) or_return + case .Importer: origin.importer=string_read(r,allocator) or_return + } + } + if seen!={.Path,.Folder,.Importer} {return {},.Corrupt} + return origin,.None +} + +@(private="file") +asset_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (a:Package_Asset,error:Error) { + Field :: enum {Kind,Name,Index,Id,Part,Owner,Dependencies} + seen:bit_set[Field] + a.dependencies=make([dynamic]base.Asset_Id,allocator) + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "kind": field=.Kind + case "name": field=.Name + case "index": field=.Index + case "id": field=.Id + case "part": field=.Part + case "owner": field=.Owner + case "dependencies": field=.Dependencies + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Kind: a.kind=kind_read(r) or_return + case .Name: a.name=string_read(r,allocator) or_return + case .Index: a.index=int(count_read(r,u64(max(i32))) or_return) + case .Id: a.id=id_read(r) or_return + case .Part: a.part=bool_read(r) or_return + case .Owner: + a.owner=id_read(r) or_return + if a.owner=={} {return {},.Corrupt} + case .Dependencies: + ids_read(r,&a.dependencies) or_return + if len(a.dependencies)==0 {return {},.Corrupt} + } + } + if seen&{.Kind,.Name,.Index,.Id}!={.Kind,.Name,.Index,.Id} || .Part in seen && !a.part {return {},.Corrupt} + return a,.None +} + +@(private="file") +output_read :: proc(r:^serial.Text_Reader) -> (o:Build_Output,error:Error) { + Field :: enum {Id,Kind,Kind_Version,Blob,Thumbnail} + seen:bit_set[Field] + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "id": field=.Id + case "kind": field=.Kind + case "kind_version": field=.Kind_Version + case "blob": field=.Blob + case "thumbnail": field=.Thumbnail + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Id: o.id=id_read(r) or_return + case .Kind: o.kind=kind_read(r) or_return + case .Kind_Version: o.kind_version=u16(count_read(r,u64(max(u16))) or_return) + case .Blob: o.blob=hash_read(r) or_return + case .Thumbnail: + o.thumbnail=hash_read(r) or_return + if o.thumbnail=={} {return {},.Corrupt} + } + } + if seen&{.Id,.Kind,.Kind_Version,.Blob}!={.Id,.Kind,.Kind_Version,.Blob} {return {},.Corrupt} + return o,.None +} + +@(private="file") +entry_read :: proc(r:^serial.Text_Reader,allocator:runtime.Allocator) -> (entry:Project_Package,error:Error) { + Field :: enum {Id,Manifest,Folder,Assets} + seen:bit_set[Field] + entry.assets=make([dynamic]base.Asset_Id,allocator) + expect(r,'{') or_return + for first:=true;!serial.text_accept(r,'}');first=false { + if !first {expect(r,',') or_return} + name:=key(r) or_return + field:Field + switch name { + case "id": field=.Id + case "manifest": field=.Manifest + case "folder": field=.Folder + case "assets": field=.Assets + case: return {},.Corrupt + } + if field in seen {return {},.Corrupt} + seen+={field} + switch field { + case .Id: entry.id=id_read(r) or_return + case .Manifest: entry.manifest=hash_read(r) or_return + case .Folder: entry.folder=string_read(r,allocator) or_return + case .Assets: ids_read(r,&entry.assets) or_return + } + } + if seen!={.Id,.Manifest,.Folder,.Assets} {return {},.Corrupt} + return entry,.None +} + +// The text of the record the store keeps under hash, in the temp allocator. +@(private="file") +record_get :: proc(s:^store.Store,hash:store.Hash) -> (text:string,error:Error) { + size,size_error:=store.blob_size(s,hash) + if size_error!=.None {return "",store_error(size_error)} + if size>RECORD_BYTES {return "",.Too_Large} + bytes,no_room:=make([]u8,int(size),context.temp_allocator) + if no_room!=nil {return "",.Out_Of_Memory} + if read_error:=store.read(s,hash,0,bytes);read_error!=.None {return "",store_error(read_error)} + return string(bytes),.None +} + +@(private="file") +store_error :: proc(error:store.Error) -> Error { + switch error { + case .None: return .None + case .Missing,.Not_Here_Yet: return .Not_Found + case .Corrupt,.Unsupported_Layout: return .Corrupt + case .Invalid_Argument: return .Invalid + case .Out_Of_Memory: return .Out_Of_Memory + case .Io: return .IO + } + return .IO +} + +@(private="file") +problem_add :: proc(resolved:^Packages_Resolved,problem:Package_Problem) -> Error { + if _,no_room:=append(&resolved.problems,problem);no_room!=nil {return .Out_Of_Memory} + return .None +} diff --git a/asset/packages_test.odin b/asset/packages_test.odin new file mode 100644 index 000000000..48420fa8f --- /dev/null +++ b/asset/packages_test.odin @@ -0,0 +1,296 @@ +package asset + +import "base:runtime" +import "core:fmt" +import "core:os" +import "core:strings" +import "core:testing" +import "mirai:std/base" +import "mirai:store" + +@(private="file") +hash_of :: proc(text:string) -> store.Hash {return store.address_of_bytes(transmute([]u8)text)} + +@(private="file") +ids :: proc(list:..base.Asset_Id) -> [dynamic]base.Asset_Id { + out:=make([dynamic]base.Asset_Id,context.temp_allocator) + append(&out,..list) + return out +} + +@(private="file") +sample_package :: proc() -> Package { + p:=Package{id={1,1},name="crate \"kit\"/ü",origin={path="C:/Downloads/Kit\\crate.glb",importer="gltf"},identity=hash_of("crate"),settings=`{"lods": "None", "nested": {"a": [1, 2]}}`} + p.inputs=make([dynamic]string,context.temp_allocator) + append(&p.inputs,"crate.bin","textures/wood.png") + p.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&p.assets, + Package_Asset{kind=KIND_SCENE,name="crate",id={2,2},dependencies=ids({3,3},{6,6})}, + Package_Asset{kind=KIND_MESH,name="body",id={3,3},part=true,owner={2,2},dependencies=ids({4,4})}, + Package_Asset{kind=KIND_MATERIAL,name="wood",id={4,4},dependencies=ids({5,5})}, + Package_Asset{kind=KIND_TEXTURE,name="wood",id={5,5}}, + Package_Asset{kind=KIND_MESH,name="body",index=1,id={6,6}}, + Package_Asset{kind=KIND_TEXTURE,name="spare",id={7,7}}, + ) + return p +} + +@(private="file") +sample_build :: proc(p:^Package,blobs:[]store.Hash) -> Build { + b:=Build{package_id=p.id,key=hash_of("key"),final=true,encoders=0x0123456789abcdef} + b.outputs=make([dynamic]Build_Output,context.temp_allocator) + for &a,at in p.assets { + info,_:=kind_info(a.kind) + append(&b.outputs,Build_Output{id=a.id,kind=a.kind,kind_version=info.version,blob=blobs[at%len(blobs)],thumbnail=at==0?hash_of("thumb"):{}}) + } + return b +} + +@(private="file") +sample_list :: proc() -> Packages { + list:Packages + list.packages=make([dynamic]Project_Package,context.temp_allocator) + append(&list.packages,Project_Package{id={1,1},manifest=hash_of("m1"),folder="props/crate",assets=ids({3,3})},Project_Package{id={9,9},manifest=hash_of("m2"),assets=ids()}) + return list +} + +// Each record reads back as it was written, field by field, and writes the +// same text again; a record that could not do so is refused when written. +@(test) +package_records_read_back_what_they_write :: proc(t:^testing.T) { + p:=sample_package() + text,error:=package_text(&p,context.temp_allocator) + if !testing.expect_value(t,error,Error.None) {return} + back,parse_error:=package_parse(text,context.temp_allocator) + if !testing.expect_value(t,parse_error,Error.None) {return} + testing.expect(t,back.id==p.id && back.name==p.name && back.origin==p.origin && back.identity==p.identity && back.settings==p.settings && len(back.inputs)==2 && back.inputs[1]==p.inputs[1] && len(back.assets)==len(p.assets),"the manifest reads back") + for &a,at in back.assets { + want:=p.assets[at] + testing.expectf(t,a.kind==want.kind && a.name==want.name && a.index==want.index && a.id==want.id && a.part==want.part && a.owner==want.owner && len(a.dependencies)==len(want.dependencies),"asset %d reads back",at) + } + again,_:=package_text(&back,context.temp_allocator) + testing.expect_value(t,again,text) + + b:=sample_build(&p,{hash_of("a"),hash_of("b")}) + build,build_error:=build_text(&b,context.temp_allocator) + if !testing.expect_value(t,build_error,Error.None) {return} + built,built_error:=build_parse(build,context.temp_allocator) + if !testing.expect_value(t,built_error,Error.None) {return} + testing.expect(t,built.package_id==b.package_id && built.key==b.key && built.final && built.encoders==b.encoders && len(built.outputs)==len(b.outputs),"the build reads back") + for o,at in built.outputs {testing.expectf(t,o==b.outputs[at],"output %d reads back",at)} + rebuilt,_:=build_text(&built,context.temp_allocator) + testing.expect_value(t,rebuilt,build) + + list:=sample_list() + listed,list_error:=packages_text(&list,context.temp_allocator) + if !testing.expect_value(t,list_error,Error.None) {return} + read,read_error:=packages_parse(listed,context.temp_allocator) + if !testing.expect_value(t,read_error,Error.None) {return} + testing.expect(t,len(read.packages)==2 && read.packages[0].id==list.packages[0].id && read.packages[0].manifest==list.packages[0].manifest && read.packages[0].folder=="props/crate" && len(read.packages[0].assets)==1 && len(read.packages[1].assets)==0,"the list reads back") + + // Through the store and the file of a project. + folder:=base.test_folder(t,"packages-records");if folder=="" {return} + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + put,put_error:=package_put(&s,&p) + testing.expect(t,put_error==.None && put==hash_of(text),"a manifest is the blob of its text") + got,get_error:=package_get(&s,put,context.temp_allocator) + testing.expect(t,get_error==.None && got.id==p.id,"and reads back from the store") + _,missing:=package_get(&s,hash_of("nothing"),context.temp_allocator) + testing.expect_value(t,missing,Error.Not_Found) + empty,found,load_error:=packages_load(folder,context.temp_allocator) + testing.expect(t,load_error==.None && !found && len(empty.packages)==0,"a project without a list uses no package") + testing.expect_value(t,packages_save(folder,&list),Error.None) + loaded,saved,_:=packages_load(folder,context.temp_allocator) + testing.expect(t,saved && len(loaded.packages)==2,"the list of a project reads back from its file") + + // What would not read back is refused. + bad:=p + bad.settings=`{"open": ` + _,refused:=package_text(&bad,context.temp_allocator) + testing.expect_value(t,refused,Error.Invalid) + folded:=list + folded.packages[0].folder="../outside" + _,refused=packages_text(&folded,context.temp_allocator) + testing.expect_value(t,refused,Error.Invalid) +} + +// The listing of a project comes from its list, the manifests and the build +// each one's ref names, with the paths of the blobs. A package without a +// build is listed not built; a manifest the store lacks, a build that does not +// read or an asset the manifest lacks is a problem of the result; nothing is +// removed. +@(test) +packages_resolve_lists_the_blobs_of_what_a_project_uses :: proc(t:^testing.T) { + folder:=base.test_folder(t,"packages-resolve");if folder=="" {return} + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + mesh,_:=store.put_bytes(&s,transmute([]u8)string("mesh bytes")) + texture,_:=store.put_bytes(&s,transmute([]u8)string("texture bytes")) + p:=sample_package() + manifest,_:=package_put(&s,&p) + blobs:=[6]store.Hash{mesh,mesh,mesh,texture,mesh,texture} + b:=sample_build(&p,blobs[:]) + // The second mesh is of a version this program does not read. + b.outputs[4].kind_version+=1 + build,_:=build_put(&s,&b) + digits:[64]u8 + testing.expect_value(t,store.ref_set(&s,REF_BUILDS,store.hex(build_select(manifest,&p),&digits),build),store.Error.None) + + other:=Package{id={20,20},name="rock",identity=hash_of("rock")} + other.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&other.assets,Package_Asset{kind=KIND_TEXTURE,name="wood",id={21,21}}) + unbuilt,_:=package_put(&s,&other) + broken:=Package{id={30,30},name="broken",identity=hash_of("broken")} + broken.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&broken.assets,Package_Asset{kind=KIND_MESH,name="m",id={31,31}}) + broken_manifest,_:=package_put(&s,&broken) + not_a_build,_:=store.put_bytes(&s,transmute([]u8)string("not a build")) + testing.expect_value(t,store.ref_set(&s,REF_BUILDS,store.hex(build_select(broken_manifest,&broken),&digits),not_a_build),store.Error.None) + + list:Packages + list.packages=make([dynamic]Project_Package,context.temp_allocator) + append(&list.packages, + Project_Package{id=p.id,manifest=manifest,folder="kit",assets=ids({2,2},{99,99})}, + Project_Package{id=other.id,manifest=unbuilt,folder="kit",assets=ids()}, + Project_Package{id={40,40},manifest=hash_of("absent")}, + Project_Package{id=broken.id,manifest=broken_manifest}, + ) + resolved,error:=packages_resolve(&s,&list,context.temp_allocator) + if !testing.expect_value(t,error,Error.None) {return} + mesh_path,_:=store.blob_path(&s,mesh,context.temp_allocator) + texture_path,_:=store.blob_path(&s,texture,context.temp_allocator) + Want :: struct {id:base.Asset_Id,path,file:string,state:Listed_State} + wanted:=[]Want{ + {{3,3},"kit/body.mesh",mesh_path,.Built}, + {{5,5},"kit/wood (crate _kit__ü).tex",texture_path,.Built}, + {{6,6},"kit/body (1).mesh","",.Not_Built}, + {{21,21},"kit/wood (rock).tex","",.Not_Built}, + {{31,31},"m.mesh","",.Not_Built}, + } + testing.expectf(t,len(resolved.listed)==len(wanted),"the cache assets used, with what they depend on: %v",resolved.listed) + for w in wanted { + found:=false + for l in resolved.listed { + if l.id!=w.id {continue} + found=true + testing.expectf(t,l.path==w.path && l.file==w.file && l.state==w.state,"%v is %s at %s (%v), not %s at %s (%v)",w.id,w.path,w.file,w.state,l.path,l.file,l.state) + } + testing.expectf(t,found,"%v is listed",w.id) + } + Problem :: struct {package_id,asset:base.Asset_Id,code:Package_Problem_Code} + problems:=[]Problem{{p.id,{99,99},.Asset_Unknown},{{40,40},{},.Manifest_Missing},{broken.id,{},.Build_Unreadable}} + testing.expectf(t,len(resolved.problems)==len(problems),"the problems: %v",resolved.problems) + for want in problems { + seen:=false + for got in resolved.problems {seen||=got.package_id==want.package_id && got.asset==want.asset && got.code==want.code} + testing.expectf(t,seen,"%v is reported",want) + } + for blob in ([]store.Hash{mesh,texture,manifest,build,unbuilt,not_a_build}) {testing.expect(t,store.has(&s,blob),"resolving removes no blob")} + _,ref_error:=store.ref_get(&s,REF_BUILDS,store.hex(build_select(manifest,&p),&digits)) + testing.expect_value(t,ref_error,store.Error.None) + + // The name of a build changes with the versions of the kinds a program reads. + _,ok:=import_key(p.identity,make([]u64,IMPORT_KEY_VERSIONS+1,context.temp_allocator)) + testing.expect(t,!ok,"an import key takes at most IMPORT_KEY_VERSIONS versions") + first,_:=import_key(p.identity,{1,2}) + second,_:=import_key(p.identity,{1,3}) + testing.expect(t,first!=second && first!=build_select(manifest,&p),"keys of other versions differ") + buffer:[store.NAME_MAX]u8 + testing.expect_value(t,cache_namespace("textures",&buffer),"cache-textures") + testing.expect_value(t,cache_namespace(strings.repeat("x",store.NAME_MAX,context.temp_allocator),&buffer),"") +} + +// text with the first of before replaced; "" when text does not have it. +@(private="file") +swap :: proc(text,before,after:string) -> string { + at:=strings.index(text,before) + if at<0 {return ""} + return strings.concatenate({text[:at],after,text[at+len(before):]},context.temp_allocator) +} + +@(private="file") +failing_allocator :: proc() -> runtime.Allocator { + return {procedure=proc(data:rawptr,mode:runtime.Allocator_Mode,size,alignment:int,old_memory:rawptr,old_size:int,loc:=#caller_location) -> ([]byte,runtime.Allocator_Error) { + #partial switch mode { + case .Free,.Free_All: return nil,.None + } + return nil,.Out_Of_Memory + }} +} + +// No text makes a parser of the records stop the program: random bytes, +// every cut of a valid text, oversized, deep or contradictory text, and +// memory that runs out give an error. +@(test) +package_records_refuse_hostile_text :: proc(t:^testing.T) { + Parser :: enum {Package,Build,Packages} + parse :: proc(parser:Parser,text:string,allocator:runtime.Allocator) -> Error { + switch parser { + case .Package: _,error:=package_parse(text,allocator);return error + case .Build: _,error:=build_parse(text,allocator);return error + case .Packages: _,error:=packages_parse(text,allocator);return error + } + return .None + } + p:=sample_package() + b:=sample_build(&p,{hash_of("a")}) + list:=sample_list() + valid:[Parser]string + valid[.Package],_=package_text(&p,context.temp_allocator) + valid[.Build],_=build_text(&b,context.temp_allocator) + valid[.Packages],_=packages_text(&list,context.temp_allocator) + id:=`"00000000000000010000000000000001"` + Case :: struct {parser:Parser,text:string,want:Error} + cases:=[]Case{ + {.Package,"",.Corrupt}, + {.Package,strings.repeat(" ",RECORD_BYTES+1,context.temp_allocator),.Too_Large}, + {.Package,strings.concatenate({`{"version": 1, "settings": `,strings.repeat("[",200,context.temp_allocator)},context.temp_allocator),.Corrupt}, + {.Package,swap(valid[.Package],`"version": 1`,`"version": 2`),.Version}, + {.Package,swap(valid[.Package],`"id": "00000000000000070000000000000007"`,`"id": "00000000000000050000000000000005"`),.Corrupt}, + {.Package,swap(valid[.Package],`"version": 1`,`"version": 1, "version": 1`),.Corrupt}, + {.Package,swap(valid[.Package],`"version": 1`,`"version": 1, "extra": 1`),.Corrupt}, + {.Package,swap(valid[.Package],`"kind": "TEX"`,`"kind": "TOOLONG"`),.Corrupt}, + {.Package,swap(valid[.Package],`"index": 1`,`"index": -1`),.Corrupt}, + {.Package,swap(valid[.Package],`"part": true`,`"part": true, "part": false`),.Corrupt}, + {.Build,swap(valid[.Build],`"kind_version": `,`"kind_version": 70000`),.Corrupt}, + {.Build,swap(valid[.Build],`"final": true`,`"final": 1`),.Corrupt}, + {.Build,swap(valid[.Build],`"package": `,`"package": "00000000000000000000000000000000", "x": `),.Corrupt}, + {.Build,`{"version": 1}`,.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 1, "packages": [{{"id": %s, "manifest": "nothex", "folder": "", "assets": []}}]}}`,id),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 1, "packages": [{{"id": %s, "manifest": "%s", "folder": "a/../b", "assets": []}}]}}`,id,strings.repeat("0",64,context.temp_allocator)),.Corrupt}, + {.Packages,fmt.tprintf(`{{"version": 1, "packages": [{{"id": %s, "manifest": "%s", "folder": "", "assets": [%s, %s]}}]}}`,id,strings.repeat("0",64,context.temp_allocator),id,id),.Corrupt}, + {.Packages,`{"version": 1, "packages": []} {}`,.Corrupt}, + {.Packages,`{"version": 18446744073709551616, "packages": []}`,.Corrupt}, + } + for c,at in cases { + testing.expectf(t,at==0 || c.text!="","case %d changes a text it has",at) + got:=parse(c.parser,c.text,context.temp_allocator) + testing.expectf(t,got==c.want,"case %d (%v): %v, not %v",at,c.parser,got,c.want) + } + h:=base.Hostile{state=7} + random:=make([]u8,4096,context.temp_allocator) + for parser in Parser { + text:=strings.trim_right(valid[parser],"\n") + for cut in 0.. (outp return output,true } -// A folder under the folder of a source: relative, with forward slashes, -// and never out of it. -@(private="file") -folder_valid :: proc(folder:string) -> bool { - if folder=="" {return true} - if folder[0]=='/' || strings.contains_any(folder,"\\:") {return false} - rest:=folder - for part in strings.split_iterator(&rest,"/") { - if part=="" || part=="." || part==".." {return false} - } - return true -} - // The paths the registry lists the cache outputs of a sidecar at, under // listed (the path of the sidecar's folder in the registry, "" or ending // with '/'): / with the name as safe_path gives it, @@ -215,10 +202,9 @@ sidecar_paths :: proc(sidecar:Sidecar,listed:string) -> []string { for output,at in sidecar.outputs { info,known:=kind_info(output.kind) if !known || !import_cached(output.kind) {continue} - name:=safe_path(output.name) repeated:=false for other in sidecar.outputs {repeated||=other.kind==output.kind && other.name==output.name && other.index Date: Wed, 18 Nov 2026 18:15:35 -0300 Subject: [PATCH 005/123] store: completion publishes only checked bytes; kept tables and readers Completion claims the partial folder by renaming it, checks it under the claim and places from there; a full partial is completed by whoever finds it, and a failed placement gives the folder back. Range reads of partial blobs check each piece against its table. Temporary names come from system randomness. A complete blob keeps its piece table, and a Reader holds it open for range reads. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-store/SKILL.md | 21 +- store/blobs.odin | 120 +++++++++-- store/hostile_test.odin | 12 +- store/receive.odin | 303 ++++++++++++++++++++-------- store/space.odin | 29 ++- store/store.odin | 58 ++++-- store/store_test.odin | 130 ++++++++++++ 7 files changed, 532 insertions(+), 141 deletions(-) diff --git a/.agents/skills/mirai-store/SKILL.md b/.agents/skills/mirai-store/SKILL.md index a01dc4abe..234b31c84 100644 --- a/.agents/skills/mirai-store/SKILL.md +++ b/.agents/skills/mirai-store/SKILL.md @@ -17,7 +17,7 @@ derived data cache that use it decide what a ref means and what is live. - Address, pieces and hex: `store/address.odin`. - Root, layout version, errors and atomic writes: `store/store.odin`. -- Complete blobs (put, read, table, verify, remove): `store/blobs.odin`. +- Complete blobs, their kept tables and readers: `store/blobs.odin`. - Receiving a blob piece by piece and range reads of a partial one: `store/receive.odin`. - Refs and their names: `store/refs.odin`. - Usage and sweep: `store/space.odin`. @@ -35,6 +35,8 @@ derived data cache that use it decide what a ref means and what is live. - The address is a pure function of the bytes: the length and the hash of every fixed-size piece, one rule for every size. Piece hashes and addresses are keyed apart so none is taken for another. + A piece hash depends only on its piece and index, so pieces can be hashed in parallel; the + address only orders them. - Only the two private procedures of `address.odin` (the value of a piece, the address of a table) touch the hash primitive; everything else, tests included, goes through them or the API, so the primitive can be replaced in one place. @@ -42,10 +44,16 @@ derived data cache that use it decide what a ref means and what is live. Every file appears whole through a rename from the root's tmp folder; a receiver writes only its own piece and its own byte of the piece marks, so concurrent receivers of one blob agree. - A table is checked against the address before anything is kept, a piece against the table - before it is written, and the whole blob against its address before it becomes complete; a - piece that fails then is marked missing again. + before it is written and again before a range read returns it, and the whole blob against its + address before it becomes complete; a piece that fails then is marked missing again. +- Completion publishes only bytes it checked: whoever finds every piece marked claims the partial + folder by renaming it, checks it under the claim and places it from there. A completion that + cannot finish (a damaged piece, a placement refused) gives the folder back with nothing lost, + and the next receiver that finds it full retries. +- A complete blob's piece table is kept beside it, written once and checked against the address + whenever it is read, so serving a blob never rehashes it; a damaged kept table is rebuilt. - A complete blob is a plain file of its raw bytes, never written in place: readers open it by - path. + path or hold it open with a `Reader`. - The root carries its layout version; a root of another layout is refused, never read. - The store never decides what is live: sweep removes the complete blobs outside the set it is given, and partial blobs only go through remove. @@ -59,5 +67,6 @@ derived data cache that use it decide what a ref means and what is live. - `mirai check store` (it runs the allocation lint) and `mirai test store`. - Focal tests: `address_is_the_same_through_put_bytes_put_file_and_receiving` (sizes around the piece boundaries), `receiving_resumes_after_reopening_and_reads_the_pieces_it_has`, - `two_stores_on_one_root_put_and_receive_from_several_threads` and - `hostile_inputs_return_errors`. + `two_stores_on_one_root_put_and_receive_from_several_threads`, + `completion_publishes_only_the_bytes_it_checked`, + `a_failed_placement_loses_nothing_and_is_retried` and `hostile_inputs_return_errors`. diff --git a/store/blobs.odin b/store/blobs.odin index 61e5130c1..3ff905bca 100644 --- a/store/blobs.odin +++ b/store/blobs.odin @@ -1,6 +1,7 @@ package store import "base:runtime" +import "core:mem" import "core:os" // Complete blobs are plain files of their bytes at blobs//
, @@ -18,7 +19,8 @@ put_bytes :: proc(s:^Store,data:[]u8) -> (hash:Hash,error:Error) { os.remove(temp) return {},os_error(write_error) if write_error!=nil else .Io } - return hash,blob_place(s,temp,hash) + if error=blob_place(s,temp,hash);error!=.None {os.remove(temp)} + return hash,error } // Puts the file at path, read once piece by piece. @@ -39,7 +41,8 @@ put_file :: proc(s:^Store,path:string) -> (hash:Hash,error:Error) { os.remove(temp) return hash if error==.None else {},error } - return hash,blob_place(s,temp,hash) + if error=blob_place(s,temp,hash);error!=.None {os.remove(temp)} + return hash,error } // Whether the blob is complete here. @@ -90,28 +93,59 @@ read :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { return .Missing } -// The piece hashes of a complete blob, checked against its address on the way. +// The piece hashes of a complete blob: its kept table when that one matches +// the address and the size, otherwise hashed from the file once and kept. blob_table :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (pieces:[]Hash,error:Error) { - buffer:[PATH_BYTES]u8 - path:=blob_file(s,hash,&buffer) or_return - file,open_error:=os.open(path) - if open_error!=nil {return nil,os_error(open_error)} - defer os.close(file) - size,size_error:=os.file_size(file) - if size_error!=nil {return nil,os_error(size_error)} + r:=reader_open(s,hash) or_return + defer reader_close(&r) make_error:runtime.Allocator_Error - pieces,make_error=make([]Hash,piece_count(size),allocator) + pieces,make_error=make([]Hash,piece_count(r.size),allocator) if make_error!=nil {return nil,.Out_Of_Memory} + if table_load(s,hash,r.size,pieces)==.None {return pieces,.None} address:Hash - address,error=file_hash(file,size,nil,pieces) + address,error=file_hash(r.file,r.size,nil,pieces) if error==.None && address!=hash {error=.Corrupt} if error!=.None { delete(pieces,allocator) return nil,error } + table_keep(s,hash,r.size,pieces) return pieces,.None } +// A complete blob held open for many range reads; its owner closes it. +Reader :: struct { + file:^os.File, + size:i64, +} + +reader_open :: proc(s:^Store,hash:Hash) -> (r:Reader,error:Error) { + buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + open_error:os.Error + r.file,open_error=os.open(path) + if open_error!=nil {return {},os_error(open_error)} + size_error:os.Error + r.size,size_error=os.file_size(r.file) + if size_error!=nil { + reader_close(&r) + return {},os_error(size_error) + } + return r,.None +} + +// Fills buffer with the bytes of the blob from offset. +reader_read :: proc(r:^Reader,offset:i64,buffer:[]u8) -> Error { + if r.file==nil {return .Invalid_Argument} + if offset<0 || offset>r.size || i64(len(buffer))>r.size-offset {return .Invalid_Argument} + return read_exact(r.file,buffer,offset) +} + +reader_close :: proc(r:^Reader) { + if r.file!=nil {os.close(r.file)} + r^={} +} + // Corrupt when the file of a complete blob no longer has the bytes of its address. verify :: proc(s:^Store,hash:Hash) -> Error { buffer:[PATH_BYTES]u8 @@ -128,6 +162,7 @@ remove :: proc(s:^Store,hash:Hash) -> Error { path:=blob_file(s,hash,&buffer) or_return removed:=os.remove(path) if removed!=nil && removed!=.Not_Exist {return os_error(removed)} + table_drop(s,hash) discarded:=partial_discard(s,hash) if discarded!=.None && discarded!=.Missing {return discarded} return .Missing if removed!=nil && discarded==.Missing else .None @@ -140,7 +175,8 @@ blob_file :: proc(s:^Store,hash:Hash,buffer:^[PATH_BYTES]u8) -> (string,Error) { return join(buffer,s.root,"blobs",text[:2],text) } -// Renames a finished file of tmp into the blob's place; a blob already there stays. +// Renames a finished file into the blob's place; a blob already there stays +// and the file goes. A failed rename leaves the file where it was. @(private) blob_place :: proc(s:^Store,temp:string,hash:Hash) -> Error { buffer,folder_buffer:[PATH_BYTES]u8 @@ -149,21 +185,67 @@ blob_place :: proc(s:^Store,temp:string,hash:Hash) -> Error { text:=hex(hash,&name) folder_make(join(&folder_buffer,s.root,"blobs",text[:2]) or_return) or_return if os.is_file(path) || os.rename(temp,path)!=nil { + if !os.is_file(path) {return .Io} os.remove(temp) - return .None if os.is_file(path) else .Io } return .None } @(private) read_complete :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { - path_buffer:[PATH_BYTES]u8 - path:=blob_file(s,hash,&path_buffer) or_return + r:=reader_open(s,hash) or_return + defer reader_close(&r) + return reader_read(&r,offset,buffer) +} + +// The kept table of a blob: tables//
, its size as u64 +// little endian and its piece hashes, as the table of a partial blob. +@(private) +table_file :: proc(s:^Store,hash:Hash,buffer:^[PATH_BYTES]u8) -> (string,Error) { + name:[64]u8 + text:=hex(hash,&name) + return join(buffer,s.root,"tables",text[:2],text) +} + +// Fills pieces from the kept table; Corrupt when it does not match size and address. +@(private) +table_load :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { + buffer:[PATH_BYTES]u8 + path:=table_file(s,hash,&buffer) or_return file,open_error:=os.open(path) if open_error!=nil {return os_error(open_error)} defer os.close(file) - size,size_error:=os.file_size(file) + table_size,size_error:=os.file_size(file) if size_error!=nil {return os_error(size_error)} - if offset>size || i64(len(buffer))>size-offset {return .Invalid_Argument} - return read_exact(file,buffer,offset) + if len(pieces)!=piece_count(size) || table_size!=TABLE_HEADER+i64(len(pieces))*size_of(Hash) {return .Corrupt} + header:[TABLE_HEADER]u8 + read_exact(file,header[:],0) or_return + if header!=table_header(size) {return .Corrupt} + read_exact(file,mem.slice_to_bytes(pieces),TABLE_HEADER) or_return + return .None if table_address(size,pieces)==hash else .Corrupt +} + +// Keeps the table of a complete blob; one that cannot be written is hashed again when asked for. +@(private) +table_keep :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) { + buffer,folder_buffer:[PATH_BYTES]u8 + path,error:=table_file(s,hash,&buffer) + if error!=.None {return} + name:[64]u8 + folder,folder_error:=join(&folder_buffer,s.root,"tables",hex(hash,&name)[:2]) + if folder_error!=.None || folder_make(folder)!=.None {return} + header:=table_header(size) + file_write(s.root,path,header[:],mem.slice_to_bytes(pieces)) +} + +@(private) +table_drop :: proc(s:^Store,hash:Hash) { + buffer:[PATH_BYTES]u8 + if path,error:=table_file(s,hash,&buffer);error==.None {os.remove(path)} +} + +@(private) +table_header :: proc "contextless" (size:i64) -> (header:[TABLE_HEADER]u8) { + for i in 0..>(8*uint(i)))} + return } diff --git a/store/hostile_test.odin b/store/hostile_test.odin index bc24d4f65..b4332edd0 100644 --- a/store/hostile_test.odin +++ b/store/hostile_test.odin @@ -88,6 +88,16 @@ hostile_inputs_return_errors :: proc(t:^testing.T) { _,missing_error=receive_missing(&s,hash,context.temp_allocator) testing.expect_value(t,missing_error,Error.None) + // A marked piece whose bytes on disk do not match its table entry. + testing.expect_value(t,receive_piece(&s,hash,1,test_piece(data,1)),Error.None) + partial_data:=path_in(folder,"partial",hex_of(hash),"data") + on_disk,_:=os.read_entire_file(partial_data,context.temp_allocator) + on_disk[PIECE_BYTES+7]~=1 + testing.expect(t,os.write_entire_file(partial_data,on_disk)==nil) + testing.expect_value(t,read(&s,hash,PIECE_BYTES,random[:10]),Error.Not_Here_Yet) + missing_list,_:=receive_missing(&s,hash,context.temp_allocator) + testing.expect(t,len(missing_list)==len(pieces) && missing_list[1]==1,"a piece that does not match is missing again") + // Refs. for name in ([]string{"","."," ..","..","a/b","a\\b","A","con","com1.txt","x.","a b",strings.repeat("x",NAME_MAX+1,context.temp_allocator)}) { testing.expectf(t,ref_set(&s,"ns",name,hash)==.Invalid_Argument,"name %q",name) @@ -119,7 +129,7 @@ hostile_inputs_return_errors :: proc(t:^testing.T) { _,sweep_error:=sweep(&out,{good}) testing.expect_value(t,sweep_error,Error.Out_Of_Memory) other,_:=put_bytes(&s,data[:20]) - testing.expect_value(t,receive_begin(&out,address_of_bytes(data[:PIECE_BYTES+1]),PIECE_BYTES+1,test_table(data[:PIECE_BYTES+1])),Error.Out_Of_Memory) + testing.expect_value(t,receive_begin(&out,address_of_bytes(data[:PIECE_BYTES+1]),PIECE_BYTES+1,test_table(data[:PIECE_BYTES+1])),Error.None) testing.expect(t,has(&s,other)) _,open_error:=open(path_in(folder,"other"),failing_allocator()) testing.expect_value(t,open_error,Error.Out_Of_Memory) diff --git a/store/receive.odin b/store/receive.odin index 1a931556f..fa29de6c4 100644 --- a/store/receive.odin +++ b/store/receive.odin @@ -2,6 +2,7 @@ package store import "core:mem" import "core:os" +import "core:time" import "mirai:std/base" // A blob being received lives in partial/
/: `table` (its size as @@ -9,6 +10,10 @@ import "mirai:std/base" // offsets) and `have` (one byte per piece, 1 once it is in data). A folder // appears there whole through a rename; each writer writes only its own // piece and its own byte, so receivers in several processes need no lock. +// Whoever finds every piece marked completes the blob: it claims the folder +// by renaming it to partial/
.claimed, checks it there and publishes +// only what it checked; a folder it cannot complete goes back. The rename +// lets one claimer through, and the others keep using the claimed folder. @(private) TABLE_HEADER :: 8 @@ -23,9 +28,22 @@ Partial :: struct { @(private) Partial_File :: enum {Data,Have} +// A partial folder claimed for completion. Its strings point into its own +// buffers: it is not copied. +@(private) +Claim :: struct { + buffer:[PATH_BYTES]u8, + partial:string, // partial/
.claimed; "" when nothing is claimed + published,discard:bool, +} + +@(private) +CLAIMED :: ".claimed" + // Starts receiving a blob from its address and piece table; the table is // checked against the address before anything is kept. A blob already -// complete or being received is left as it is. +// complete or being received is left as it is; one being received whose +// pieces are all here is completed. receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { if size<0 || len(pieces)!=piece_count(size) {return .Invalid_Argument} if table_address(size,pieces)!=hash {return .Corrupt} @@ -38,15 +56,18 @@ receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { return error } existing,existing_error:=partial_open(s,hash,{.Data,.Have}) - partial_close(&existing) - if existing_error==.None {return .None} - if existing_error==.Corrupt {partial_discard(s,hash)} - folder_buffer,tmp_buffer:[PATH_BYTES]u8 + if existing_error==.None { + full,full_error:=partial_full(&existing) + partial_close(&existing) + if full_error!=.None {return full_error} + if !full {return .None} + if finished:=partial_finish(s,hash);finished!=.Corrupt {return finished} + } else if existing_error==.Corrupt { + partial_discard(s,hash) + } + folder_buffer,made_buffer:[PATH_BYTES]u8 folder:=partial_folder(s,hash,&folder_buffer) or_return - tmp:=join(&tmp_buffer,s.root,"tmp") or_return - made,made_error:=os.make_directory_temp(tmp,"receive-*",s.allocator) - if made_error!=nil {return os_error(made_error)} - defer delete(made,s.allocator) + made:=temp_folder(s.root,"receive-",&made_buffer) or_return error:=partial_write(made,size,pieces) if error==.None && os.rename(made,folder)!=nil { error=.None if os.is_directory(folder) else .Io @@ -56,7 +77,8 @@ receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { } // Keeps one received piece: checked against the table and written at its -// offset; a duplicate is harmless. The last missing piece completes the blob. +// offset; a duplicate is harmless. The piece that leaves none missing +// completes the blob. receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { p,error:=partial_open(s,hash,{.Data,.Have}) if error==.Missing {return complete_piece(s,hash,index,piece)} @@ -70,44 +92,28 @@ receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { mark:=[1]u8{1} if _,write_error:=os.write_at(p.have,mark[:],i64(index));write_error!=nil {return os_error(write_error)} full:=partial_full(&p) or_return + partial_close(&p) if !full {return .None} - return partial_complete(s,hash,&p) + return partial_finish(s,hash) } -// The indices of the pieces a blob being received still lacks; none for a complete blob. +// The indices of the pieces a blob being received still lacks; none for a +// complete blob. One with none missing is completed first. receive_missing :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (missing:[]int,error:Error) { - if has(s,hash) {return nil,.None} - p:Partial - p,error=partial_open(s,hash,{.Have}) - if error==.Missing && has(s,hash) {return nil,.None} - if error!=.None {return nil,error} - defer partial_close(&p) - absent:=0 - marks:[4096]u8 - for at:=0;at0 || attempt==1 {return missing,error} + delete(missing,allocator) + finished:=partial_finish(s,hash) + if finished!=.None {return nil,finished} } - return list[:n],.None + return nil,.None } @(private) @@ -120,8 +126,23 @@ partial_folder :: proc(s:^Store,hash:Hash,buffer:^[PATH_BYTES]u8) -> (string,Err // sizes of the table and of have are checked. @(private) partial_open :: proc(s:^Store,hash:Hash,files:bit_set[Partial_File]) -> (p:Partial,error:Error) { - folder_buffer,buffer:[PATH_BYTES]u8 - folder:=partial_folder(s,hash,&folder_buffer) or_return + folder_buffer:[PATH_BYTES]u8 + p,error=partial_open_at(partial_folder(s,hash,&folder_buffer) or_return,files) + if error!=.Missing {return} + return partial_open_at(claimed_folder(s,hash,&folder_buffer) or_return,files) +} + +@(private) +claimed_folder :: proc(s:^Store,hash:Hash,buffer:^[PATH_BYTES]u8) -> (string,Error) { + name:[64+len(CLAIMED)]u8 + hex(hash,(^[64]u8)(&name)) + copy(name[64:],CLAIMED) + return join(buffer,s.root,"partial",string(name[:])) +} + +@(private) +partial_open_at :: proc(folder:string,files:bit_set[Partial_File]) -> (p:Partial,error:Error) { + buffer:[PATH_BYTES]u8 open_error:os.Error p.table,open_error=os.open(join(&buffer,folder,"table") or_return) if open_error!=nil {return {},os_error(open_error)} @@ -190,50 +211,148 @@ partial_full :: proc(p:^Partial) -> (full:bool,error:Error) { return true,.None } -// Checks every piece of data against the table and the table against the -// address. A piece that does not match is marked missing again; a table that -// does not match discards the blob. When all match, data becomes the blob. +// Completes the blob of a partial folder whose pieces are all marked: +// claims it, checks it and publishes it, or gives it back. Nothing to claim +// is no error: another receiver holds it or it is gone. +@(private) +partial_finish :: proc(s:^Store,hash:Hash) -> Error { + c:Claim + claimed:=partial_claim(s,hash,&c) + if claimed==.Missing {return .None} + if claimed!=.None {return claimed} + defer claim_end(s,hash,&c) + complete:=claim_verify(hash,&c) or_return + if !complete {return .None} + return claim_publish(s,hash,&c) +} + +// Renames the partial folder to its claimed name. A claim older than +// TMP_STALE was left by a process that ended: it goes back first. @(private) -partial_complete :: proc(s:^Store,hash:Hash,p:^Partial) -> Error { +partial_claim :: proc(s:^Store,hash:Hash,c:^Claim) -> Error { + from_buffer:[PATH_BYTES]u8 + from:=partial_folder(s,hash,&from_buffer) or_return + claimed:=claimed_folder(s,hash,&c.buffer) or_return + if !os.is_directory(from) { + if at,time_error:=os.modification_time_by_path(claimed);time_error==nil && time.since(at)>TMP_STALE { + os.rename(claimed,from) + } + if !os.is_directory(from) {return .Missing} + } + now:=time.now() + os.change_times(from,now,now) + if os.rename(from,claimed)!=nil {return .Missing} + c.partial=claimed + return .None +} + +// Whether every piece of the claimed folder is marked and matches the table, +// and the table the address. A piece that does not match is marked missing +// again; a table that does not match is Corrupt and the folder is discarded. +@(private) +claim_verify :: proc(hash:Hash,c:^Claim) -> (complete:bool,error:Error) { + p:Partial + p,error=partial_open_at(c.partial,{.Data,.Have}) + if error==.Corrupt {c.discard=true} + if error!=.None {return false,error} + defer partial_close(&p) + full:=partial_full(&p) or_return + if !full {return false,.None} Checking :: struct { buffer:[PIECE_BYTES]u8, p:^Partial, damaged:bool, } - checking:=Checking{p=p} + checking:=Checking{p=&p} address:=address_value(p.size,proc(user:rawptr,index:int) -> (Hash,Error) { c:=(^Checking)(user) - entry:Hash - if error:=read_exact(c.p.table,entry[:],TABLE_HEADER+i64(index)*size_of(Hash));error!=.None {return {},error} - piece:=c.buffer[:piece_size(c.p.size,index)] - got:=read_exact(c.p.data,piece,i64(index)*PIECE_BYTES) - if got!=.None && got!=.Corrupt {return {},got} - if got==.Corrupt || piece_value(piece,index)!=entry { - c.damaged=true - unmark:=[1]u8{0} - if _,write_error:=os.write_at(c.p.have,unmark[:],i64(index));write_error!=nil {return {},os_error(write_error)} - } - return entry,.None + entry,_,good,error:=partial_piece(c.p,index,&c.buffer) + if !good {c.damaged=true} + return entry,error },&checking) or_return - damaged:=checking.damaged data_size,size_error:=os.file_size(p.data) - if size_error!=nil {return os_error(size_error)} + if size_error!=nil {return false,os_error(size_error)} if address!=hash || data_size>p.size { - partial_close(p) - partial_discard(s,hash) - return .Corrupt + c.discard=true + return false,.Corrupt + } + return !checking.damaged,.None +} + +// Moves the checked data into the blob's place, and its table to the kept tables. +@(private) +claim_publish :: proc(s:^Store,hash:Hash,c:^Claim) -> Error { + buffer,table_buffer,kept_buffer,folder_buffer:[PATH_BYTES]u8 + blob_place(s,join(&buffer,c.partial,"data") or_return,hash) or_return + c.published=true + name:[64]u8 + if folder_make(join(&folder_buffer,s.root,"tables",hex(hash,&name)[:2]) or_return)==.None { + os.rename(join(&table_buffer,c.partial,"table") or_return,table_file(s,hash,&kept_buffer) or_return) } - if damaged {return .None} - partial_close(p) - folder_buffer,data_buffer:[PATH_BYTES]u8 - folder:=partial_folder(s,hash,&folder_buffer) or_return - data:=join(&data_buffer,folder,"data") or_return - if !os.is_file(data) {return .None if has(s,hash) else .Missing} - blob_place(s,data,hash) or_return - partial_discard(s,hash) return .None } +// Gives a claimed folder that was neither published nor discarded back to +// partial/, and removes one that was. +@(private) +claim_end :: proc(s:^Store,hash:Hash,c:^Claim) { + if c.partial=="" {return} + if !c.published && !c.discard { + back_buffer:[PATH_BYTES]u8 + if back,error:=partial_folder(s,hash,&back_buffer);error==.None && os.rename(c.partial,back)==nil { + c.partial="" + return + } + } + folder_discard(s,c.partial) + c.partial="" +} + +// Reads piece index of a partial blob and checks it against its table entry; +// a piece that does not match has its mark cleared. +@(private) +partial_piece :: proc(p:^Partial,index:int,buffer:^[PIECE_BYTES]u8) -> (entry:Hash,piece:[]u8,good:bool,error:Error) { + read_exact(p.table,entry[:],TABLE_HEADER+i64(index)*size_of(Hash)) or_return + piece=buffer[:piece_size(p.size,index)] + got:=read_exact(p.data,piece,i64(index)*PIECE_BYTES) + if got!=.None && got!=.Corrupt {return entry,nil,false,got} + if got==.None && piece_value(piece,index)==entry {return entry,piece,true,.None} + unmark:=[1]u8{0} + if _,write_error:=os.write_at(p.have,unmark[:],i64(index));write_error!=nil {return entry,nil,false,os_error(write_error)} + return entry,nil,false,.None +} + +// The indices of the pieces not marked. +@(private) +partial_absent :: proc(p:^Partial,allocator:=context.allocator) -> (missing:[]int,error:Error) { + absent:=0 + marks:[4096]u8 + for at:=0;at Error { // Moves the partial folder of a blob into tmp and removes it there. @(private) partial_discard :: proc(s:^Store,hash:Hash) -> Error { - folder_buffer,tmp_buffer,moved_buffer:[PATH_BYTES]u8 - folder:=partial_folder(s,hash,&folder_buffer) or_return + folder_buffer:[PATH_BYTES]u8 + return folder_discard(s,partial_folder(s,hash,&folder_buffer) or_return) +} + +@(private) +folder_discard :: proc(s:^Store,folder:string) -> Error { + made_buffer,moved_buffer:[PATH_BYTES]u8 if !os.is_directory(folder) {return .Missing} - tmp:=join(&tmp_buffer,s.root,"tmp") or_return - made,made_error:=os.make_directory_temp(tmp,"discard-*",s.allocator) - if made_error!=nil {return os_error(made_error)} - defer delete(made,s.allocator) + made:=temp_folder(s.root,"discard-",&made_buffer) or_return moved:=join(&moved_buffer,made,"blob") or_return renamed:=os.rename(folder,moved) base.remove_all(made) @@ -261,7 +382,7 @@ partial_discard :: proc(s:^Store,hash:Hash) -> Error { return .None } -// The range of a blob being received, when all its pieces are here. +// The range of a blob being received, when all its pieces are here and match the table. @(private) read_partial :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { p:=partial_open(s,hash,{.Data,.Have}) or_return @@ -270,13 +391,17 @@ read_partial :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { if len(buffer)==0 {return .None} first:=int(offset/PIECE_BYTES) last:=int((offset+i64(len(buffer))-1)/PIECE_BYTES) - marks:[4096]u8 - for at:=first;at<=last;at+=len(marks) { - chunk:=marks[:min(len(marks),last+1-at)] - read_exact(p.have,chunk,i64(at)) or_return - for mark in chunk { - if mark==0 {return .Not_Here_Yet} - } + piece_buffer:[PIECE_BYTES]u8=--- + for index in first..=last { + mark:[1]u8 + read_exact(p.have,mark[:],i64(index)) or_return + if mark[0]==0 {return .Not_Here_Yet} + _,piece,good:=partial_piece(&p,index,&piece_buffer) or_return + if !good {return .Not_Here_Yet} + start:=i64(index)*PIECE_BYTES + from:=max(offset,start) + to:=min(offset+i64(len(buffer)),start+i64(len(piece))) + copy(buffer[from-offset:to-offset],piece[from-start:to-start]) } - return read_exact(p.data,buffer,offset) + return .None } diff --git a/store/space.odin b/store/space.odin index 585932a60..250f964cb 100644 --- a/store/space.odin +++ b/store/space.odin @@ -18,7 +18,7 @@ Usage :: struct { // What the complete blobs take. usage :: proc(s:^Store) -> (total:Usage,error:Error) { - error=blobs_walk(s,proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { + error=blobs_walk(s,"blobs",proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { total:=(^Usage)(user) total.bytes+=size total.blobs+=1 @@ -27,10 +27,10 @@ usage :: proc(s:^Store) -> (total:Usage,error:Error) { return } -// Removes every complete blob whose address is not in live, and the stale -// files of tmp: what it freed. The caller decides what is live. +// Removes every complete blob whose address is not in live, with its kept +// table, and the stale files of tmp: what it freed. The caller decides what is live. sweep :: proc(s:^Store,live:[]Hash) -> (freed:Usage,error:Error) { - Sweep :: struct {live:[]Hash,freed:Usage} + Sweep :: struct {live:[]Hash,freed:Usage,tables:bool} sorted,make_error:=make([]Hash,len(live),s.allocator) if make_error!=nil {return {},.Out_Of_Memory} defer delete(sorted,s.allocator) @@ -40,7 +40,7 @@ sweep :: proc(s:^Store,live:[]Hash) -> (freed:Usage,error:Error) { return mem.compare(a[:],b[:])<0 }) state:=Sweep{live=sorted} - error=blobs_walk(s,proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { + visit:=proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { state:=(^Sweep)(user) _,found:=slice.binary_search_by(state.live,hash,proc(a,b:Hash) -> slice.Ordering { a,b:=a,b @@ -50,10 +50,16 @@ sweep :: proc(s:^Store,live:[]Hash) -> (freed:Usage,error:Error) { removed:=os.remove(path) if removed==.Not_Exist {return .None} if removed!=nil {return os_error(removed)} - state.freed.bytes+=size - state.freed.blobs+=1 + if !state.tables { + state.freed.bytes+=size + state.freed.blobs+=1 + } return .None - },&state) + } + error=blobs_walk(s,"blobs",visit,&state) + if error!=.None {return state.freed,error} + state.tables=true + error=blobs_walk(s,"tables",visit,&state) if error!=.None {return state.freed,error} buffer:[PATH_BYTES]u8 tmp:=join(&buffer,s.root,"tmp") or_return @@ -67,11 +73,12 @@ sweep :: proc(s:^Store,live:[]Hash) -> (freed:Usage,error:Error) { return state.freed,.None } -// Visits every complete blob: a file of blobs// named by its address. +// Visits every file of // named by an address: the complete +// blobs, or the kept tables. @(private) -blobs_walk :: proc(s:^Store,visit:proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error,user:rawptr) -> Error { +blobs_walk :: proc(s:^Store,kind:string,visit:proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error,user:rawptr) -> Error { buffer:[PATH_BYTES]u8 - blobs:=join(&buffer,s.root,"blobs") or_return + blobs:=join(&buffer,s.root,kind) or_return folders,folders_error:=os.read_all_directory_by_path(blobs,s.allocator) if folders_error!=nil {return os_error(folders_error)} defer os.file_info_slice_delete(folders,s.allocator) diff --git a/store/store.odin b/store/store.odin index 2d59c2f2b..34e7ff91a 100644 --- a/store/store.odin +++ b/store/store.odin @@ -1,6 +1,7 @@ package store import "base:runtime" +import "core:crypto" import "core:os" // A content-addressed store of the machine: blobs named by their address and @@ -41,7 +42,7 @@ open :: proc(root:string,allocator:=context.allocator) -> (s:Store,error:Error) } buffer:[PATH_BYTES]u8 folder_make(trimmed) or_return - for folder in ([4]string{"blobs","partial","refs","tmp"}) { + for folder in ([5]string{"blobs","tables","partial","refs","tmp"}) { folder_make(join(&buffer,trimmed,folder) or_return) or_return } layout_check(trimmed) or_return @@ -127,13 +128,16 @@ os_error :: proc(error:os.Error) -> Error { return .Io } -// Writes data to a file of the root's tmp folder and renames it to path, +// Writes chunks to a file of the root's tmp folder and renames it to path, // which then holds all of it or what it held before. @(private) -file_write :: proc(root:string,path:string,data:[]u8) -> Error { +file_write :: proc(root:string,path:string,chunks:..[]u8) -> Error { temp_buffer:[PATH_BYTES]u8 temp,file:=temp_create(root,&temp_buffer) or_return - _,write_error:=os.write(file,data) + write_error:os.Error + for chunk in chunks { + if write_error==nil {_,write_error=os.write(file,chunk)} + } sync_error:=os.sync(file) os.close(file) if write_error!=nil || sync_error!=nil { @@ -150,16 +154,40 @@ file_write :: proc(root:string,path:string,data:[]u8) -> Error { // A new file of the root's tmp folder, open to write; its path is in buffer. @(private) temp_create :: proc(root:string,buffer:^[PATH_BYTES]u8) -> (path:string,file:^os.File,error:Error) { - folder_buffer:[PATH_BYTES]u8 - folder:=join(&folder_buffer,root,"tmp") or_return - created,create_error:=os.create_temp_file(folder,"write-*") - if create_error!=nil {return "",nil,os_error(create_error)} - name:=os.name(created) - if len(name)>len(buffer) { - os.close(created) - os.remove(name) - return "",nil,.Invalid_Argument + for _ in 0.. (path:string,error:Error) { + for _ in 0.., random from the system so that processes sharing +// the root do not collide. +@(private) +temp_name :: proc(root,prefix:string,buffer:^[PATH_BYTES]u8) -> (string,Error) { + random:Hash + crypto.rand_bytes(random[:16]) + text:[64]u8 + name:[96]u8 + if len(prefix)>len(name)-32 {return "",.Invalid_Argument} + copy(name[:],prefix) + copy(name[len(prefix):],hex(random,&text)[:32]) + return join(buffer,root,"tmp",string(name[:len(prefix)+32])) } diff --git a/store/store_test.odin b/store/store_test.odin index 80fbe046a..955728b62 100644 --- a/store/store_test.odin +++ b/store/store_test.odin @@ -209,3 +209,133 @@ sweep_keeps_exactly_the_live_blobs :: proc(t:^testing.T) { _,missing_error=receive_missing(&s,partial_hash,context.temp_allocator) testing.expect_value(t,missing_error,Error.Missing) } + +// Receives every piece but piece 0, whose bytes and mark are then written as +// a receiver killed before completing would leave them. +@(private) +test_full_unfinished :: proc(t:^testing.T,s:^Store,data:[]u8) -> Hash { + hash:=address_of_bytes(data) + pieces:=test_table(data) + testing.expect_value(t,receive_begin(s,hash,i64(len(data)),pieces),Error.None) + for index in 1.. Date: Wed, 18 Nov 2026 18:18:33 -0300 Subject: [PATCH 006/123] net: bulk streams find the path rate and leave game channels on time - Receivers acknowledge at once every ACK_EVERY datagrams with reliable chunks. - SENT_WINDOW 4096 and reliable windows up to 16384 chunks, sized by the channel's queue_bytes; chunk records get room of their own. Acks go as gamma-coded runs (TRANSPORT_VERSION 9). - Congestion control starts low, doubles until a queue appears, then grows additively and cuts below the delivered rate; send_rate, min_send_rate, packets_per_update and Channel_Desc.window are removed. - Unreliable game channels go before reliable ones, reliable game channels take chunks in turn, and lost chunks are resent from a queue instead of a scan. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-networking/SKILL.md | 6 +- .../mirai-networking/references/networking.md | 46 +++-- editor/collab.odin | 2 +- editor/collab_edit_test.odin | 2 +- net/channel.odin | 143 +++++++++---- net/congestion_test.odin | 2 +- net/connection_test.odin | 85 +++++--- net/flow_test.odin | 195 +++++++++++++++++- net/host.odin | 21 +- net/packing_test.odin | 6 +- net/peer.odin | 162 +++++++++++---- net/replication.odin | 2 +- net/scope.odin | 2 +- net/worker_test.odin | 2 +- plugins/multiplayer/runtime/journal.odin | 2 +- 15 files changed, 522 insertions(+), 156 deletions(-) diff --git a/.agents/skills/mirai-networking/SKILL.md b/.agents/skills/mirai-networking/SKILL.md index b7a66d232..a2498377c 100644 --- a/.agents/skills/mirai-networking/SKILL.md +++ b/.agents/skills/mirai-networking/SKILL.md @@ -153,6 +153,9 @@ In `plugins/multiplayer/runtime`: buffer; every index, type, mask and length is validated before use. A datagram that cannot be interpreted closes the connection with `Protocol_Error`; it is never half-applied over another connection. +- **A connection finds its rate by itself.** Nothing in `Host_Desc` starts or caps it; a reliable + channel holds in flight what its `queue_bytes` holds, and the game channels of a connection stay + on time next to a stream on it. - **Datagram acks and losses are final.** Late datagrams are not acked and only allow [the partial delivery described in connections](references/networking.md#connections). Do not extend that exception without reviewing channels, links and baselines. @@ -251,7 +254,8 @@ In `plugins/multiplayer/runtime`: - `mirai test net [test...]`, `mirai test std/serial` (bitstream) and `mirai test plugins/multiplayer/runtime`. Run the `*_test.odin` of the touched area: `address`, - `connection`, `secure`, `invite`, `congestion`, `flow` (reliable window, a streamed part), + `connection`, `secure`, `invite`, `congestion`, `flow` (reliable window, a streamed part, a bulk + stream on a 100 Mbit/s link, a shared link, ack runs), `conditioner`, `replication`, `descriptor_boundary`, `large_state`, `collection`, `references`, `convergence`, `grid`, `scope`, `authority`, `interpolation`, `clock`, `worker`, `baseline` (pools and shared baselines), diff --git a/.agents/skills/mirai-networking/references/networking.md b/.agents/skills/mirai-networking/references/networking.md index 2bfe94cec..f5d1d91ee 100644 --- a/.agents/skills/mirai-networking/references/networking.md +++ b/.agents/skills/mirai-networking/references/networking.md @@ -124,10 +124,14 @@ like `replication.hysteresis`) and the values it admits. - **Amplification.** The Challenge is smaller than the padded request, and the server does X25519 exchanges (two per accepted client, one per refused proof) only for whoever returned the cookie, `handshakes_per_update` at a time per update. -- **Acks.** Each data datagram carries its sequence, the last one received and a variable number of - ack bits: from what the other end might not have resolved when reading the last ack this end - received (`ack_known`), capped by `ACK_WINDOW`. Everything the sender has in flight arrives this - way with its real fate, also after a receiver stall. +- **Acks.** Each data datagram carries its sequence, the last one received and which of the ones + before it arrived, from what the other end might not have resolved when reading the last ack this + end received (`ack_known`), capped by `ACK_WINDOW`, as runs of arrived and missing datagrams in + Elias gamma code: a stream with few losses acks thousands of datagrams in a few bytes. Everything + the sender has in flight arrives this way with its real fate, also after a receiver stall. + - A receiver acks within `ACK_DELAY`, or with what it sends anyway, and at once after + `ACK_EVERY` datagrams that carried reliable chunks: a stream is acked as it arrives, and a + connection without one sends no datagram more for it. - A datagram older than the last accepted is not acked: when a later one is acked, the fate of the earlier ones is final and what was in a lost one is resent. - If it arrives late (at most `LATE_WINDOW` behind) and had not been received, only what is safe @@ -141,11 +145,18 @@ like `replication.hysteresis`) and the values it admits. no ack. - **Send limit**: a token bucket with the connection's rate that stores at most one tick of rate (`send_burst`), so a network half that updates more often than the tick spreads the datagrams. - The object budget of each send is the lesser of `budget_bytes` and the tokens: replication - follows the rate. -- **Congestion**: AIMD driven by the queue (the mean RTT of the interval above the lowest of the - recent window), not by loss, to cut back before the path buffer overflows; a link that loses at - random without a queue keeps its rate, which never drops below `min_send_rate`. + An update sends while the tokens last, up to `SENT_WINDOW` datagrams in flight. A reliable + channel with a datagram of chunks ready sends at every update the tokens allow; the rest goes once + a tick. The object budget of each send is the lesser of `budget_bytes` and the tokens: + replication follows the rate. +- **Congestion**: the connection finds the rate of its path; nothing in `Host_Desc` sets it. It + starts at `CC_START_RATE` and doubles each interval that wanted more, until the queue appears + (the mean RTT of the interval above the lowest of the recent window, beyond `CC_QUEUE_DELAY` and + two datagrams at the rate). Then it drops below what the path delivered and grows by the same + bytes per second for every connection (at most a quarter each interval), so connections on one + path converge to equal shares and the queue stays short for the game traffic on it. Losses cut it + only with a queue or past `CC_OVERRUN_SHARE`; a link that loses at random keeps its rate, which + never drops below `CC_MIN_RATE`. - **Close**: `host_disconnect` tells the other end with its reason; silence closes with `Timeout`, and an authentic datagram that cannot be interpreted, with `Protocol_Error`. The reasons for rejection and closing are `Disconnect_Reason` (`host.odin`). @@ -153,7 +164,10 @@ like `replication.hysteresis`) and the values it admits. ## Channels Game channels in `Host_Desc.channels`, the same at both ends; `message_send` and the `Message` -command. `budget` limits the bytes of a channel per update. +command. `budget` limits the bytes of a channel per update. In each datagram the library's channels +go first, then the unreliable and sequenced game channels, then the reliable game channels one +chunk each in turn, starting with a different one each datagram: a channel that streams leaves the +others their share. - `Unreliable` is sent once; `Sequenced` too, and a message that is still waiting is replaced by the next one. @@ -167,10 +181,11 @@ command. `budget` limits the bytes of a channel per update. receiver joins; the library's own reliable channel takes only whole messages. - **Window.** The chunks a reliable channel keeps in flight grow by what the path acknowledges (`channel_window`, each congestion interval), so they double each round trip while they hold the - channel back. Twice what the connection's rate sends in a round trip bounds them, - so congestion control and `send_rate` decide what a stream takes, and `Channel_Desc.window`, a - power of two the same at both ends, is the most a channel holds and receives ahead. A loss does - not shrink the window: its resend waits a round trip with the chunks after it acknowledged. + channel back. Twice what the connection's rate sends in a round trip bounds them, so congestion + control decides what a stream takes, and the channel's `queue_bytes` sets the most it holds in + flight and receives ahead (`channel_chunks`, a power of two up to `MAX_WINDOW`): a stream that + fills a path holds twice what the path carries in a round trip. A loss does not shrink the + window: its resend waits a round trip with the chunks after it acknowledged, ahead of new ones. - While objects wait for the update, the reliable game channels leave half of each datagram to them: a part that streams does not starve the state. - The library channels count their queue the same way: `event_send` of a `Reliable` event and the @@ -449,7 +464,8 @@ The client predicts with copies of the world and interpolates with `replica_blen - Congestion control does not tell a queue from a path that lengthens: a route change to a higher RTT cuts the rate until the window forgets the previous minimum. A buffer smaller than the queue - threshold overflows without the queue being seen, and then the rate does not drop. + threshold overflows without the queue being seen, and the rate drops only once the losses pass + `CC_OVERRUN_SHARE`. - A late datagram is not acked: even though its reliable chunks were already delivered, the sender resends them (the receiver discards them), and its objects, unreliable messages and time are lost. diff --git a/editor/collab.odin b/editor/collab.odin index 96de86b86..782e85cdf 100644 --- a/editor/collab.odin +++ b/editor/collab.odin @@ -45,7 +45,7 @@ COLLAB_PRESENCE_INTERVAL :: 1.0/12 COLLAB_EDIT_INTERVAL :: 1.0/30 COLLAB_PRESENCE_SEEN :: 3.0 // seconds COLLAB_MESSAGE_BYTES :: 1<<30 // what one message may hold: a peer that sends more is cut off -COLLAB_CHANNELS :: []wire.Channel_Desc{{kind=.Reliable,queue_bytes=256<<10,window=512},{kind=.Unreliable}} +COLLAB_CHANNELS :: []wire.Channel_Desc{{kind=.Reliable,queue_bytes=256<<10},{kind=.Unreliable}} COLLAB_HELLO_TIMEOUT :: 10.0 COLLAB_WORKER_INTERVAL :: 2*time.Millisecond // between updates of the network half COLLAB_PLAYING :: u8(1) // presence: inside a Play or a Simulate of its own diff --git a/editor/collab_edit_test.odin b/editor/collab_edit_test.odin index cf3cea68d..270fe47f3 100644 --- a/editor/collab_edit_test.odin +++ b/editor/collab_edit_test.odin @@ -278,7 +278,7 @@ a_file_as_large_as_a_session_carries_crosses_as_one_message :: proc(t:^testing.T defer free(host);defer free(guest) host.role,guest.role=.Host,.Guest for st,i in ([2]^Collab_State{host,guest}) { - desc:=wire.Host_Desc{mode=i==0?.Server:.Client,transport=wire.sim_endpoint_transport(&network,i),protocol_id=0x4d495241,seed=u64(7+i),channels=COLLAB_CHANNELS,send_rate=1<<30,packets_per_update=64} + desc:=wire.Host_Desc{mode=i==0?.Server:.Client,transport=wire.sim_endpoint_transport(&network,i),protocol_id=0x4d495241,seed=u64(7+i),channels=COLLAB_CHANNELS} testing.expect_value(t,wire.host_init(&st.host,desc),wire.Error.None) } defer {wire.host_destroy(&host.host);wire.host_destroy(&guest.host)} diff --git a/net/channel.odin b/net/channel.odin index 1dcfb4305..819b235e1 100644 --- a/net/channel.odin +++ b/net/channel.odin @@ -14,27 +14,36 @@ import "mirai:std/serial" // the queue, so one can exceed the datagram and the queue: // message_send streams it, and it arrives in pieces of at // most PIECE_BYTES (Command.more). The chunks in flight grow -// to what the path carries (channel_window), up to window. +// to what the path carries (channel_window), up to what the +// queue holds. Channel_Kind :: enum u8 {Unreliable,Sequenced,Reliable} Channel_Desc :: struct { kind:Channel_Kind, budget:int, // bytes of this channel per update of a connection; 0 has no limit - queue_bytes:int, // messages waiting to be sent, default 32 KiB + queue_bytes:int, // messages waiting to be sent, default 32 KiB; Reliable: also the most it keeps in flight and holds ahead on receiving, up to 4 MiB max_message_bytes:int, // Unreliable and Sequenced: at most MAX_UNRELIABLE_BYTES, the default. Reliable has no limit - window:int, // Reliable: chunks of 256 bytes in flight at most, and held ahead on receiving; a power of two up to 4096, default 64 } MAX_UNRELIABLE_BYTES :: 1024 PIECE_BYTES :: 16<<10 -@(private) MAX_WINDOW :: 4096 +@(private) MAX_WINDOW :: 16384 @(private) CHUNK_BYTES :: 256 -@(private) CHUNK_WINDOW :: 64 // chunks a reliable channel starts with in flight, and its default window +@(private) CHUNK_WINDOW :: 64 // chunks a reliable channel starts with in flight, and the fewest it holds @(private) CHANNEL_SYSTEM :: 6 @(private) CHANNEL_EVENTS :: 7 @(private) CHANNEL_ORDER :: [MAX_CHANNELS]int{CHANNEL_SYSTEM,CHANNEL_EVENTS,0,1,2,3,4,5} +// The chunks a reliable channel holds in flight and ahead on receiving: what its queue holds, as a +// power of two so that sequence numbers map to slots across their wraparound. +@(private) +channel_chunks :: proc "contextless" (queue_bytes:int) -> int { + chunks:=CHUNK_WINDOW + for chunks bool { return false } +// A reliable channel has at least a datagram of chunks it may send now: it streams, and goes out as +// the rate allows instead of once a tick. +@(private) +channels_streaming :: proc(p:^Peer) -> bool { + for &c in p.channels { + if !c.active || c.kind!=.Reliable {continue} + room:=c.window-int(u16(c.send_next-c.send_base)) + if c.ready+min(room,c.queue.used/CHUNK_BYTES)>=DATAGRAM_CHUNKS {return true} + } + return false +} +@(private) DATAGRAM_CHUNKS :: PACKET_BYTES/(CHUNK_BYTES+8) // chunks a datagram carries + // Cuts queued messages into chunks while the window has room and the bytes // of the next chunk are in the queue. @(private) @@ -193,8 +221,8 @@ channel_fill :: proc(c:^Channel) { // The chunks in flight a reliable channel may keep. Each byte the path acknowledged lets one more be in // flight, so the window doubles each round trip while it holds the channel back; a loss does not shrink // it, since a resend waits a round trip with the chunks after it acknowledged. What the path carries -// bounds it through the connection's rate, which congestion control sets within send_rate: never above -// twice what that rate sends in a round trip, nor below CHUNK_WINDOW, nor above what the channel holds. +// bounds it through the connection's rate, which congestion control finds: never above twice what that +// rate sends in a round trip, nor below CHUNK_WINDOW, nor above what the channel holds. @(private) channel_window :: proc "contextless" (c:^Channel,rtt,rate:f64) { acked:=c.acked @@ -204,36 +232,20 @@ channel_window :: proc "contextless" (c:^Channel,rtt,rate:f64) { c.window=max(int(chunks),min(CHUNK_WINDOW,len(c.chunks))) } -// yield: bits the reliable game channels leave in the datagram for the objects that wait, so a part that -// streams does not starve the state. +// The library's channels go first, then the unreliable and sequenced game channels, which are small and +// late when they wait; then the reliable game channels take one chunk each in turn, starting with the +// next one at every datagram, so a channel that streams leaves the others their share. yield: bits the +// reliable game channels leave in the datagram for the objects that wait, so a part that streams does +// not starve the state. @(private) channels_write :: proc(h:^Host,p:^Peer,w:^serial.Bit_Writer,reserve,yield:int) -> (wrote:bool) { for index in CHANNEL_ORDER { c:=&p.channels[index] if !c.active {continue} if c.kind==.Reliable { + if index0 && c.spent>0 && c.spent+length>c.budget {break} - if !record_push(p,{kind=.Chunk,flags=u8(index),aux=seq}) {break} - serial.bit_write(w,1,1) - serial.bit_write(w,u64(index),3) - serial.bit_write(w,u64(seq),16) - serial.bit_write_bool(w,chunk.more) - serial.bit_write(w,u64(length),9) - serial.bit_write_bytes(w,c.chunk_data[slot*CHUNK_BYTES:][:length]) - chunk.state=.Flight - c.ready-=1 - c.spent+=length - wrote=true - } + for chunk_write(p,c,index,w,reserve) {wrote=true} } else { for c.queue.used>0 { length:=ring_peek_size(&c.queue) @@ -251,10 +263,56 @@ channels_write :: proc(h:^Host,p:^Peer,w:^serial.Bit_Writer,reserve,yield:int) - } } } + for index in 0.. bool { + if c.ready==0 {return false} + seq:=c.resend_count>0?c.resend[c.resend_head]:c.fresh + slot:=int(seq)%len(c.chunks) + chunk:=&c.chunks[slot] + length:=int(chunk.length) + if serial.bit_writer_remaining(w)<30+length*8+left {return false} + if c.budget>0 && c.spent>0 && c.spent+length>c.budget {return false} + if !record_push(p,{kind=.Chunk,flags=u8(index),aux=seq}) {return false} + if c.resend_count>0 { + c.resend_head=(c.resend_head+1)%len(c.resend) + c.resend_count-=1 + } else { + c.fresh+=1 + } + serial.bit_write(w,1,1) + serial.bit_write(w,u64(index),3) + serial.bit_write(w,u64(seq),16) + serial.bit_write_bool(w,chunk.more) + serial.bit_write(w,u64(length),9) + serial.bit_write_bytes(w,c.chunk_data[slot*CHUNK_BYTES:][:length]) + chunk.state=.Flight + c.ready-=1 + c.spent+=length + return true +} + @(private) channel_resolve :: proc(p:^Peer,record:Record,received:bool) { c:=&p.channels[record.flags] @@ -265,6 +323,8 @@ channel_resolve :: proc(p:^Peer,record:Record,received:bool) { if chunk.state!=.Flight {return} if !received { chunk.state=.Ready + c.resend[(c.resend_head+c.resend_count)%len(c.resend)]=seq + c.resend_count+=1 c.ready+=1 p.stats.chunks_resent+=1 return @@ -286,6 +346,8 @@ channel_resolve :: proc(p:^Peer,record:Record,received:bool) { // with it by waiting for the resend. channels_read :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,late:=false) -> bool { scratch:=h.back.scratch[:] + chunked:=false + defer if chunked {p.ack_chunks+=1} for serial.bit_read_bool(r) { index:=int(serial.bit_read(r,3)) c:=&p.channels[index] @@ -302,6 +364,7 @@ channels_read :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,late:=false) -> bool more:=serial.bit_read_bool(r) length:=int(serial.bit_read(r,9)) if length>CHUNK_BYTES || !serial.bit_read_bytes(r,scratch[:length]) {return false} + chunked=true if late && index>=USER_CHANNELS {continue} size:=len(c.recv_chunks) if int(u16(seq-c.recv_base))>=size {continue} diff --git a/net/congestion_test.odin b/net/congestion_test.odin index 105de2a91..33af71eac 100644 --- a/net/congestion_test.odin +++ b/net/congestion_test.odin @@ -10,7 +10,7 @@ FLOOD_CHANNELS :: []Channel_Desc{{kind=.Unreliable,queue_bytes=16<<10}} @(test) congestion_control_follows_a_link_that_narrows_and_recovers :: proc(t:^testing.T) { w:Test_World;test_world_init(&w,2,seed=5);defer test_world_destroy(&w) - server:=test_host(&w,{mode=.Server,channels=FLOOD_CHANNELS,send_rate=256<<10,packets_per_update=8}) + server:=test_host(&w,{mode=.Server,channels=FLOOD_CHANNELS,}) client:=test_host(&w,{mode=.Client,channels=FLOOD_CHANNELS}) on_server,on_client:=test_connect(&w,server,client) testing.expect(t,on_server!=0 && on_client!=0) diff --git a/net/connection_test.odin b/net/connection_test.odin index 1fed2eb92..068ea27cb 100644 --- a/net/connection_test.odin +++ b/net/connection_test.odin @@ -446,38 +446,6 @@ a_reliable_message_larger_than_its_queue_streams_and_arrives_in_pieces :: proc(t testing.expect(t,false,"the message and the one after it arrive") } -@(test) -send_rate_limits_the_bytes_a_connection_puts_on_the_wire :: proc(t:^testing.T) { - w:Test_World;test_world_init(&w,2);defer test_world_destroy(&w) - channels:=[]Channel_Desc{{kind=.Reliable,queue_bytes=1<<20}} - server:=test_host(&w,{mode=.Server,channels=channels,command_bytes=4<<20}) - client:=test_host(&w,{mode=.Client,channels=channels,send_rate=20000,command_bytes=4<<20,frame_bytes=1<<20}) - _,on_client:=test_connect(&w,server,client) - blob:=make([]u8,200000);defer delete(blob) - for &b,i in blob {b=u8(i)} - testing.expect_value(t,message_send(client,on_client,0,blob),Error.None) - test_step(&w,30) - host_poll(client);host_poll(server) - before,_:=connection_stats(client,on_client) - arrived:=false - steps,got:=0,0 - for !arrived && steps<1200 { - test_step(&w) - steps+=1 - host_poll(client) - for command in host_poll(server) { - if command.kind!=.Message {continue} - got+=len(command.data) - arrived=!command.more && got==len(blob) - } - } - testing.expect(t,arrived) - after,_:=connection_stats(client,on_client) - rate:=f64(after.bytes_sent-before.bytes_sent)/(f64(steps)*TEST_STEP) - testing.expect(t,rate<22000 && rate>15000) - testing.expect(t,after.send_rate>10000 && after.send_rate<24000) -} - // A transport that keeps the last datagram of a connection its host sent. @(private="file") Capture :: struct {inner:Transport,last:[PACKET_BYTES]u8,size:int} @@ -720,3 +688,56 @@ udp_sockets_carry_a_connection_on_localhost :: proc(t:^testing.T) { stats,_:=connection_stats(client,on_client) testing.expect(t,stats.packets_received>0) } + +// Prompt acknowledgements answer only a stream of reliable chunks: an idle connection sends its +// keepalives and their answers, and one that carries game traffic both ways (a message every tick, a +// small reliable one every few ticks and a larger one now and then) sends about a datagram a tick. +@(test) +idle_and_game_connections_send_no_more_datagrams_for_prompt_acknowledgements :: proc(t:^testing.T) { + channels:=[]Channel_Desc{{kind=.Reliable},{kind=.Unreliable}} + w:Test_World;test_world_init(&w,2);defer test_world_destroy(&w) + server:=test_host(&w,{mode=.Server,channels=channels}) + client:=test_host(&w,{mode=.Client,channels=channels}) + on_server,on_client:=test_connect(&w,server,client) + if !testing.expect(t,on_server!=0 && on_client!=0) {return} + sim_endpoint_conditions(&w.sim,0,{latency=0.02}) + sim_endpoint_conditions(&w.sim,1,{latency=0.02}) + SECONDS :: 10 + for game in ([]bool{false,true}) { + host_poll(server);host_poll(client) + before_server,_:=connection_stats(server,on_server) + before_client,_:=connection_stats(client,on_client) + next:=w.now + tick:=0 + small:[200]u8 + large:[2000]u8 + start:=w.now + for w.now-start=next { + next+=1.0/60 + tick+=1 + for pair in ([][2]^Host{{server,client},{client,server}}) { + connection:=pair[0]==server?on_server:on_client + message_send(pair[0],connection,1,small[:8]) + if tick%6==0 {message_send(pair[0],connection,0,small[:])} + if tick%60==0 {message_send(pair[0],connection,0,large[:])} + } + } + host_update(server,w.now);host_update(client,w.now) + for command in host_poll(server) {testing.expect(t,command.kind!=.Disconnected)} + for command in host_poll(client) {testing.expect(t,command.kind!=.Disconnected)} + } + after_server,_:=connection_stats(server,on_server) + after_client,_:=connection_stats(client,on_client) + per_second:=[2]f64{f64(after_server.packets_sent-before_server.packets_sent)/SECONDS,f64(after_client.packets_sent-before_client.packets_sent)/SECONDS} + for rate in per_second { + if game { + testing.expectf(t,rate<=1.05*60+1,"with game traffic %.1f datagrams a second, more than one a tick",rate) + } else { + testing.expectf(t,rate<1/ACK_DELAY,"idle %.1f datagrams a second: acknowledgements come sooner than ACK_DELAY",rate) + } + } + } +} diff --git a/net/flow_test.odin b/net/flow_test.odin index c7a2fa3be..43cf73ae4 100644 --- a/net/flow_test.odin +++ b/net/flow_test.odin @@ -24,9 +24,9 @@ a_large_reliable_part_streams_at_what_the_path_carries_and_the_state_keeps_its_r for c in cases { tp:Test_Protocol;test_protocol_init(&tp);defer test_protocol_destroy(&tp) w:Test_World;test_world_init(&w,2,seed=21,packets=16384);defer test_world_destroy(&w) - channels:=[]Channel_Desc{{kind=.Reliable,queue_bytes=1<<20,window=1024}} + channels:=[]Channel_Desc{{kind=.Reliable,queue_bytes=1<<20}} replication:=Replication_Desc{max_objects=128,baseline_interval=-1} - server:=test_host(&w,{mode=.Server,protocol=&tp.protocol,channels=channels,replication=replication,send_rate=FLOW_CAPACITY,packets_per_update=64}) + server:=test_host(&w,{mode=.Server,protocol=&tp.protocol,channels=channels,replication=replication}) client:=test_host(&w,{mode=.Client,protocol=&tp.protocol,channels=channels,replication=replication}) on_server,on_client:=test_connect(&w,server,client) if !testing.expect(t,on_server!=0 && on_client!=0) {return} @@ -88,7 +88,7 @@ a_large_reliable_part_streams_at_what_the_path_carries_and_the_state_keeps_its_r @(test) hostile_acknowledgements_are_refused_and_never_push_a_window_past_its_bounds :: proc(t:^testing.T) { w:Test_World;test_world_init(&w,2);defer test_world_destroy(&w) - channels:=[]Channel_Desc{{kind=.Reliable,window=1024}} + channels:=[]Channel_Desc{{kind=.Reliable,queue_bytes=256<<10}} server:=test_host(&w,{mode=.Server,channels=channels}) client:=test_host(&w,{mode=.Client,channels=channels}) on_server,_:=test_connect(&w,server,client) @@ -133,9 +133,196 @@ hostile_acknowledgements_are_refused_and_never_push_a_window_past_its_bounds :: serial.bit_writer_init(&wr,buffer[:]) serial.bit_write_bool(&wr,true) serial.bit_write(&wr,u64(p.send_seq+10),16) - serial.bit_write(&wr,0,8) + serial.bit_write(&wr,0,ACK_COUNT_BITS) serial.bit_write(&wr,0,8) rd:serial.Bit_Reader serial.bit_reader_init(&rd,serial.bit_writer_flush(&wr)) testing.expect(t,!peer_receive(server,p,&rd,p.recv_counter+1),"an acknowledgement of what was never sent is malformed") } + +// Which datagrams arrived reads back as written, whatever the pattern, and a record whose runs are +// empty, too long, past the count or cut short is refused; random bits never read past what they say. +@(test) +acknowledgement_runs_read_back_and_refuse_what_does_not_add_up :: proc(t:^testing.T) { + state:=u64(17) + Pattern :: enum {All,None,Alternate,Sparse_Loss,Random} + for pattern in Pattern { + for count in ([]int{0,1,63,64,65,700,ACK_WINDOW}) { + bits:Ack_Bits + for i in 0.. (result:Bulk_Result,ok:bool) { + blob:=make([]u8,BULK_PART);defer delete(blob) + w:Test_World;test_world_init(&w,1+clients,seed=31,packets=1<<16);defer test_world_destroy(&w) + server:=test_host(&w,{mode=.Server,channels=BULK_CHANNELS}) + hosts:[2]^Host + on_server:[2]Connection + for i in 0..=BULK_WARM + for i in 0..=next_game { + next_game+=1.0/60 + for i in 0..0.6*BULK_LINK,"%v: %.0f B/s arrive over a link of %.0f",c,r.delivered[0],BULK_LINK) + testing.expectf(t,r.drop_share<0.03,"%v: the link's queue drops %.1f%% of the datagrams",c,r.drop_share*100) + testing.expectf(t,r.game_got>=int(0.95*(1-f64(c.loss))*f64(r.game_sent)),"%v: %d of %d game messages arrive",c,r.game_got,r.game_sent) + testing.expectf(t,r.game_mean<0.03 && r.game_worst<0.08,"%v: game messages take %.0f ms more than the latency on average, %.0f at worst",c,r.game_mean*1000,r.game_worst*1000) + } +} + +// Two connections that stream over the same link each get a fair share of it. +@(test) +two_bulk_streams_on_one_link_share_it :: proc(t:^testing.T) { + r,ok:=bulk_stream(t,0.04,0,2,16) + if !ok {return} + fmt.printfln("shared: %.2f and %.2f MB/s",r.delivered[0]/1e6,r.delivered[1]/1e6) + total:=r.delivered[0]+r.delivered[1] + testing.expectf(t,total>0.6*BULK_LINK,"together %.0f B/s over a link of %.0f",total,BULK_LINK) + testing.expectf(t,min(r.delivered[0],r.delivered[1])>0.3*total,"shares of %.0f and %.0f B/s",r.delivered[0],r.delivered[1]) +} diff --git a/net/host.odin b/net/host.odin index 74b823459..a19545407 100644 --- a/net/host.odin +++ b/net/host.odin @@ -73,9 +73,6 @@ Host_Desc :: struct { max_connections:int, // server; default 32. A client has one channels:[]Channel_Desc, // at most USER_CHANNELS tick_rate:int, // Hz, default 60 - send_rate:int, // bytes per second and connection at most, default 256 KiB - min_send_rate:int, // bytes per second below which congestion control does not go, default send_rate/16 - packets_per_update:int, // datagrams per connection in one update, default 4 timeout:f64, // seconds of silence that close a connection, default 10 connect_timeout:f64, // default 5 keepalive:f64, // seconds, default 0.5 @@ -201,6 +198,7 @@ Host :: struct { failed:bool, refused:Refusal, // why host_init returned Invalid history:int, // ticks of Replication_Desc.history_seconds at tick_rate + records:int, // of a connection: those of Replication_Desc.records and one per chunk its reliable channels hold // Simulation half. staging:^Frame, @@ -294,9 +292,6 @@ host_init :: proc(h:^Host,desc:Host_Desc) -> Error { if !(d.replication.history_seconds<=HISTORY_MAX_SECONDS) {return refuse(h,"replication.history_seconds","at most 2 seconds; 0 or less takes 0.5")} if d.replication.history_seconds<=0 {d.replication.history_seconds=HISTORY_SECONDS} h.history=history_ticks(d.replication.history_seconds,d.tick_rate) - if d.send_rate<=0 {d.send_rate=256<<10} - if d.min_send_rate<=0 {d.min_send_rate=max(d.send_rate/16,PACKET_BYTES*4)} - if d.packets_per_update<=0 {d.packets_per_update=4} if d.timeout<=0 {d.timeout=10} if d.connect_timeout<=0 {d.connect_timeout=5} if d.keepalive<=0 {d.keepalive=0.5} @@ -307,11 +302,8 @@ host_init :: proc(h:^Host,desc:Host_Desc) -> Error { if c.queue_bytes<=0 {c.queue_bytes=32<<10} if c.kind==.Reliable { c.max_message_bytes=max(int) - if c.window<=0 {c.window=CHUNK_WINDOW} - if c.window>MAX_WINDOW || c.window&(c.window-1)!=0 {return refuse(h,"channels.window","a power of two up to 4096; 0 takes 64")} } else { c.max_message_bytes=c.max_message_bytes<=0?MAX_UNRELIABLE_BYTES:min(c.max_message_bytes,MAX_UNRELIABLE_BYTES) - c.window=0 } h.channels[i]=c } @@ -319,7 +311,7 @@ host_init :: proc(h:^Host,desc:Host_Desc) -> Error { d.channels=nil if d.protocol!=nil { if !d.protocol.sealed {return refuse(h,"protocol","sealed by protocol_seal")} - h.channels[CHANNEL_SYSTEM]={kind=.Reliable,queue_bytes=64<<10,max_message_bytes=RELIABLE_EVENT_MAX_BYTES+16,window=CHUNK_WINDOW} + h.channels[CHANNEL_SYSTEM]={kind=.Reliable,queue_bytes=64<<10,max_message_bytes=RELIABLE_EVENT_MAX_BYTES+16} h.channels[CHANNEL_EVENTS]={kind=.Unreliable,queue_bytes=16<<10,max_message_bytes=EVENT_MAX_BYTES+16} } h.start=time.tick_now() @@ -354,10 +346,15 @@ host_init :: proc(h:^Host,desc:Host_Desc) -> Error { // What one datagram can turn into, so a full command buffer stops the reading before it and not halfway. b.reserve_bytes=PACKET_BYTES*2 b.reserve_count=PACKET_BYTES*8/15 + if d.replication.records<=0 {d.replication.records=d.protocol!=nil?4096:0} + // Every chunk in flight takes a record: the reliable channels bring room of their own. + h.records=d.replication.records for i in 0.. (bits:Ack_Bits,count:u64) { - count=serial.bit_read(r,8) - for i:=u64(0);i int { + i:=at + for i>offset))) + i+=run + if run<64-int(offset) {break} + } + return min(i,count)-at +} +// Not ok: a count past ACK_WINDOW, or runs that are empty, too long or do not add up to the count. +@(private) +ack_bits_read :: proc(r:^serial.Bit_Reader) -> (bits:Ack_Bits,count:u64,ok:bool) { + count=serial.bit_read(r,ACK_COUNT_BITS) + if r.error || count>ACK_WINDOW {return} + if count==0 {return bits,0,true} + set:=serial.bit_read_bool(r) + for at:=u64(0);at=ACK_COUNT_BITS {return} + } + run:=u64(1)<count-at {return} + if set { + for i:=at;i (counter:u64,ok: @(private) peer_allocate :: proc(h:^Host,p:^Peer) { p.index=u32(mem.ptr_sub(p,&h.back.peers[0])) - p.records=host_make(h,Record,h.desc.replication.records) + p.records=host_make(h,Record,h.records) for desc,i in h.channels { if i=USER_CHANNELS && h.desc.protocol!=nil) {channel_allocate(h,&p.channels[i],desc)} } @@ -245,13 +303,14 @@ peer_reset :: proc(h:^Host,p:^Peer,address:Address) { p.client_salt,p.server_salt,p.cookie,p.stamp=0,0,0,0 p.started,p.last_send,p.last_receive,p.next_handshake=now,now,now,now p.send_counter,p.recv_counter,p.send_seq,p.sent_tail,p.recv_bits,p.late_bits,p.ack_known=0,0,0,0,{},0,0 - p.recv_any,p.ack_pending,p.has_rtt=false,false,false + p.recv_any,p.ack_pending,p.has_rtt,p.ack_chunks=false,false,false,0 p.record_head,p.record_tail=0,0 p.rtt,p.rtt_variance,p.jitter=0,0,0 - p.rate=f64(h.desc.send_rate) + p.rate,p.starting=CC_START_RATE,true p.tokens,p.token_time=send_burst(h,p.rate),now p.min_rtt,p.window_min,p.window_time=0,0,now - p.cc_time,p.cc_rtt,p.cc_acked,p.cc_lost,p.cc_samples,p.cc_limited=now,0,0,0,0,false + p.cc_time,p.cc_rtt,p.cc_samples,p.cc_acked,p.cc_lost,p.cc_limited=now,0,0,0,0,false + p.delivered,p.cc_delivered,p.channel_turn=0,0,0 p.rate_time,p.rate_sent,p.rate_received=now,0,0 p.stats={} p.secure={} @@ -350,29 +409,40 @@ peer_service :: proc(h:^Host,p:^Peer) { } } -@(private) CC_QUEUE_DELAY :: 0.06 // s of RTT above the smallest that mean a queue is building on the path -@(private) CC_LOSS_SHARE :: 0.02 // share of an interval's datagrams lost that, with that queue, is congestion -@(private) CC_WINDOW :: 60.0 // s over which the smallest RTT is remembered: longer than any queue lasts +@(private) CC_START_RATE :: 256<<10 // bytes per second a connection starts at +@(private) CC_MIN_RATE :: 16<<10 // bytes per second congestion control does not go below +@(private) CC_MAX_RATE :: 1<<30 +@(private) CC_GROWTH :: 2<<20 // bytes per second gained each second that wants more and finds no queue +@(private) CC_QUEUE_DELAY :: 0.02 // s of RTT above the smallest, besides two datagrams at the rate, that is a queue +@(private) CC_LOSS_SHARE :: 0.02 // share of an interval's datagrams lost that, with that queue, is congestion +@(private) CC_OVERRUN_SHARE :: 0.2 // share lost that is congestion with or without a queue: a buffer overflows +@(private) CC_WINDOW :: 60.0 // s over which the smallest RTT is remembered: longer than any queue lasts -// Additive increase, multiplicative decrease of the byte rate, steered by the queue the connection builds -// on its path: the RTT of an interval above the smallest one of the last CC_WINDOW seconds. Losses alone do -// not cut the rate, so a link that drops datagrams at random keeps it. A queue cuts it by a tenth, before -// the buffer overflows; a queue with losses, by a quarter. An interval without a queue that wanted to send -// more than the rate allowed adds send_rate/128. +// The rate starts low and doubles each interval that wanted more than it allowed, until the path shows a +// queue: the mean RTT of an interval above the smallest of the last CC_WINDOW seconds. Then the rate +// drops below what the path delivered in that interval, by a fifth, or by more with the losses of a +// buffer that overflows, and from there grows by the same CC_GROWTH a second while it wants more and +// finds no queue: connections that share a path converge to equal shares, and the queue stays short for +// what else travels on it. A slow path grows by at most a quarter of its rate each interval. Losses +// without a queue do not cut the rate, so a link that drops datagrams at random keeps it. @(private) congestion_update :: proc(h:^Host,p:^Peer) { now:=h.back.now - if now-p.cc_time0 && p.cc_rtt/f64(p.cc_samples)-p.min_rtt>CC_QUEUE_DELAY - if queued { - congested:=lost>0 && f64(lost)>CC_LOSS_SHARE*f64(acked+lost) - p.rate=max(p.rate*(congested?0.75:0.9),floor) + delivered:=f64(p.delivered-p.cc_delivered)/elapsed + // Two datagrams at the rate are not a queue: a slow link takes that long to carry them. + queued:=p.cc_samples>0 && p.cc_rtt/f64(p.cc_samples)-p.min_rtt>CC_QUEUE_DELAY+2*PACKET_BYTES/p.rate + overrun:=acked+lost>=16 && f64(lost)>CC_OVERRUN_SHARE*f64(acked+lost) + if queued || overrun { + congested:=overrun || lost>0 && f64(lost)>CC_LOSS_SHARE*f64(acked+lost) + p.rate=clamp(min(p.rate,delivered)*(congested?0.7:0.8),CC_MIN_RATE,CC_MAX_RATE) + p.starting=false p.stats.send_limit_cuts+=1 } else if p.cc_limited { - p.rate=min(p.rate+ceiling/128,ceiling) + p.rate=min(p.starting?p.rate*2:p.rate+min(CC_GROWTH*min(elapsed,1),p.rate/4),CC_MAX_RATE) } if now-p.window_time>CC_WINDOW { if p.window_min>0 {p.min_rtt=p.window_min} @@ -383,7 +453,7 @@ congestion_update :: proc(h:^Host,p:^Peer) { } p.stats.send_limit=f32(p.rate) p.cc_time,p.cc_rtt,p.cc_samples,p.cc_limited=now,0,0,false - p.cc_acked,p.cc_lost=p.stats.packets_acked,p.stats.packets_lost + p.cc_acked,p.cc_lost,p.cc_delivered=p.stats.packets_acked,p.stats.packets_lost,p.delivered } // Tokens the bucket holds at most: one tick of the rate, so that a connection paces its datagrams over @@ -529,7 +599,7 @@ handshake_accept :: proc(h:^Host,p:^Peer) { host_protocol_hash :: proc(h:^Host) -> u64 { hash:=base.hash_combine(h.desc.protocol!=nil?h.desc.protocol.hash:0,TRANSPORT_VERSION) for c in h.channels[:USER_CHANNELS] { - hash=base.hash_combine(base.hash_combine(hash,u64(c.kind)<<32|u64(c.max_message_bytes)),u64(c.window)) + hash=base.hash_combine(base.hash_combine(hash,u64(c.kind)<<32|u64(c.max_message_bytes)),u64(c.kind==.Reliable?channel_chunks(c.queue_bytes):0)) } return hash } @@ -717,7 +787,12 @@ record_push :: proc(p:^Peer,record:Record) -> bool { @(private) peer_resolve :: proc(h:^Host,p:^Peer,seq:u16,received:bool) { sent:=&p.sent[seq%SENT_WINDOW] - if received {p.stats.packets_acked+=1} else {p.stats.packets_lost+=1} + if received { + p.stats.packets_acked+=1 + p.delivered+=u64(sent.bytes) + } else { + p.stats.packets_lost+=1 + } p.stats.loss+=((received?0:1)-p.stats.loss)*0.02 for i in sent.first.. bool { now:=h.back.now has_ack:=serial.bit_read_bool(r) ack:=u16(serial.bit_read(r,16)) - ack_bits,ack_count:=ack_bits_read(r) + ack_bits,ack_count,acks_ok:=ack_bits_read(r) delay:=f64(serial.bit_read(r,8))*0.001 - if r.error {return false} + if !acks_ok || r.error {return false} if p.recv_any && counter<=p.recv_counter { // Within the window of acknowledgements and not received before: it came late. What in it is safe // out of order is taken; it is not acknowledged, so the sender resends the rest as if it were lost. @@ -793,8 +868,9 @@ peer_wants_send :: proc(h:^Host,p:^Peer) -> bool { b:=&h.back idle:=b.now-p.last_send if b.send_pending || b.clock_due {return true} - if idle>=h.desc.keepalive || (p.ack_pending && idle>=ACK_DELAY) {return true} + if idle>=h.desc.keepalive || (p.ack_pending && (idle>=ACK_DELAY || p.ack_chunks>=ACK_EVERY)) {return true} if idle>=1/f64(h.desc.tick_rate) && channels_pending(p) {return true} + if u16(p.send_seq-p.sent_tail)=PACKET_BYTES && channels_streaming(p) {return true} return p.sent_tail!=p.send_seq && idle>=clamp(p.rtt*2+p.rtt_variance*4,0.1,1) } @@ -809,7 +885,7 @@ peer_send :: proc(h:^Host,p:^Peer) { } for &c in p.channels {c.spent=0} budget:=repl_budget(h,p) - for packet in 0..=SENT_WINDOW-1 {break} if packet>0 && p.tokens<=0 {break} first_record:=p.record_tail @@ -824,7 +900,7 @@ peer_send :: proc(h:^Host,p:^Peer) { yield:=b.repl.key_cursor0?min(budget,PACKET_BYTES*4):0 if channels_write(h,p,&w,SECTION_RESERVE,yield) {wrote=true} more:=false - last:=packet+1>=h.desc.packets_per_update || u16(p.send_seq+1-p.sent_tail)>=SENT_WINDOW-1 || p.tokens<=PACKET_BYTES + last:=u16(p.send_seq+1-p.sent_tail)>=SENT_WINDOW-1 || p.tokens<=PACKET_BYTES if objects_write(h,p,&w,seq,&budget,&more,last) {wrote=true} if packet>0 && !wrote { p.record_tail=first_record @@ -836,7 +912,7 @@ peer_send :: proc(h:^Host,p:^Peer) { p.send_counter+=1 p.tokens-=f64(len(bytes)) p.last_send=b.now - p.ack_pending=false + p.ack_pending,p.ack_chunks=false,0 p.stats.packets_sent+=1 p.stats.bytes_sent+=u64(len(bytes)) transport_send(h,p.address,bytes) diff --git a/net/replication.odin b/net/replication.odin index 45247c9bf..da6f03437 100644 --- a/net/replication.odin +++ b/net/replication.odin @@ -51,7 +51,7 @@ Replication_Desc :: struct { // object_create refuses with Full an object whose state does not fit baseline_words:int, // baselines, likewise. A server shares each one among the connections that take it in the same state; // default: eight per object (two per connection below four), and on a client two per replica - records:int, // per connection: objects and chunks in datagrams not yet acknowledged, default 4096 + records:int, // per connection: objects and inputs in datagrams not yet acknowledged, default 4096; chunks have room of their own budget_bytes:int, // object data per connection and tick, default 4096 max_updates:int, // objects considered per connection and tick, default 256 send_every:int, // ticks between sends, default 1 diff --git a/net/scope.odin b/net/scope.odin index fd11b9dfe..b3b31cb50 100644 --- a/net/scope.odin +++ b/net/scope.odin @@ -262,7 +262,7 @@ repl_budget :: proc(h:^Host,p:^Peer) -> int { r:=&b.repl r.key_count,r.key_cursor,r.key_overflow,r.key_wrote=0,0,false,false if r.protocol==nil {return 0} - if !r.server {return PACKET_BYTES*8*h.desc.packets_per_update} + if !r.server {return int(max(p.tokens,PACKET_BYTES))*8} if !b.send_pending {return 0} pr:=&p.repl keys:=r.keys diff --git a/net/worker_test.odin b/net/worker_test.odin index 48576ebc0..0c2e408b6 100644 --- a/net/worker_test.odin +++ b/net/worker_test.odin @@ -205,7 +205,7 @@ server_benchmark :: proc(t:^testing.T,everywhere,latency:bool) { tp:Test_Protocol;test_protocol_init(&tp);defer test_protocol_destroy(&tp) w:=new(Test_World);defer free(w) test_world_init(w,CLIENTS+1,seed=3,packets=16384);defer test_world_destroy(w) - server:=test_host(w,{mode=.Server,protocol=&tp.protocol,max_connections=CLIENTS,send_rate=1<<20, + server:=test_host(w,{mode=.Server,protocol=&tp.protocol,max_connections=CLIENTS, replication={max_objects=2048,budget_bytes=1200,cell_size=40,filter_interval=4,object_stats=true}}) clients:[CLIENTS]^Host on_server:[CLIENTS]Connection diff --git a/plugins/multiplayer/runtime/journal.odin b/plugins/multiplayer/runtime/journal.odin index 28a335ee1..84fc335d5 100644 --- a/plugins/multiplayer/runtime/journal.odin +++ b/plugins/multiplayer/runtime/journal.odin @@ -35,7 +35,7 @@ JOURNAL_CHANNEL :: 0 // of the host: the channel of parts JOURNAL_QUEUE :: 64<<10 // bytes of parts a connection has waiting for the channel before the journal hands it more // The channels of a session's host. The window of parts holds 256 KiB in flight: what 2.5 MiB/s keep // in flight over 100 ms. -JOURNAL_CHANNELS :: []wire.Channel_Desc{{kind=.Reliable,window=1024,queue_bytes=JOURNAL_QUEUE}} +JOURNAL_CHANNELS :: []wire.Channel_Desc{{kind=.Reliable,queue_bytes=JOURNAL_QUEUE}} JOURNAL_ASK_TICKS :: 30 // a client asks for parts again after this long without an answer, and for a base after a gap this long JOURNAL_AHEAD_TICKS :: 240 // server: an ask for a tick further ahead than this is let out at once (Journal_Held.ahead) JOURNAL_LOG :: 4096 // operations of a key a client holds ahead of where it stands From 9c7b4e49247666c156a0368088b9555ed4ad16ac Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:18:44 -0300 Subject: [PATCH 007/123] store: address blobs with standard BLAKE3 Pieces are BLAKE3 groups of 64 KiB: a piece's value is its group value, the address is the BLAKE3 root of the table, and a blob of one piece has its plain BLAKE3 hash; so the address equals blake3.hash of the bytes. Hashing a file holds its table. Layout version 2 refuses roots of the old addresses. std/blake3 joins the std engine library. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-store/SKILL.md | 14 ++- cook/build/engine.odin | 2 +- store/address.odin | 140 +++++++++++----------------- store/blobs.odin | 29 ++++-- store/concurrent_test.odin | 6 +- store/hostile_test.odin | 13 ++- store/receive.odin | 33 ++++--- store/store.odin | 2 +- store/store_test.odin | 34 +++++-- 9 files changed, 140 insertions(+), 133 deletions(-) diff --git a/.agents/skills/mirai-store/SKILL.md b/.agents/skills/mirai-store/SKILL.md index 234b31c84..989c54613 100644 --- a/.agents/skills/mirai-store/SKILL.md +++ b/.agents/skills/mirai-store/SKILL.md @@ -1,7 +1,7 @@ --- name: mirai-store description: Mirai machine-wide content-addressed store (the `store` package) - blobs named by their - BLAKE2b address and cut in pieces, receiving a blob piece by piece with resume, range reads of + BLAKE3 address and cut in pieces, receiving a blob piece by piece with resume, range reads of partial blobs, refs that name an address, usage and sweep, and the default root. Use when storing or sharing heavy content between projects or peers, building a blob transport, an asset library or a derived data cache on top of it, or diagnosing a corrupt or missing blob. @@ -33,10 +33,14 @@ derived data cache that use it decide what a ref means and what is live. ## Invariants -- The address is a pure function of the bytes: the length and the hash of every fixed-size piece, - one rule for every size. Piece hashes and addresses are keyed apart so none is taken for another. - A piece hash depends only on its piece and index, so pieces can be hashed in parallel; the - address only orders them. +- The address is the standard BLAKE3 hash of the bytes, so any BLAKE3 tool computes it. A piece is + a BLAKE3 group: its value is the group's chaining value, and the address is the BLAKE3 root of + the table; a blob of one piece has its hash as its only entry. A piece value depends only on + its bytes and index, so pieces can be hashed in parallel. Hashing a file holds its table, 32 + bytes a piece, never its bytes. The root version carries the address rule: a root of another + rule is refused, never migrated. +- A table does not carry the length of its last piece: a size with the right piece count passes + the table check, and then no true last piece fits it, so nothing is published. - Only the two private procedures of `address.odin` (the value of a piece, the address of a table) touch the hash primitive; everything else, tests included, goes through them or the API, so the primitive can be replaced in one place. diff --git a/cook/build/engine.odin b/cook/build/engine.odin index 31f71c658..b6823ae07 100644 --- a/cook/build/engine.odin +++ b/cook/build/engine.odin @@ -77,7 +77,7 @@ Engine_Package :: struct { ENGINE_PACKAGES :: [?]Engine_Package{ {.Base,"std/base","std"},{.Base,"std/subprocess","std"},{.Base,"std/math","std"},{.Base,"std/geom","std"},{.Base,"std/serial","std"}, {.Base,"std/spatial","std"},{.Base,"std/jobs","std"},{.Base,"std/anim","std"},{.Base,"std/ai","std"},{.Base,"std/input","std"}, - {.Base,"std/ui_format","std"}, + {.Base,"std/ui_format","std"},{.Base,"std/blake3","std"}, {.Base,"audio","system"},{.Base,"platform","system"},{.Base,"rhi","system"}, {.Base,"project","system"},{.Base,"store","system"},{.Base,"core",""}, diff --git a/store/address.odin b/store/address.odin index b9ebff019..8a8dac342 100644 --- a/store/address.odin +++ b/store/address.odin @@ -1,25 +1,16 @@ package store -import "core:crypto/blake2b" import "core:os" +import "mirai:std/blake3" -// An address is a pure function of a blob's bytes. Only piece_value and -// address_value know the hash primitive, so it can be replaced in one place. +// An address is the standard BLAKE3 hash of a blob's bytes. Only piece_value +// and address_value know the hash primitive, so it can be replaced in one place. Hash :: distinct [32]u8 -// A blob is cut in pieces of PIECE_BYTES, the last one shorter; an empty blob has none. +// A blob is cut in pieces of PIECE_BYTES, the last one shorter; an empty blob +// has none. A piece is a BLAKE3 group, so the address is the root of the table. PIECE_BYTES :: 64<<10 - -// BLAKE2b-256 keyed apart for pieces and addresses, so a piece hash, a piece -// table and an address can never be taken for one another. -@(private) -KEY_PIECE :: "mirai store piece" -@(private) -KEY_ADDRESS :: "mirai store address" - -// Gives the piece hash at index of a table, called for every index in order. -@(private) -Piece_Source :: proc(user:rawptr,index:int) -> (Hash,Error) +#assert(PIECE_BYTES==blake3.GROUP_BYTES) piece_count :: proc "contextless" (size:i64) -> int { if size<=0 {return 0} @@ -32,23 +23,26 @@ piece_size :: proc "contextless" (size:i64,index:int) -> int { return int(min(size-i64(index)*PIECE_BYTES,PIECE_BYTES)) } -address_of_bytes :: proc(data:[]u8) -> Hash { - data:=data - hash,_:=address_value(i64(len(data)),proc(user:rawptr,index:int) -> (Hash,Error) { - data:=(^[]u8)(user)^ - return piece_value(data[index*PIECE_BYTES:][:piece_size(i64(len(data)),index)],index),.None - },&data) - return hash +// The address of bytes in memory; its piece table, 32 bytes a piece, is held +// with allocator while it is computed. +address_of_bytes :: proc(data:[]u8,allocator:=context.allocator) -> (hash:Hash,error:Error) { + pieces:=bytes_table(data,allocator) or_return + defer delete(pieces,allocator) + return address_value(i64(len(data)),pieces),.None } -// The address of the file at path, read piece by piece. -address_of_file :: proc(path:string) -> (hash:Hash,size:i64,error:Error) { +// The address of the file at path, read once piece by piece; its piece table +// is held with allocator. +address_of_file :: proc(path:string,allocator:=context.allocator) -> (hash:Hash,size:i64,error:Error) { file,open_error:=os.open(path) if open_error!=nil {return {},0,os_error(open_error)} defer os.close(file) file_size,size_error:=os.file_size(file) if size_error!=nil {return {},0,os_error(size_error)} - hash,error=file_hash(file,file_size,nil,nil) + pieces,make_error:=make([]Hash,piece_count(file_size),allocator) + if make_error!=nil {return {},0,.Out_Of_Memory} + defer delete(pieces,allocator) + hash,error=file_hash(file,file_size,nil,pieces) if error==.Corrupt {error=.Io} return hash,file_size,error } @@ -84,80 +78,58 @@ hex_digit :: proc "contextless" (c:u8) -> (u8,bool) { return 0,false } -// The hash of the piece at index of a blob, from its bytes. +// The value of piece index of a blob of size, from its bytes: the BLAKE3 +// hash of a blob of one piece, otherwise its BLAKE3 group value. Out-of-range +// input gives a value no table holds. @(private) -piece_value :: proc(piece:[]u8,index:int) -> Hash { - ctx:blake2b.Context - key:=KEY_PIECE - blake2b.init_mac(&ctx,transmute([]u8)key,size_of(Hash)) - blake2b.update(&ctx,piece) - hash:Hash - blake2b.final(&ctx,hash[:]) - return hash +piece_value :: proc(piece:[]u8,index:int,size:i64) -> Hash { + if size<=PIECE_BYTES {return Hash(blake3.hash(piece))} + value,_:=blake3.group_value(piece,u64(max(index,0))) + return Hash(value) } -// The address of a blob of size bytes, from its piece table, which pieces -// gives one entry at a time so a table is never held whole. +// The address of a blob of size, from its piece table: the BLAKE3 hash of +// nothing for an empty blob, the single value of a blob of one piece, +// otherwise the BLAKE3 root of the table. A table that does not fit size +// gives a value no blob has. @(private) -address_value :: proc(size:i64,pieces:Piece_Source,user:rawptr) -> (Hash,Error) { - ctx:blake2b.Context - key:=KEY_ADDRESS - blake2b.init_mac(&ctx,transmute([]u8)key,size_of(Hash)) - length:[8]u8 - for i in 0..<8 {length[i]=u8(u64(size)>>(8*uint(i)))} - blake2b.update(&ctx,length[:]) - for index in 0.. Hash { + if size<0 || len(pieces)!=piece_count(size) {return {}} + if size==0 {return Hash(blake3.hash(nil))} + if len(pieces)==1 {return pieces[0]} + root,_:=blake3.root(transmute([]blake3.Hash)pieces,u64(size)) + return Hash(root) } -// The address of a table held in memory. +// The piece table of bytes in memory. @(private) -table_address :: proc(size:i64,pieces:[]Hash) -> Hash { - pieces:=pieces - hash,_:=address_value(size,proc(user:rawptr,index:int) -> (Hash,Error) { - return (^[]Hash)(user)[index],.None - },&pieces) - return hash +bytes_table :: proc(data:[]u8,allocator:=context.allocator) -> ([]Hash,Error) { + size:=i64(len(data)) + pieces,make_error:=make([]Hash,piece_count(size),allocator) + if make_error!=nil {return nil,.Out_Of_Memory} + for &piece,index in pieces {piece=piece_value(data[index*PIECE_BYTES:][:piece_size(size,index)],index,size)} + return pieces,.None } -// Reads size bytes of file piece by piece: their address. Each piece is also -// written to copy_to and its hash to table, when given. A file shorter than -// size, or longer, is Corrupt. +// Reads size bytes of file piece by piece into table: their address. Each +// piece is also written to copy_to, when given. A file shorter than size, or +// longer, is Corrupt. @(private) file_hash :: proc(file:^os.File,size:i64,copy_to:^os.File,table:[]Hash) -> (hash:Hash,error:Error) { - if table!=nil && len(table)!=piece_count(size) {return {},.Invalid_Argument} - Reading :: struct { - buffer:[PIECE_BYTES]u8, - file,copy_to:^os.File, - size:i64, - table:[]Hash, - } - reading:=Reading{file=file,copy_to=copy_to,size=size,table=table} - hash=address_value(size,proc(user:rawptr,index:int) -> (Hash,Error) { - r:=(^Reading)(user) + if len(table)!=piece_count(size) {return {},.Invalid_Argument} + buffer:[PIECE_BYTES]u8=--- + for index in 0..0 {return {},.Corrupt} - return hash,.None + return address_value(size,table),.None } // Fills buffer from file at offset; Corrupt when the file ends first. diff --git a/store/blobs.odin b/store/blobs.odin index 3ff905bca..ee5c49d51 100644 --- a/store/blobs.odin +++ b/store/blobs.odin @@ -8,7 +8,9 @@ import "core:os" // both in hex, and are never written in place. put_bytes :: proc(s:^Store,data:[]u8) -> (hash:Hash,error:Error) { - hash=address_of_bytes(data) + pieces:=bytes_table(data,s.allocator) or_return + defer delete(pieces,s.allocator) + hash=address_value(i64(len(data)),pieces) if has(s,hash) {return hash,.None} temp_buffer:[PATH_BYTES]u8 temp,file:=temp_create(s.root,&temp_buffer) or_return @@ -19,8 +21,12 @@ put_bytes :: proc(s:^Store,data:[]u8) -> (hash:Hash,error:Error) { os.remove(temp) return {},os_error(write_error) if write_error!=nil else .Io } - if error=blob_place(s,temp,hash);error!=.None {os.remove(temp)} - return hash,error + if error=blob_place(s,temp,hash);error!=.None { + os.remove(temp) + return hash,error + } + table_keep(s,hash,i64(len(data)),pieces) + return hash,.None } // Puts the file at path, read once piece by piece. @@ -30,9 +36,12 @@ put_file :: proc(s:^Store,path:string) -> (hash:Hash,error:Error) { defer os.close(source) size,size_error:=os.file_size(source) if size_error!=nil {return {},os_error(size_error)} + pieces,make_error:=make([]Hash,piece_count(size),s.allocator) + if make_error!=nil {return {},.Out_Of_Memory} + defer delete(pieces,s.allocator) temp_buffer:[PATH_BYTES]u8 temp,file:=temp_create(s.root,&temp_buffer) or_return - hash,error=file_hash(source,size,file,nil) + hash,error=file_hash(source,size,file,pieces) sync_error:=os.sync(file) if error==.None else nil os.close(file) if error==.Corrupt {error=.Io} @@ -41,8 +50,12 @@ put_file :: proc(s:^Store,path:string) -> (hash:Hash,error:Error) { os.remove(temp) return hash if error==.None else {},error } - if error=blob_place(s,temp,hash);error!=.None {os.remove(temp)} - return hash,error + if error=blob_place(s,temp,hash);error!=.None { + os.remove(temp) + return hash,error + } + table_keep(s,hash,size,pieces) + return hash,.None } // Whether the blob is complete here. @@ -150,7 +163,7 @@ reader_close :: proc(r:^Reader) { verify :: proc(s:^Store,hash:Hash) -> Error { buffer:[PATH_BYTES]u8 path:=blob_file(s,hash,&buffer) or_return - address,_,error:=address_of_file(path) + address,_,error:=address_of_file(path,s.allocator) if error==.Io {return .Corrupt} if error!=.None {return error} return .None if address==hash else .Corrupt @@ -222,7 +235,7 @@ table_load :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { read_exact(file,header[:],0) or_return if header!=table_header(size) {return .Corrupt} read_exact(file,mem.slice_to_bytes(pieces),TABLE_HEADER) or_return - return .None if table_address(size,pieces)==hash else .Corrupt + return .None if address_value(size,pieces)==hash else .Corrupt } // Keeps the table of a complete blob; one that cannot be written is hashed again when asked for. diff --git a/store/concurrent_test.odin b/store/concurrent_test.odin index b22665c9d..84f07f183 100644 --- a/store/concurrent_test.odin +++ b/store/concurrent_test.odin @@ -19,7 +19,7 @@ Worker :: struct { @(private="file") worker_receive :: proc(w:^Worker,data:[]u8,pieces:[]Hash) { - hash:=address_of_bytes(data) + hash:=test_address(data) if error:=receive_begin(w.s,hash,i64(len(data)),pieces);error!=.None {w.error=error;return} for i in 0.. Error { if size<0 || len(pieces)!=piece_count(size) {return .Invalid_Argument} - if table_address(size,pieces)!=hash {return .Corrupt} + if address_value(size,pieces)!=hash {return .Corrupt} if has(s,hash) { partial_discard(s,hash) return .None @@ -87,7 +87,7 @@ receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { if index<0 || index>=p.count || len(piece)!=piece_size(p.size,index) {return .Invalid_Argument} expected:Hash read_exact(p.table,expected[:],TABLE_HEADER+i64(index)*size_of(Hash)) or_return - if piece_value(piece,index)!=expected {return .Corrupt} + if piece_value(piece,index,p.size)!=expected {return .Corrupt} if _,write_error:=os.write_at(p.data,piece,i64(index)*PIECE_BYTES);write_error!=nil {return os_error(write_error)} mark:=[1]u8{1} if _,write_error:=os.write_at(p.have,mark[:],i64(index));write_error!=nil {return os_error(write_error)} @@ -221,7 +221,7 @@ partial_finish :: proc(s:^Store,hash:Hash) -> Error { if claimed==.Missing {return .None} if claimed!=.None {return claimed} defer claim_end(s,hash,&c) - complete:=claim_verify(hash,&c) or_return + complete:=claim_verify(s,hash,&c) or_return if !complete {return .None} return claim_publish(s,hash,&c) } @@ -250,7 +250,7 @@ partial_claim :: proc(s:^Store,hash:Hash,c:^Claim) -> Error { // and the table the address. A piece that does not match is marked missing // again; a table that does not match is Corrupt and the folder is discarded. @(private) -claim_verify :: proc(hash:Hash,c:^Claim) -> (complete:bool,error:Error) { +claim_verify :: proc(s:^Store,hash:Hash,c:^Claim) -> (complete:bool,error:Error) { p:Partial p,error=partial_open_at(c.partial,{.Data,.Have}) if error==.Corrupt {c.discard=true} @@ -258,25 +258,24 @@ claim_verify :: proc(hash:Hash,c:^Claim) -> (complete:bool,error:Error) { defer partial_close(&p) full:=partial_full(&p) or_return if !full {return false,.None} - Checking :: struct { - buffer:[PIECE_BYTES]u8, - p:^Partial, - damaged:bool, + pieces,make_error:=make([]Hash,p.count,s.allocator) + if make_error!=nil {return false,.Out_Of_Memory} + defer delete(pieces,s.allocator) + read_exact(p.table,mem.slice_to_bytes(pieces),TABLE_HEADER) or_return + buffer:[PIECE_BYTES]u8=--- + damaged:=false + for index in 0.. (Hash,Error) { - c:=(^Checking)(user) - entry,_,good,error:=partial_piece(c.p,index,&c.buffer) - if !good {c.damaged=true} - return entry,error - },&checking) or_return + address:=address_value(p.size,pieces) data_size,size_error:=os.file_size(p.data) if size_error!=nil {return false,os_error(size_error)} if address!=hash || data_size>p.size { c.discard=true return false,.Corrupt } - return !checking.damaged,.None + return !damaged,.None } // Moves the checked data into the blob's place, and its table to the kept tables. @@ -316,7 +315,7 @@ partial_piece :: proc(p:^Partial,index:int,buffer:^[PIECE_BYTES]u8) -> (entry:Ha piece=buffer[:piece_size(p.size,index)] got:=read_exact(p.data,piece,i64(index)*PIECE_BYTES) if got!=.None && got!=.Corrupt {return entry,nil,false,got} - if got==.None && piece_value(piece,index)==entry {return entry,piece,true,.None} + if got==.None && piece_value(piece,index,p.size)==entry {return entry,piece,true,.None} unmark:=[1]u8{0} if _,write_error:=os.write_at(p.have,unmark[:],i64(index));write_error!=nil {return entry,nil,false,os_error(write_error)} return entry,nil,false,.None diff --git a/store/store.odin b/store/store.odin index 34e7ff91a..7e7760179 100644 --- a/store/store.odin +++ b/store/store.odin @@ -9,7 +9,7 @@ import "core:os" // one root without locks: every file appears whole through a rename. // The first line of the root's layout file; another one is refused. -LAYOUT :: "mirai store 1\n" +LAYOUT :: "mirai store 2\n" ROOT_MAX :: 1024 // bytes of a root path NAME_MAX :: 128 // bytes of a ref namespace or name diff --git a/store/store_test.odin b/store/store_test.odin index 955728b62..cd7557d3e 100644 --- a/store/store_test.odin +++ b/store/store_test.odin @@ -4,6 +4,7 @@ import "core:os" import "core:strings" import "core:testing" import "mirai:std/base" +import "mirai:std/blake3" @(private) test_bytes :: proc(size:int,seed:u64) -> []u8 { @@ -13,6 +14,12 @@ test_bytes :: proc(size:int,seed:u64) -> []u8 { return data } +@(private) +test_address :: proc(data:[]u8) -> Hash { + hash,_:=address_of_bytes(data,context.temp_allocator) + return hash +} + @(private) test_piece :: proc(data:[]u8,index:int) -> []u8 { return data[index*PIECE_BYTES:min((index+1)*PIECE_BYTES,len(data))] @@ -21,7 +28,7 @@ test_piece :: proc(data:[]u8,index:int) -> []u8 { @(private) test_table :: proc(data:[]u8) -> []Hash { pieces:=make([]Hash,piece_count(i64(len(data))),context.temp_allocator) - for &piece,i in pieces {piece=piece_value(test_piece(data,i),i)} + for &piece,i in pieces {piece=piece_value(test_piece(data,i),i,i64(len(data)))} return pieces } @@ -47,8 +54,9 @@ address_is_the_same_through_put_bytes_put_file_and_receiving :: proc(t:^testing. defer close(&r) put,put_error:=put_bytes(&a,data) testing.expect_value(t,put_error,Error.None) - testing.expect(t,put==address_of_bytes(data),"put bytes gives the address of the bytes") - testing.expect(t,put==table_address(i64(c.size),test_table(data)),"the address is that of the table") + testing.expect(t,put==test_address(data),"put bytes gives the address of the bytes") + testing.expect(t,put==Hash(blake3.hash(data)),"the address is the BLAKE3 hash of the bytes") + testing.expect(t,put==address_value(i64(c.size),test_table(data)),"the address is that of the table") source:=join_test(root,"source",n) testing.expect(t,os.write_entire_file(source,data)==nil) from_file,file_size,file_error:=address_of_file(source) @@ -98,13 +106,13 @@ a_wrong_table_or_piece_is_refused_and_leaves_no_blob :: proc(t:^testing.T) { s,ok:=test_store(t,folder);if !ok {return} defer close(&s) data:=test_bytes(2*PIECE_BYTES+5,7) - hash:=address_of_bytes(data) + hash:=test_address(data) pieces:=test_table(data) wrong:=make([]Hash,len(pieces),context.temp_allocator) copy(wrong,pieces) wrong[1][0]~=1 testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),wrong),Error.Corrupt) - testing.expect_value(t,receive_begin(&s,hash,i64(len(data))+1,pieces),Error.Corrupt) + testing.expect_value(t,receive_begin(&s,hash,i64(len(data))+PIECE_BYTES,pieces),Error.Invalid_Argument) testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),pieces[:2]),Error.Invalid_Argument) testing.expect_value(t,receive_begin(&s,hash,-1,nil),Error.Invalid_Argument) _,missing_error:=receive_missing(&s,hash,context.temp_allocator) @@ -124,13 +132,19 @@ a_wrong_table_or_piece_is_refused_and_leaves_no_blob :: proc(t:^testing.T) { testing.expect(t,!has(&s,hash),"refused pieces complete nothing") total,_:=usage(&s) testing.expect_value(t,total.blobs,0) + // The table does not hold the length of the last piece: a size that keeps + // the piece count is taken, and then no last piece of the true length fits it. + testing.expect_value(t,remove(&s,hash),Error.None) + testing.expect_value(t,receive_begin(&s,hash,i64(len(data))-1,pieces),Error.None) + for index in 0.. Hash { - hash:=address_of_bytes(data) + hash:=test_address(data) pieces:=test_table(data) testing.expect_value(t,receive_begin(s,hash,i64(len(data)),pieces),Error.None) for index in 1.. Date: Wed, 18 Nov 2026 18:20:20 -0300 Subject: [PATCH 008/123] store: the address of bytes is their BLAKE3 hash, without a table Co-Authored-By: Claude Fable 5.2 --- store/address.odin | 9 +++------ store/hostile_test.odin | 2 -- store/store_test.odin | 3 +-- 3 files changed, 4 insertions(+), 10 deletions(-) diff --git a/store/address.odin b/store/address.odin index 8a8dac342..14466a833 100644 --- a/store/address.odin +++ b/store/address.odin @@ -23,12 +23,9 @@ piece_size :: proc "contextless" (size:i64,index:int) -> int { return int(min(size-i64(index)*PIECE_BYTES,PIECE_BYTES)) } -// The address of bytes in memory; its piece table, 32 bytes a piece, is held -// with allocator while it is computed. -address_of_bytes :: proc(data:[]u8,allocator:=context.allocator) -> (hash:Hash,error:Error) { - pieces:=bytes_table(data,allocator) or_return - defer delete(pieces,allocator) - return address_value(i64(len(data)),pieces),.None +// The address of bytes in memory. +address_of_bytes :: proc "contextless" (data:[]u8) -> Hash { + return Hash(blake3.hash(data)) } // The address of the file at path, read once piece by piece; its piece table diff --git a/store/hostile_test.odin b/store/hostile_test.odin index 43d203b97..821d727a3 100644 --- a/store/hostile_test.odin +++ b/store/hostile_test.odin @@ -123,8 +123,6 @@ hostile_inputs_return_errors :: proc(t:^testing.T) { testing.expect_value(t,table_error,Error.Out_Of_Memory) _,missing_error=receive_missing(&s,hash,out.allocator) testing.expect_value(t,missing_error,Error.Out_Of_Memory) - _,bytes_error:=address_of_bytes(data,out.allocator) - testing.expect_value(t,bytes_error,Error.Out_Of_Memory) _,file_error:=put_file(&out,path_in(folder,"layout")) testing.expect_value(t,file_error,Error.Out_Of_Memory) _,path_error:=blob_path(&s,good,out.allocator) diff --git a/store/store_test.odin b/store/store_test.odin index cd7557d3e..5a3766ae6 100644 --- a/store/store_test.odin +++ b/store/store_test.odin @@ -16,8 +16,7 @@ test_bytes :: proc(size:int,seed:u64) -> []u8 { @(private) test_address :: proc(data:[]u8) -> Hash { - hash,_:=address_of_bytes(data,context.temp_allocator) - return hash + return address_of_bytes(data) } @(private) From b68029ffdb591a211bbc2434efa192ebcba5c8c1 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:46:40 -0300 Subject: [PATCH 009/123] asset: the registry lists the outputs of packages, not sidecars A mount and the root take the entries packages_resolve gives; a scan lists them by their blob, with no source, sidecar or imported folder. An entry not built is listed Not_Built, a blob of another asset is Broken, hostile entries are refused. Registry_File.listed replaces the imported key, the cache layout goes to 6 and memory holds no listed path. Callers that passed an imported folder pass nothing until they resolve their packages. Co-Authored-By: Claude Opus 5.5 --- .../skills/mirai-assets/references/formats.md | 26 +- .agents/skills/mirai-imports/SKILL.md | 4 +- .../mirai-imports/references/imports.md | 16 +- asset/registry.odin | 277 +++++++++++------- asset/registry_memory.odin | 13 +- asset/registry_test.odin | 187 ++++++++---- asset/sidecar.odin | 25 +- cook/author/command.odin | 4 +- cook/build/cook.odin | 4 +- cook/build/world.odin | 4 +- cook/import/identity.odin | 4 +- cook/import/outputs_test.odin | 2 +- cook/import/sources/reimport_test.odin | 4 +- cook/import/sources/sync.odin | 2 +- cook/import/sources/write_test.odin | 4 +- editor/asset_delete.odin | 2 +- editor/asset_move.odin | 6 +- editor/assets_parts.odin | 2 +- editor/import.odin | 8 +- host/assets.odin | 4 +- host/content.odin | 2 +- tools/mirai/asset.odin | 4 +- 22 files changed, 362 insertions(+), 242 deletions(-) diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index cc2f9a366..308f144f4 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -221,18 +221,20 @@ scan opens every file and writes it again. is a prefab ([prefab](scenes.md#prefab)). - `registry_dependents` comes from an inverse index built with each change: how many use an asset is its length, without walking the files (`asset.list` reports it as `used_by`). -- **Imported folders.** An assets folder may have an imported folder with it (that of the root - and of each mount: `.mirai/imported/` of the project or the plugin): what the - [imports](../../mirai-imports/references/imports.md#identity-and-reimport) of its sources build - that nobody edits, never versioned, each file named by its id and the key of what it was built - from. A scan reads the sidecars it walks (`asset/sidecar.odin`, the one reader of the format) - and lists each output of the cache like any asset (`Registry_File.imported` is that key) at - `//`; two outputs of one kind and - name are told apart by their index, and a path another file or output has takes the name of - the source: never the order of a scan. One whose source is gone is not listed, and one that - moved is listed there at the next scan, which a refresh of a sidecar or of its source runs. - `registry_file_path` gives the file of either folder. A cooked folder has none: the cook - writes every asset at its path. +- **Listed outputs.** What the + [imports](../../mirai-imports/references/imports.md#identity-and-reimport) build that nobody + edits is not a file of the assets folder but a blob of the machine store. The root and each + mount take the entries `packages_resolve` gives for the packages their root pins + (`asset/packages.odin`), and a scan lists each like any asset (`Registry_File.listed`), with no + source, sidecar or imported folder anywhere; the registry reads no sidecar. An entry whose + build is not on this machine, or whose blob is gone, stays listed as `Not_Built` (its id and + kind, no file, not found by id), never removed and never an error of the scan; a blob that is + not the asset its entry names is `Broken`. A path a file, a folder or another entry has takes + the id of the entry after its name, never the order of a scan, and an entry with no id or a + path that is empty or leaves its folder is refused (`Registry.refused`). Blobs do not change: + a refresh keeps what is listed, unless a file of the folder appears at its path (which runs a + scan), and memory holds no listed path. `registry_file_path` gives the blob of a listed path. + A cooked folder has none: the cook writes every asset at its path. - **Mounts.** `registry_mount` adds a folder under a prefix (a plugin's assets, in `plugins/`). A prefix that starts with a dot belongs to the program's assets (the editor's, in `.editor`): they are found by id and by path, but a project's lists omit them diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index 145a20858..06b2f5630 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -219,8 +219,8 @@ the assets. a plugin for a whole source whose inputs changed), `cook/import/outputs_test.odin` (the models of a whole source share their parts by name, names with `/` go to subfolders, inputs count in the key), - `asset/registry_test.odin` (the listing of the cache by where the sidecar is, never by the order - of a scan), `cook/import/sources/settings_test.odin` (a sidecar that is not one is refused) and + `asset/registry_test.odin` (the cache a project pins is listed from the store without its + sources), `cook/import/sources/settings_test.odin` (a sidecar that is not one is refused) and `cook/import/sources/write_test.odin` (importing again keeps what was changed in a material and takes the rest from the source). After changing what an import writes, prove it on the demo: delete `demo/.mirai/imported`, `mirai build editor demo`, and `git status` shows no sidecar diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index 2415c7556..8425fb349 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -205,7 +205,7 @@ the asset: the [sidecar](#identity-and-reimport) is. A source lives in the assets folder of a project or a plugin, beside what it makes, in the folder of its theme (never a folder per asset type: the browser filters by type). `.import`, the sidecar, is what is versioned of an import with its source (`asset/sidecar.odin`, read by the -registry and by the importer alike): a text with the settings, the `folder` beside the source its +importer): a text with the settings, the `folder` beside the source its assets go to (a model's: its stem, or what `Format.folder` says; an image's, a sound's: none, they go beside it), the files the import read besides the source (`inputs`) and one output per asset the source gives, by kind, name and index, with its id. Nothing else decides an id: the first @@ -220,10 +220,10 @@ source, the engine's or a plugin's: - **The cache**: meshes, textures, sounds and fonts. Nothing edits them after the import, so the source and the sidecar give them back whole. Each is a file of the imported folder of the root (`project.imported_path`: `.mirai/imported/`), named by its id and by the key of what it was - built from (`import_key`), never versioned. Nothing stores where it is listed: the - [registry](../../mirai-assets/references/formats.md#registry) lists it at - `//`. [Sync](#sync) builds it again where it is - missing. + built from (`import_key`), never versioned. The + [registry](../../mirai-assets/references/formats.md#registry) lists the cache only from the + entries of the packages a root pins, never from sidecars. [Sync](#sync) builds it again where it + is missing. - **Documents**: materials, prefabs, skeletons, clips, texts and prompts. The editor edits them, so they are files of the assets folder, versioned. An import finds each by its id in the registry of the root, wherever a person moved it, writes again what the source gives and keeps @@ -241,10 +241,8 @@ source, the engine's or a plugin's: a warning: its file goes with it when it is of the cache, and stays as a plain asset when it is a document. - **A source moved or renamed** keeps its import as long as its sidecar goes with it, and takes - what it has in the cache with it without an import: the registry lists those assets where the - sidecar is at its next scan, which a refresh of a sidecar or of its source triggers (the editor - moves a source with its sidecar, and a folder with its sources). A source that is deleted takes - them out of the listing at once. A copy of the source without its sidecar is another import, + what it has in the cache with it without an import (the editor moves a source with its + sidecar, and a folder with its sources). A copy of the source without its sidecar is another import, with ids of its own. - **An import never touches another.** Two models that embed the same image get a texture asset each; the [texture cache](#texture-cache) keeps one encoded copy. No import writes a file or a diff --git a/asset/registry.odin b/asset/registry.odin index 1887d18a8..859558b14 100644 --- a/asset/registry.odin +++ b/asset/registry.odin @@ -20,7 +20,7 @@ import "mirai:std/serial" REGISTRY_HEAD :: 4096 REGISTRY_MAX_DEPENDENCIES :: 1<<20 REGISTRY_CACHE_MAGIC :: u32('M')|u32('R')<<8|u32('E')<<16|u32('G')<<24 -REGISTRY_CACHE_VERSION :: 5 // of its layout; a cache or index of another one, or of another container VERSION, is not read +REGISTRY_CACHE_VERSION :: 6 // of its layout; a cache or index of another one, or of another container VERSION, is not read // The packed index of a folder of cooked assets, inside it: what a scan of // the folder would find, written by whoever cooked it. registry_load reads it // instead of looking at the files. @@ -30,6 +30,7 @@ Registry_State :: enum u8 { Other, // not an asset Asset, Broken, // starts as an asset and contradicts itself: truncated, or with a directory that does not fit + Not_Built, // a listed output of a package whose build is not on this machine: its id and kind, no file } Registry_File :: struct { @@ -37,7 +38,7 @@ Registry_File :: struct { size:i64, time:i64, // of its last change, nanoseconds since 1970 state:Registry_State, - imported:u64, // not 0: its file is in the imported folder that goes with its assets folder, named by its id and this key (sidecar.odin) + listed:bool, // an output of a package the registry was given (Package_Listed): its file is a blob of the store id:base.Asset_Id, owner:base.Asset_Id, // the asset it is a part of; zero for one of its own kind:Fourcc, @@ -81,8 +82,8 @@ Registry_Stats :: struct { // A folder of assets seen under prefix/, beside the root: the assets of a // plugin. A prefix that starts with a dot is of the program's own assets. // A guest is a folder the program did not ship (registry_mount_guest). -// imported: the imported folder that goes with it; "" for none. -Registry_Mount :: struct {folder,prefix:string,guest:bool,imported:string} +// listed: the outputs of the packages of its root, at paths under the prefix. +Registry_Mount :: struct {folder,prefix:string,guest:bool,listed:[]Package_Listed} // Whether lists of what a project has show a file: those under a mount whose // prefix starts with a dot are the program's own, found by their path. @@ -90,7 +91,8 @@ registry_listed :: proc "contextless" (file:^Registry_File) -> bool {return len( Registry :: struct { root:string, - imported:string, // the imported folder that goes with the root; "" for none + listed:[]Package_Listed, // the outputs of the packages of the root, listed with its files + listed_files:map[string]string, // the path where each listed output is, to the blob of its entry ("" when not built) cache:string, // "" for none mounts:[dynamic]Registry_Mount, files:[dynamic]Registry_File, // every file, sorted by path @@ -126,20 +128,24 @@ registry_slashed :: proc(path:string,allocator:runtime.Allocator) -> (slashed:st return string(bytes),nil } -// False, with a registry of no folder that registry_destroy takes, without memory for it. -registry_init :: proc(r:^Registry,root:string,cache:="",allocator:=context.allocator,imported:="") -> bool { +// listed: the outputs of the packages of the root (packages_resolve), which +// scans list beside its files without a source or a sidecar. False, with a +// registry of no folder that registry_destroy takes, without memory for it. +registry_init :: proc(r:^Registry,root:string,cache:="",allocator:=context.allocator,listed:[]Package_Listed=nil) -> bool { r^={allocator=allocator} - no_root,no_cache,no_imported:runtime.Allocator_Error + no_root,no_cache:runtime.Allocator_Error r.root,no_root=registry_slashed(root,allocator) r.cache,no_cache=registry_slashed(cache,allocator) - r.imported,no_imported=registry_slashed(imported,allocator) - if no_root!=nil || no_cache!=nil || no_imported!=nil { + kept,no_listed:=registry_listed_clone(listed,allocator) + if no_root!=nil || no_cache!=nil || !no_listed { delete(r.root,allocator) delete(r.cache,allocator) - delete(r.imported,allocator) - r.root,r.cache,r.imported="","","" + registry_listed_free(kept,allocator) + r.root,r.cache,kept="","",nil no_root=.Out_Of_Memory } + r.listed=kept + r.listed_files=make(map[string]string,allocator) r.files=make([dynamic]Registry_File,allocator) r.folders=make([dynamic]string,allocator) r.dependencies=make([dynamic]base.Asset_Id,allocator) @@ -157,24 +163,54 @@ registry_init :: proc(r:^Registry,root:string,cache:="",allocator:=context.alloc } @(private="file") -registry_mount_add :: proc(r:^Registry,folder,prefix:string,guest:bool,imported:="") -> bool { +registry_mount_add :: proc(r:^Registry,folder,prefix:string,guest:bool,listed:[]Package_Listed=nil) -> bool { kept_folder,no_folder:=strings.clone(folder,r.allocator) kept_prefix,no_prefix:=strings.clone(prefix,r.allocator) - kept_imported,no_imported:=strings.clone(imported,r.allocator) - if no_folder==nil && no_prefix==nil && no_imported==nil { - if _,no_room:=append(&r.mounts,Registry_Mount{kept_folder,kept_prefix,guest,kept_imported});no_room==nil {return true} + kept_listed,listed_ok:=registry_listed_clone(listed,r.allocator) + if no_folder==nil && no_prefix==nil && listed_ok { + if _,no_room:=append(&r.mounts,Registry_Mount{kept_folder,kept_prefix,guest,kept_listed});no_room==nil {return true} } delete(kept_folder,r.allocator) delete(kept_prefix,r.allocator) - delete(kept_imported,r.allocator) + registry_listed_free(kept_listed,r.allocator) return false } -// Adds folder to what the next scan sees, its files under prefix/. The prefix +// Adds folder to what the next scan sees, its files under prefix/, and the +// outputs of the packages of its root, listed under prefix/ too. The prefix // is a relative path with '/' that names no folder of the root. False without // memory for it. -registry_mount :: proc(r:^Registry,folder,prefix:string,imported:="") -> bool { - return registry_mount_add(r,folder,prefix,false,imported) +registry_mount :: proc(r:^Registry,folder,prefix:string,listed:[]Package_Listed=nil) -> bool { + return registry_mount_add(r,folder,prefix,false,listed) +} + +// A copy of entries whose strings live in allocator. False without memory for it. +@(private="file") +registry_listed_clone :: proc(entries:[]Package_Listed,allocator:runtime.Allocator) -> (kept:[]Package_Listed,ok:bool) { + if len(entries)==0 {return nil,true} + no_room:runtime.Allocator_Error + if kept,no_room=make([]Package_Listed,len(entries),allocator);no_room!=nil {return nil,false} + for entry,at in entries { + no_path,no_file:runtime.Allocator_Error + copied:=entry + copied.path,no_path=strings.clone(entry.path,allocator) + copied.file,no_file=strings.clone(entry.file,allocator) + kept[at]=copied + if no_path!=nil || no_file!=nil { + registry_listed_free(kept,allocator) + return nil,false + } + } + return kept,true +} + +@(private="file") +registry_listed_free :: proc(entries:[]Package_Listed,allocator:runtime.Allocator) { + for entry in entries { + delete(entry.path,allocator) + delete(entry.file,allocator) + } + delete(entries,allocator) } // Mounts now a folder the program did not ship (what a player saved or @@ -197,7 +233,7 @@ registry_mount_guest :: proc(r:^Registry,folder,prefix:string) -> (refused:int,e if !registry_mount_add(r,folder,prefix,true) {return 0,.Out_Of_Memory} r.revision+=1 mount:=r.mounts[len(r.mounts)-1] - if error=registry_walk(r,mount,nil,nil,nil,nil);error!=.None { + if error=registry_walk(r,mount,nil,nil,nil);error!=.None { registry_unmount(r,prefix) return } @@ -243,7 +279,7 @@ registry_unmount :: proc(r:^Registry,prefix:string) -> Error { } base.truncate(&r.folders,kept) registry_refused_clear(r,prefix) - delete(r.mounts[at].folder,r.allocator);delete(r.mounts[at].prefix,r.allocator);delete(r.mounts[at].imported,r.allocator) + delete(r.mounts[at].folder,r.allocator);delete(r.mounts[at].prefix,r.allocator);registry_listed_free(r.mounts[at].listed,r.allocator) ordered_remove(&r.mounts,at) indexed:=registry_index(r) if r.cache!="" {registry_cache_write(r)} @@ -289,11 +325,21 @@ registry_refused_clear :: proc(r:^Registry,prefix:string) { base.truncate(&r.refused,kept) } -// The file a path of the registry is: under the root, under the folder of -// the mount whose prefix it starts with, or itself when it is outside them. -// "" without memory for it. +// The file a path of the registry is: the blob of a listed output, under the +// root, under the folder of the mount whose prefix it starts with, or itself +// when it is outside them. "" for a listed output not built, and without +// memory for it. registry_file_path :: proc(r:^Registry,path:string,allocator:=context.allocator) -> string { - if at,known:=r.by_path[path];known && r.files[at].imported!=0 {return registry_imported_file(r,path,r.files[at].id,r.files[at].imported,allocator)} + if blob,listed:=r.listed_files[path];listed { + file,error:=strings.clone(blob,allocator) + return file if error==nil else "" + } + return registry_disk_path(r,path,allocator) +} + +// The file of the folders at a path of the registry, whatever is listed there. +@(private="file") +registry_disk_path :: proc(r:^Registry,path:string,allocator:=context.allocator) -> string { parts:=[3]string{r.root,"/",path} if registry_outside(path) { parts={path,"",""} @@ -325,14 +371,17 @@ registry_destroy :: proc(r:^Registry) { for entry in r.reassigned {registry_reassigned_free(r,entry)} delete(r.reassigned) delete(r.dependents);delete(r.dependent_files) - for mount in r.mounts {delete(mount.folder,r.allocator);delete(mount.prefix,r.allocator);delete(mount.imported,r.allocator)} + for mount in r.mounts {delete(mount.folder,r.allocator);delete(mount.prefix,r.allocator);registry_listed_free(mount.listed,r.allocator)} delete(r.mounts) + registry_listed_free(r.listed,r.allocator) + for path in r.listed_files {delete(path,r.allocator)} + delete(r.listed_files) for &held in r.memory {registry_memory_free(r,&held)} delete(r.memory) delete(r.removed) registry_refused_clear(r,"") delete(r.refused) - delete(r.root,r.allocator);delete(r.cache,r.allocator);delete(r.imported,r.allocator) + delete(r.root,r.allocator);delete(r.cache,r.allocator) r^={} } @@ -475,13 +524,12 @@ registry_scan :: proc(r:^Registry) -> Error { registry_refused_clear(r,"") if !os.is_directory(r.root) {return .Not_Found} - sidecars:=make([dynamic]Sidecar_Found,context.temp_allocator) - if error:=registry_walk(r,{folder=r.root,imported=r.imported},known,previous[:],previous_dependencies[:],&sidecars);error!=.None {return error} + if error:=registry_walk(r,{folder=r.root},known,previous[:],previous_dependencies[:]);error!=.None {return error} for mount in r.mounts { if !os.is_directory(mount.folder) {continue} - if error:=registry_walk(r,mount,known,previous[:],previous_dependencies[:],&sidecars);error!=.None {return error} + if error:=registry_walk(r,mount,known,previous[:],previous_dependencies[:]);error!=.None {return error} } - if error:=registry_imports_list(r,sidecars[:],known,previous[:],previous_dependencies[:]);error!=.None {return error} + if error:=registry_listed_add(r,known,previous[:],previous_dependencies[:]);error!=.None {return error} indexed:=registry_index(r) slice.sort(r.folders[:]) if r.cache!="" && (r.stats.opened>0 || len(previous)!=len(r.files)) {registry_cache_write(r)} @@ -504,7 +552,7 @@ registry_scan :: proc(r:^Registry) -> Error { // known says previous had with the same size and time is not opened again. @(private) registry_walk_reuse :: proc(r:^Registry,file:^Registry_File,full,path:string,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { - if at,seen:=known[path];seen && previous[at].size==file.size && previous[at].time==file.time && previous[at].imported==file.imported { + if at,seen:=known[path];seen && previous[at].size==file.size && previous[at].time==file.time && previous[at].listed==file.listed { file^=previous[at] file.first_dependency=u32(len(r.dependencies)) append(&r.dependencies,..previous_dependencies[previous[at].first_dependency:][:previous[at].dependency_count]) @@ -516,13 +564,8 @@ registry_walk_reuse :: proc(r:^Registry,file:^Registry_File,full,path:string,kno return registry_probe(r,full,file,&r.dependencies) } -// A sidecar a walk found, with the imported folder of its mount. -@(private="file") -Sidecar_Found :: struct {path,full,imported:string} - -// sidecars receives the sidecars the walk finds; nil for a guest, whose sources nobody imports. @(private="file") -registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id,sidecars:^[dynamic]Sidecar_Found) -> Error { +registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { root,root_error:=os.get_absolute_path(folder.folder,context.temp_allocator) if root_error!=nil {return .IO} relative:=make([dynamic]u8,context.temp_allocator) @@ -560,9 +603,6 @@ registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,pre } continue } - if sidecars!=nil && strings.has_suffix(path,SIDECAR_EXTENSION) && folder.imported!="" { - append(sidecars,Sidecar_Found{strings.clone(path,context.temp_allocator),strings.clone(info.fullpath,context.temp_allocator),folder.imported}) - } file:=Registry_File{size=info.size,time=time.time_to_unix_nano(info.modification_time)} r.stats.files+=1 registry_walk_reuse(r,&file,info.fullpath,path,known,previous,previous_dependencies) or_return @@ -572,76 +612,83 @@ registry_walk :: proc(r:^Registry,folder:Registry_Mount,known:map[string]u32,pre return .None } -// Appends what the sidecars a scan found list of the cache whose file their -// imported folder has, at the paths sidecar_paths gives. A path a file of the -// registry or another output has is no identity: every output that would be -// there is listed with the file name of its source, " ()". +// Appends the outputs the packages of the root and of each mount that is no +// guest list, at their paths under its prefix, without looking for a source: +// a built one is its blob, opened as any file; one whose blob is not on this +// machine is Not_Built, never left out. A path a file or a folder of the +// registry or another entry has is no identity: the entry is listed with its +// id after its name, " ()". An entry without an id, or whose path is +// empty, leaves its folder or is one a scan leaves out, goes to refused. @(private="file") -registry_imports_list :: proc(r:^Registry,sidecars:[]Sidecar_Found,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { - Held :: struct {key:u64,size,time:i64} - Listed :: struct {path,file,source:string,held:Held} - folders:=make(map[string]map[base.Asset_Id]Held,context.temp_allocator) - listed:=make([dynamic]Listed,context.temp_allocator) - claims:=make(map[string]int,context.temp_allocator) - for file in r.files {claims[file.path]=2} - for folder in r.folders {claims[folder]=1} - for found in sidecars { - source:=found.full[:len(found.full)-len(SIDECAR_EXTENSION)] - if !os.is_file(source) {continue} - text,read_error:=serial.file_read(found.full,context.temp_allocator) - if read_error!=.None {continue} - sidecar,ok:=sidecar_parse(string(text),context.temp_allocator) - if !ok {continue} - held,seen:=folders[found.imported] - if !seen { - held=make(map[base.Asset_Id]Held,context.temp_allocator) - entries,_:=os.read_all_directory_by_path(found.imported,context.temp_allocator) - for entry in entries { - if id,key,named:=imported_name(entry.name);named && entry.type==.Regular {held[id]={key,entry.size,time.time_to_unix_nano(entry.modification_time)}} - } - folders[found.imported]=held - } - slash:=strings.last_index_byte(found.path,'/') - for path,at in sidecar_paths(sidecar,found.path[:slash+1]) { - output:=sidecar.outputs[at] - kept,built:=held[output.id] - if path=="" || !built {continue} - extension:=path[strings.last_index_byte(path,'.'):] - append(&listed,Listed{path,imported_file(found.imported,output.id,kept.key,extension,context.temp_allocator),source[strings.last_index_any(source,"/\\")+1:],kept}) - claims[path]+=1 - } +registry_listed_add :: proc(r:^Registry,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { + for path in r.listed_files {delete(path,r.allocator)} + clear(&r.listed_files) + claims:=make(map[string]bool,context.temp_allocator) + for file in r.files {claims[file.path]=true} + for folder in r.folders {claims[folder]=true} + registry_listed_under(r,"",r.listed,&claims,known,previous,previous_dependencies) or_return + for mount in r.mounts { + if !mount.guest {registry_listed_under(r,mount.prefix,mount.listed,&claims,known,previous,previous_dependencies) or_return} } - for &l in listed { - if claims[l.path]>1 { - dot:=strings.last_index_byte(l.path,'.') - l.path=fmt.tprintf("%s (%s)%s",l.path[:dot],l.source,l.path[dot:]) + return .None +} + +@(private="file") +registry_listed_under :: proc(r:^Registry,prefix:string,entries:[]Package_Listed,claims:^map[string]bool,known:map[string]u32,previous:[]Registry_File,previous_dependencies:[]base.Asset_Id) -> Error { + for entry in entries { + path:=prefix=="" ? entry.path : fmt.tprintf("%s/%s",prefix,entry.path) + valid:=entry.id!={} && entry.path!="" && folder_valid(entry.path) && !registry_skipped(entry.path) + if valid && (path in claims^ || fmt.tprintf("%s/",path) in claims^) { + dot:=strings.last_index_byte(path,'.') + if dot<=strings.last_index_byte(path,'/') {dot=len(path)} + buffer:[base.ASSET_ID_TEXT]u8 + path=fmt.tprintf("%s (%s)%s",path[:dot],base.asset_id_format(entry.id,buffer[:]),path[dot:]) + valid=!(path in claims^) } + if !valid { + refused,no_room:=strings.clone(path,r.allocator) + if no_room!=nil {return .Out_Of_Memory} + if _,no_append:=append(&r.refused,refused);no_append!=nil { + delete(refused,r.allocator) + return .Out_Of_Memory + } + continue + } + claims^[path]=true // The folders it is in are folders of the listing, also without a file of the assets. - for at:=strings.index_byte(l.path,'/');at>=0;at=strings.index_byte(l.path[at+1:],'/')<0?-1:at+1+strings.index_byte(l.path[at+1:],'/') { - if claims[l.path[:at+1]]>0 {continue} - folder,no_folder:=strings.clone(l.path[:at+1],r.allocator) + for at:=strings.index_byte(path,'/');at>=0;at=strings.index_byte(path[at+1:],'/')<0?-1:at+1+strings.index_byte(path[at+1:],'/') { + if path[:at+1] in claims^ {continue} + folder,no_folder:=strings.clone(path[:at+1],r.allocator) if no_folder!=nil {return .Out_Of_Memory} - append(&r.folders,folder) - claims[folder]=1 + if _,no_room:=append(&r.folders,folder);no_room!=nil { + delete(folder,r.allocator) + return .Out_Of_Memory + } + claims^[folder]=true + } + file:=Registry_File{listed=true,state=.Not_Built,id=entry.id,kind=entry.kind,first_dependency=u32(len(r.dependencies))} + if entry.state==.Built && entry.file!="" { + if info,stat_error:=os.stat(entry.file,context.temp_allocator);stat_error==nil && info.type==.Regular { + file=Registry_File{listed=true,size=info.size,time=time.time_to_unix_nano(info.modification_time)} + registry_walk_reuse(r,&file,entry.file,path,known,previous,previous_dependencies) or_return + // A blob that is not the asset its entry names contradicts it. + if file.state!=.Asset || file.id!=entry.id || file.kind!=entry.kind {file.state=.Broken} + } } - registered:=Registry_File{size=l.held.size,time=l.held.time,imported=l.held.key} r.stats.files+=1 - registry_walk_reuse(r,®istered,l.file,l.path,known,previous,previous_dependencies) or_return - if registered.state==.Asset {registry_file_add(r,registered,l.path) or_return} + registry_file_add(r,file,path) or_return + key,no_key:=strings.clone(path,r.allocator) + if no_key!=nil {return .Out_Of_Memory} + _,blob,_,no_entry:=map_entry(&r.listed_files,key) + if no_entry!=nil { + delete(key,r.allocator) + return .Out_Of_Memory + } + blob^=entry.file } return .None } -// The file of the imported asset the registry lists at path, built from key. -@(private="file") -registry_imported_file :: proc(r:^Registry,path:string,id:base.Asset_Id,key:u64,allocator:=context.allocator) -> string { - folder:=r.imported - for mount in r.mounts { - if registry_under(path,mount.prefix) {folder=mount.imported;break} - } - return imported_file(folder,id,key,path[strings.last_index_byte(path,'.'):],allocator) -} - // Gives every file with the id of an older one an id of its own, in memory: // a copy made outside an editor is a new asset. The copy is no longer a part // of the asset that owned it nor the output of an import (no owner, no import @@ -763,13 +810,13 @@ registry_copy_id :: proc(r:^Registry,id:base.Asset_Id,path:string) -> base.Asset // the files an editor wrote or a watch saw change. A path that is no file // now leaves the registry, one that is new joins it, and one whose size and // time did not change is not opened. A path held in memory keeps what memory -// says, and names a scan leaves out are ignored. A sidecar, or a source that -// has one, scans the folder. +// says, and names a scan leaves out are ignored. A listed output stays as it +// is, unless a file of the folders is at its path now, which scans them. registry_refresh :: proc(r:^Registry,paths:[]string) -> Error { if !os.is_directory(r.root) {return .Not_Found} - // A sidecar or its source changes what the sidecars list, which a scan derives. + // Which of a file and a listed output takes a path is what a scan derives. for path in paths { - if strings.has_suffix(path,SIDECAR_EXTENSION) || strings.concatenate({path,SIDECAR_EXTENSION},context.temp_allocator) in r.by_path {return registry_scan(r)} + if path in r.listed_files && os.is_file(registry_disk_path(r,path,context.temp_allocator)) {return registry_scan(r)} } changed,exhausted:=false,false r.stats.opened,r.stats.reused=0,0 @@ -786,11 +833,15 @@ registry_refresh :: proc(r:^Registry,paths:[]string) -> Error { } forgot:=registry_reassigned_forget(r,path) if _,held:=registry_memory_find(r,path);held {continue} + // Blobs do not change: what is listed changes with the entries, at a scan. + if path in r.listed_files { + r.stats.reused+=1 + continue + } full:=registry_file_path(r,path,context.temp_allocator) info,stat_error:=os.stat(full,context.temp_allocator) present:=stat_error==nil && info.type==.Regular at,known:=r.by_path[path] - imported:=known?r.files[at].imported:0 if !present { if !known {continue} if !changed {r.revision+=1;changed=true} @@ -802,7 +853,7 @@ registry_refresh :: proc(r:^Registry,paths:[]string) -> Error { r.stats.files-=1 continue } - file:=Registry_File{size=info.size,time=time.time_to_unix_nano(info.modification_time),imported=imported} + file:=Registry_File{size=info.size,time=time.time_to_unix_nano(info.modification_time)} if known && !forgot && r.files[at].size==file.size && r.files[at].time==file.time { r.stats.reused+=1 continue @@ -962,8 +1013,8 @@ registry_save :: proc(r:^Registry,path:string) -> bool { serial.write_svarint(&w,file.size) serial.write_svarint(&w,file.time) serial.write_u8(&w,u8(file.state)) - if file.state!=.Asset {continue} - serial.write_u64(&w,file.imported) + if file.state!=.Asset && file.state!=.Not_Built {continue} + serial.write_u8(&w,u8(file.listed)) serial.write_u64(&w,file.id[0]);serial.write_u64(&w,file.id[1]) serial.write_u64(&w,file.owner[0]);serial.write_u64(&w,file.owner[1]) serial.write_u32(&w,u32(file.kind)) @@ -999,7 +1050,7 @@ registry_load :: proc(r:^Registry,path:string) -> Error { state:=registry_cache_read(path,&r.files,&r.dependencies,r.allocator) r.stats.files,r.stats.reused=len(r.files),len(r.files) for mount in r.mounts { - if mount.guest && os.is_directory(mount.folder) {registry_walk(r,mount,nil,nil,nil,nil)} + if mount.guest && os.is_directory(mount.folder) {registry_walk(r,mount,nil,nil,nil)} } indexed:=registry_index(r) slice.sort(r.folders[:]) @@ -1027,8 +1078,10 @@ registry_cache_read :: proc(path:string,files:^[dynamic]Registry_File,dependenci state:=serial.read_u8(&r) if r.failed || state>u8(max(Registry_State)) {ok=false;break} file.state=Registry_State(state) - if file.state==.Asset { - file.imported=serial.read_u64(&r) + if file.state==.Asset || file.state==.Not_Built { + listed:=serial.read_u8(&r) + if listed>1 {ok=false;break} + file.listed=listed==1 file.id={serial.read_u64(&r),serial.read_u64(&r)} file.owner={serial.read_u64(&r),serial.read_u64(&r)} file.kind=Fourcc(serial.read_u32(&r)) diff --git a/asset/registry_memory.odin b/asset/registry_memory.odin index 4169432dd..3e75c545d 100644 --- a/asset/registry_memory.odin +++ b/asset/registry_memory.odin @@ -16,7 +16,8 @@ import "mirai:std/serial" // file in memory takes the size of its content and the time it last changed. // A file outside the folders, named by its absolute path (the Slang source of // a program), is held the same way but is no file of the registry: no record, -// scan or lookup by id sees it. +// scan or lookup by id sees it. A listed output of a package is a blob of the +// store, which nothing changes: memory holds none. Registry_Memory :: struct { path:string, // as Registry_File.path @@ -24,8 +25,10 @@ Registry_Memory :: struct { removed:bool, } -// path holds data in memory: a copy of it. Out_Of_Memory leaves it as it was. +// path holds data in memory: a copy of it. Out_Of_Memory leaves it as it was; +// Invalid for a listed output. registry_memory_set :: proc(r:^Registry,path:string,data:[]u8) -> Error { + if path in r.listed_files {return .Invalid} copied:=asset_bytes_clone(data,r.allocator) or_return held:=registry_memory_take(r,path) or_return registry_memory_free_data(r,held) @@ -34,8 +37,9 @@ registry_memory_set :: proc(r:^Registry,path:string,data:[]u8) -> Error { } // path is not there in memory: no file answers for it until it is written or -// dropped. +// dropped. Invalid for a listed output. registry_memory_remove :: proc(r:^Registry,path:string) -> Error { + if path in r.listed_files {return .Invalid} held:=registry_memory_take(r,path) or_return registry_memory_free_data(r,held) held.data,held.removed=nil,true @@ -134,8 +138,9 @@ registry_memory_write :: proc(r:^Registry,path:string) -> Error { // What a file of the registry holds now: its content in memory, or the file // on disk, aligned for the chunks of the asset it is. Not_Found for a removed -// file and for one that is not there. +// file, for one that is not there and for a listed output not built. registry_read :: proc(r:^Registry,path:string,allocator:=context.allocator,limit:=u64(DEFAULT_MEMORY_LIMIT)) -> (data:[]u8,error:Error) { + if blob,listed:=r.listed_files[path];listed && blob=="" {return nil,.Not_Found} if held,found:=registry_memory_find(r,path);found { if held.removed {return nil,.Not_Found} if u64(len(held.data))>limit {return nil,.Too_Large} diff --git a/asset/registry_test.odin b/asset/registry_test.odin index ec2812d7a..b9d08512b 100644 --- a/asset/registry_test.odin +++ b/asset/registry_test.odin @@ -6,6 +6,7 @@ import "core:testing" import "core:time" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" @(private="file") write_material :: proc(t:^testing.T,types:^Types,path:string,id:base.Asset_Id,texture:base.Asset_Id) { @@ -557,70 +558,152 @@ registry_mounts_a_guest_folder_that_never_takes_an_id_the_registry_has :: proc(t } } -// What the imports of the sources of a folder build that nobody edits is in -// the imported folder that goes with it, each file named by its id and the -// key of what it was built from, and is listed where the sidecar that lists it -// is now, in the folder it names: never by the order of a scan. A sidecar -// that moves with its source moves the listing, also through a refresh; one -// whose source is gone lists nothing. +// What the packages a project and a plugin pin list of the cache is listed +// from the store, with no source, sidecar or imported folder anywhere: a built +// output is its blob and loads, one whose build is not here is listed +// Not_Built, a path a file of the folder has takes the id of the output after +// its name, a refresh keeps every listed output and memory never holds one. @(test) -registry_lists_the_imported_folder_by_where_the_sources_are :: proc(t:^testing.T) { +registry_lists_what_a_project_pins_without_its_sources :: proc(t:^testing.T) { types:Types;types_init(&types);defer types_destroy(&types) - folder:=base.test_folder(t,"registry-imported") - if folder=="" {return} - root:=fmt.tprintf("%s/game/assets",folder) - imported:=fmt.tprintf("%s/game/.mirai/imported",folder) - plugin:=fmt.tprintf("%s/pack/assets",folder) - plugin_imported:=fmt.tprintf("%s/pack/.mirai/imported",folder) - for made in ([]string{fmt.tprintf("%s/props/crate",root),fmt.tprintf("%s/kit",root),imported,plugin,plugin_imported}) {os.make_directory_all(made)} - wood,body,lid,second,atlas,lost,unbuilt,other:=base.Asset_Id{1,1},base.Asset_Id{2,2},base.Asset_Id{3,3},base.Asset_Id{10,10},base.Asset_Id{5,5},base.Asset_Id{6,6},base.Asset_Id{7,7},base.Asset_Id{9,9} - pixels:[16]u8 - sidecar :: proc(t:^testing.T,source,folder:string,present:bool,outputs:[]Import_Output) { - listed:=make([dynamic]Import_Output,context.temp_allocator) - append(&listed,..outputs) - text:=sidecar_text(`{"lods": "None"}`,{folder=folder,outputs=listed},context.temp_allocator) - testing.expect_value(t,serial.file_write_atomic(fmt.tprintf("%s%s",source,SIDECAR_EXTENSION),transmute([]u8)text),serial.File_Error.None) - if present {testing.expect_value(t,serial.file_write_atomic(source,transmute([]u8)string("a source")),serial.File_Error.None)} + folder:=base.test_folder(t,"registry-listed");if folder=="" {return} + root,plugin,staging:=fmt.tprintf("%s/game/assets",folder),fmt.tprintf("%s/pack/assets",folder),fmt.tprintf("%s/staging",folder) + for made in ([]string{fmt.tprintf("%s/props/crate",root),plugin,staging}) {os.make_directory_all(made)} + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + wood,lid,later,atlas,own:=base.Asset_Id{1,1},base.Asset_Id{3,3},base.Asset_Id{7,7},base.Asset_Id{5,5},base.Asset_Id{4,4} + blob :: proc(t:^testing.T,types:^Types,s:^store.Store,staging:string,id:base.Asset_Id) -> store.Hash { + buffer:[base.ASSET_ID_TEXT]u8 + path:=fmt.tprintf("%s/%s.tex",staging,base.asset_id_format(id,buffer[:])) + write_texture(t,types,path,id) + hash,error:=store.put_file(s,path) + testing.expect_value(t,error,store.Error.None) + return hash } - sidecar(t,fmt.tprintf("%s/props/wood.png",root),"",true,{{kind=KIND_TEXTURE,name="wood",id=wood}}) - sidecar(t,fmt.tprintf("%s/props/crate.glb",root),"crate",true,{{kind=KIND_SCENE,name="crate",id={8,8}},{kind=KIND_TEXTURE,name="body",id=body,part=true},{kind=KIND_TEXTURE,name="lid",id=lid},{kind=KIND_TEXTURE,name="lid",index=1,id=second},{kind=KIND_TEXTURE,name="later",id=unbuilt}}) - sidecar(t,fmt.tprintf("%s/props/crate.obj",root),"crate",true,{{kind=KIND_TEXTURE,name="body",id=other}}) - sidecar(t,fmt.tprintf("%s/props/gone.png",root),"",false,{{kind=KIND_TEXTURE,name="gone",id=lost}}) - sidecar(t,fmt.tprintf("%s/atlas.png",plugin),"",true,{{kind=KIND_TEXTURE,name="atlas",id=atlas}}) - Built :: struct {imported:string,id:base.Asset_Id,key:u64,listed:string} - built:=[]Built{ - {imported,wood,0x11,"props/wood.tex"},{imported,body,0x22,"props/crate/body (crate.glb).tex"},{imported,lid,0x22,"props/crate/lid.tex"}, - {imported,second,0x22,"props/crate/lid (1).tex"},{imported,other,0x55,"props/crate/body (crate.obj).tex"},{plugin_imported,atlas,0x33,"plugins/pack/atlas.tex"},{imported,lost,0x44,""}, + pin :: proc(t:^testing.T,s:^store.Store,p:^Package,built:[]store.Hash,folder:string) -> Packages { + manifest,error:=package_put(s,p) + testing.expect_value(t,error,Error.None) + b:=Build{package_id=p.id,final=true} + b.outputs=make([dynamic]Build_Output,context.temp_allocator) + for hash,at in built { + info,_:=kind_info(p.assets[at].kind) + if hash!={} {append(&b.outputs,Build_Output{id=p.assets[at].id,kind=p.assets[at].kind,kind_version=info.version,blob=hash})} + } + hash,_:=build_put(s,&b) + digits:[64]u8 + testing.expect_value(t,store.ref_set(s,REF_BUILDS,store.hex(build_select(manifest,p),&digits),hash),store.Error.None) + list:Packages + list.packages=make([dynamic]Project_Package,context.temp_allocator) + append(&list.packages,Project_Package{id=p.id,manifest=manifest,folder=folder}) + return list } - for b in built {testing.expect_value(t,texture_save(imported_file(b.imported,b.id,b.key,".tex",context.temp_allocator),&types,{id=b.id,width=2,height=2,format=.RGBA8,mips={pixels[:]}}),Error.None)} - write_material(t,&types,fmt.tprintf("%s/props/crate/Wood.mat",root),{4,4},wood) + crate:=Package{id={20,20},name="crate",identity=store.address_of_bytes(transmute([]u8)string("crate"))} + crate.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&crate.assets,Package_Asset{kind=KIND_SCENE,name="crate",id={8,8}},Package_Asset{kind=KIND_TEXTURE,name="wood",id=wood},Package_Asset{kind=KIND_TEXTURE,name="lid",id=lid},Package_Asset{kind=KIND_TEXTURE,name="later",id=later}) + wood_blob,lid_blob:=blob(t,&types,&s,staging,wood),blob(t,&types,&s,staging,lid) + crate_list:=pin(t,&s,&crate,{{},wood_blob,lid_blob,{}},"props/crate") + pack:=Package{id={30,30},name="pack",identity=store.address_of_bytes(transmute([]u8)string("pack"))} + pack.assets=make([dynamic]Package_Asset,context.temp_allocator) + append(&pack.assets,Package_Asset{kind=KIND_TEXTURE,name="atlas",id=atlas}) + pack_list:=pin(t,&s,&pack,{blob(t,&types,&s,staging,atlas)},"") + os.remove_all(staging) + resolved,resolve_error:=packages_resolve(&s,&crate_list,context.temp_allocator) + plugin_resolved,plugin_error:=packages_resolve(&s,&pack_list,context.temp_allocator) + if !testing.expect(t,resolve_error==.None && plugin_error==.None && len(resolved.listed)==3 && len(plugin_resolved.listed)==1) {return} + // A file of the project at the path of a listed output keeps it. + write_texture(t,&types,fmt.tprintf("%s/props/crate/lid.tex",root),own) + write_material(t,&types,fmt.tprintf("%s/props/crate/Wood.mat",root),{9,9},wood) + buffer:[base.ASSET_ID_TEXT]u8 + renamed:=fmt.tprintf("props/crate/lid (%s).tex",base.asset_id_format(lid,buffer[:])) + wood_file,_:=store.blob_path(&s,wood_blob,context.temp_allocator) + lid_file,_:=store.blob_path(&s,lid_blob,context.temp_allocator) cache:=fmt.tprintf("%s/registry.cache",folder) for pass in 0..<2 { - r:Registry;registry_init(&r,root,cache,imported=imported);defer registry_destroy(&r) - testing.expect(t,registry_mount(&r,plugin,"plugins/pack",plugin_imported)) + r:Registry;registry_init(&r,root,cache,listed=resolved.listed[:]);defer registry_destroy(&r) + testing.expect(t,registry_mount(&r,plugin,"plugins/pack",plugin_resolved.listed[:])) testing.expect_value(t,registry_scan(&r),Error.None) testing.expectf(t,pass==0 || r.stats.opened==0,"the second scan reads its cache: %v",r.stats) - for b in built { - file,found:=registry_find(&r,b.id) - if b.listed=="" { - testing.expect(t,!found,"what a source that is gone gave is not listed") - continue - } - if !testing.expectf(t,found && file.path==b.listed && file.imported==b.key,"%v is listed at %s, not %s",b.id,b.listed,file.path) {continue} - testing.expect_value(t,registry_file_path(&r,file.path,context.temp_allocator),imported_file(b.imported,b.id,b.key,".tex",context.temp_allocator)) + testing.expect_value(t,r.stats.files,6) + Want :: struct {id:base.Asset_Id,path,file:string} + for w in ([]Want{{wood,"props/crate/wood.tex",wood_file},{lid,renamed,lid_file},{own,"props/crate/lid.tex",fmt.tprintf("%s/props/crate/lid.tex",root)}}) { + file,found:=registry_find(&r,w.id) + if !testing.expectf(t,found && file.path==w.path && file.state==.Asset && file.listed==(w.id!=own),"%v is listed at %s, not %v",w.id,w.path,found?file^:Registry_File{}) {continue} + testing.expect_value(t,registry_file_path(&r,file.path,context.temp_allocator),w.file) + } + shipped,found_atlas:=registry_find(&r,atlas) + testing.expect(t,found_atlas && shipped.path=="plugins/pack/atlas.tex" && shipped.listed) + data,read_error:=registry_read(&r,"props/crate/wood.tex") + a:Asset + if testing.expect_value(t,read_error,Error.None) && testing.expect_value(t,asset_open(&a,data),Error.None) { + testing.expect(t,a.blob.header.id==wood,"a listed output loads from its blob") + asset_close(&a) } - _,pending:=registry_find(&r,unbuilt) + delete(data) + pending,listed_pending:=registry_find_path(&r,"props/crate/later.tex") + testing.expect(t,listed_pending && pending.state==.Not_Built && pending.listed && pending.id==later && pending.kind==KIND_TEXTURE,"an output not built is listed as such") + _,by_id:=registry_find(&r,later) + _,not_built:=registry_read(&r,"props/crate/later.tex") + testing.expect(t,!by_id && not_built==.Not_Found) missing:=make([dynamic]Registry_Missing,context.temp_allocator) registry_missing(&r,&missing) - testing.expect(t,!pending && len(missing)==0,"an output that is not built is not listed, and one in the imported folder is not missing") + testing.expect_value(t,len(missing),0) + folders:=[]string{"plugins/","plugins/pack/","props/","props/crate/"} + testing.expectf(t,len(r.refused)==0 && len(r.folders)==len(folders),"%v",r.folders[:]) + for f,at in r.folders[:min(len(folders),len(r.folders))] {testing.expect_value(t,f,folders[at])} + testing.expect_value(t,registry_memory_set(&r,"props/crate/wood.tex",data),Error.Invalid) + testing.expect_value(t,registry_memory_remove(&r,"props/crate/wood.tex"),Error.Invalid) if pass==0 {continue} - // The source moves with its sidecar: its assets are listed there once the editor refreshes what moved, as they are. - for end in ([2]string{"",SIDECAR_EXTENSION}) {testing.expect(t,os.rename(fmt.tprintf("%s/props/crate.glb%s",root,end),fmt.tprintf("%s/kit/box.glb%s",root,end))==nil)} - testing.expect_value(t,registry_refresh(&r,{"props/crate.glb","kit/box.glb"+SIDECAR_EXTENSION}),Error.None) - moved,again:=registry_find(&r,body) - testing.expect(t,again && moved.path=="kit/crate/body.tex" && moved.imported==0x22,"the asset is listed where its source is, with its id and its file") - alone,listed:=registry_find(&r,other) - testing.expect(t,listed && alone.path=="props/crate/body.tex","a path no other output has takes no source name") + + // A refresh keeps what is listed, also what it names. + revision:=r.revision + testing.expect_value(t,registry_refresh(&r,{"props/crate/wood.tex","props/crate/later.tex",renamed,"plugins/pack/atlas.tex","props/crate/Wood.mat"}),Error.None) + testing.expect(t,r.revision==revision && r.stats.files==6 && r.stats.opened==0) + _,kept:=registry_find(&r,wood) + _,kept_pending:=registry_find_path(&r,"props/crate/later.tex") + testing.expect(t,kept && kept_pending) + // A file written where an output is listed takes the path over, through a scan. + write_texture(t,&types,fmt.tprintf("%s/props/crate/wood.tex",root),{11,11}) + testing.expect_value(t,registry_refresh(&r,{"props/crate/wood.tex"}),Error.None) + moved,again:=registry_find(&r,wood) + written,taken:=registry_find(&r,{11,11}) + testing.expect(t,again && moved.path==fmt.tprintf("props/crate/wood (%s).tex",base.asset_id_format(wood,buffer[:])) && taken && written.path=="props/crate/wood.tex" && !written.listed) + os.remove(fmt.tprintf("%s/props/crate/wood.tex",root)) + } +} + +// Entries no package list makes are refused or reported, never a crash nor +// an error of the scan: a path that is empty, leaves the folder or is one a +// scan leaves out, and a zero id are refused; a blob that is gone is Not_Built; +// one that is not the asset its entry names is Broken; two entries with one +// path are told apart by id. +@(test) +registry_refuses_or_reports_hostile_listed_entries :: proc(t:^testing.T) { + types:Types;types_init(&types);defer types_destroy(&types) + folder:=base.test_folder(t,"registry-listed-hostile");if folder=="" {return} + root:=fmt.tprintf("%s/assets",folder) + os.make_directory_all(root) + texture,other,text:=fmt.tprintf("%s/texture.tex",folder),fmt.tprintf("%s/other.tex",folder),fmt.tprintf("%s/text.tex",folder) + write_texture(t,&types,texture,{1,1}) + write_texture(t,&types,other,{2,2}) + serial.file_write_atomic(text,transmute([]u8)string("not an asset")) + entry :: proc(path,file:string,id:base.Asset_Id) -> Package_Listed {return {path=path,file=file,id=id,kind=KIND_TEXTURE,state=file!=""?.Built:.Not_Built}} + entries:=[]Package_Listed{ + entry("","",{3,3}),entry("../out.tex",texture,{1,1}),entry("/abs.tex",texture,{1,1}),entry("a\\b.tex",texture,{1,1}), + entry("a/./b.tex",texture,{1,1}),entry(".hidden/c.tex",texture,{1,1}),entry("zero.tex",texture,{}),entry("c:x.tex",texture,{1,1}), + entry("gone.tex",fmt.tprintf("%s/absent.tex",folder),{4,4}),entry("text.tex",text,{5,5}),entry("liar.tex",other,{6,6}), + entry("twice.tex",texture,{1,1}),entry("twice.tex","",{7,7}),entry("twice.tex","",{7,7}), + } + r:Registry;registry_init(&r,root,listed=entries);defer registry_destroy(&r) + testing.expect_value(t,registry_scan(&r),Error.None) + testing.expectf(t,len(r.refused)==9,"refused: %v",r.refused[:]) + buffer:[base.ASSET_ID_TEXT]u8 + State :: struct {path:string,state:Registry_State} + for want in ([]State{{"gone.tex",.Not_Built},{"text.tex",.Broken},{"liar.tex",.Broken},{"twice.tex",.Asset},{fmt.tprintf("twice (%s).tex",base.asset_id_format({7,7},buffer[:])),.Not_Built}}) { + file,found:=registry_find_path(&r,want.path) + testing.expectf(t,found && file.state==want.state && file.listed,"%s is %v, not %v",want.path,want.state,found?file^:Registry_File{}) } + testing.expect_value(t,len(r.files),5) + _,error:=registry_read(&r,"gone.tex") + testing.expect_value(t,error,Error.Not_Found) } diff --git a/asset/sidecar.odin b/asset/sidecar.odin index 75d64a25d..fde0a177a 100644 --- a/asset/sidecar.odin +++ b/asset/sidecar.odin @@ -12,10 +12,8 @@ import "mirai:std/serial" // source's folder its outputs go to, the files the import reads besides the // source and one output per asset the source gives, with its id. What nobody // edits (import_cached) is no file of the assets but of the imported folder -// that goes with them, -, never versioned: the registry -// lists it at //, so a source -// that moves with its sidecar takes it along, and one that is gone takes it -// out of the listing. +// that goes with them, -, never versioned. The registry +// reads neither: it lists the outputs of packages (packages.odin). SIDECAR_EXTENSION :: ".import" SIDECAR_VERSION :: 3 @@ -190,25 +188,6 @@ output_read :: proc(r:^serial.Text_Reader,allocator:=context.allocator) -> (outp return output,true } -// The paths the registry lists the cache outputs of a sidecar at, under -// listed (the path of the sidecar's folder in the registry, "" or ending -// with '/'): / with the name as safe_path gives it, -// and " ()" before the extension for an output whose kind and name one -// of a lower index has. "" for an output that is not of the cache. In the -// temp allocator. -sidecar_paths :: proc(sidecar:Sidecar,listed:string) -> []string { - paths:=make([]string,len(sidecar.outputs),context.temp_allocator) - folder:=sidecar.folder!=""?fmt.tprintf("%s%s/",listed,sidecar.folder):listed - for output,at in sidecar.outputs { - info,known:=kind_info(output.kind) - if !known || !import_cached(output.kind) {continue} - repeated:=false - for other in sidecar.outputs {repeated||=other.kind==output.kind && other.name==output.name && other.index (problem:st p.root=strings.clone(opened.root) p.name=strings.clone(base.text_get(opened.manifest.name[:])) p.assets_root=project.folder_path(p.root,.Assets) - asset.registry_init(&p.registry,p.assets_root,imported=project.imported_path(p.root,context.temp_allocator)) - for mount in host.project_asset_mounts(&opened,context.temp_allocator) {asset.registry_mount(&p.registry,mount.folder,mount.prefix,mount.imported)} + asset.registry_init(&p.registry,p.assets_root) + for mount in host.project_asset_mounts(&opened,context.temp_allocator) {asset.registry_mount(&p.registry,mount.folder,mount.prefix,mount.listed)} asset.types_init(&p.types) p.types.codecs=asset.project_codecs(root,engine) p.assets={&p.registry,&p.types} diff --git a/cook/build/cook.odin b/cook/build/cook.odin index 2e8caac9b..2936d411c 100644 --- a/cook/build/cook.odin +++ b/cook/build/cook.odin @@ -192,8 +192,8 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { if problem!="" {cook_fail(&report,.Manifest_Invalid,problem,allocator);return} registry:asset.Registry - asset.registry_init(®istry,desc.assets,"",temp,desc.root!=""?project.imported_path(desc.root,temp):"") - for folder in desc.plugins {asset.registry_mount(®istry,folder.path,folder.prefix,folder.imported)} + asset.registry_init(®istry,desc.assets,"",temp) + for folder in desc.plugins {asset.registry_mount(®istry,folder.path,folder.prefix)} defer asset.registry_destroy(®istry) if asset.registry_scan(®istry)!=.None {cook_fail(&report,.Assets_Missing,desc.assets,allocator);return} files:=registry.files[:] diff --git a/cook/build/world.odin b/cook/build/world.odin index 43479a0fd..e1da3d55d 100644 --- a/cook/build/world.odin +++ b/cook/build/world.odin @@ -3371,10 +3371,10 @@ asset_paths_check :: proc(g:^Generator,p:^project.Project) { assets:=project.folder_path(p.root,.Assets,context.temp_allocator) if !os.is_dir(assets) {return} registry:asset.Registry - asset.registry_init(®istry,assets,"",context.temp_allocator,project.imported_path(p.root,context.temp_allocator)) + asset.registry_init(®istry,assets,"",context.temp_allocator) defer asset.registry_destroy(®istry) for folder in project.asset_folders(p,context.temp_allocator) { - if folder.prefix!="" {asset.registry_mount(®istry,folder.path,folder.prefix,folder.imported)} + if folder.prefix!="" {asset.registry_mount(®istry,folder.path,folder.prefix)} } if asset.registry_scan(®istry)!=.None {return} extensions:=make(map[string]bool,context.temp_allocator) diff --git a/cook/import/identity.odin b/cook/import/identity.odin index 72984590e..31c301434 100644 --- a/cook/import/identity.odin +++ b/cook/import/identity.odin @@ -62,7 +62,7 @@ writer_source :: proc(w:^Writer,path,folder:string) -> bool { w.registry=w.options.registry if w.registry==nil { w.registry=new(asset.Registry,context.temp_allocator) - asset.registry_init(w.registry,project.folder_path(w.options.root,.Assets,context.temp_allocator),"",context.temp_allocator,w.imported) + asset.registry_init(w.registry,project.folder_path(w.options.root,.Assets,context.temp_allocator),"",context.temp_allocator) asset.registry_scan(w.registry) } return true @@ -126,7 +126,7 @@ parts_plan :: proc(w:^Writer,parts:[]Part) { p.part=p.decided && w.sidecar.outputs[entry].part p.path=imported_file(w,p.id,p.extension) case: - if file,found:=asset.registry_find(w.registry,p.id);found && file.kind==p.kind && file.imported==0 { + if file,found:=asset.registry_find(w.registry,p.id);found && file.kind==p.kind && !file.listed { p.path=asset.registry_file_path(w.registry,file.path,context.temp_allocator) p.decided,p.part=true,file.owner!={} } else { diff --git a/cook/import/outputs_test.odin b/cook/import/outputs_test.odin index dda99e784..2e9232061 100644 --- a/cook/import/outputs_test.odin +++ b/cook/import/outputs_test.odin @@ -164,7 +164,7 @@ the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { // The registry lists the cache where the names say. r:asset.Registry - asset.registry_init(&r,assets,allocator=context.temp_allocator,imported=fmt.tprintf("%s/.mirai/imported",root)) + asset.registry_init(&r,assets,allocator=context.temp_allocator) testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) listed:=make(map[string]base.Asset_Id,context.temp_allocator) for file in r.files {listed[file.path]=file.id} diff --git a/cook/import/sources/reimport_test.odin b/cook/import/sources/reimport_test.odin index cde5d4d6d..feac42a86 100644 --- a/cook/import/sources/reimport_test.odin +++ b/cook/import/sources/reimport_test.odin @@ -423,12 +423,12 @@ a_checkout_without_its_imported_folder_gets_the_same_assets_from_its_sources :: Listed :: struct {id:base.Asset_Id,path:string} listing :: proc(t:^testing.T,root:string) -> (listed:[dynamic]Listed,missing:int) { r:asset.Registry - asset.registry_init(&r,fmt.tprintf("%s/assets",root),imported=fmt.tprintf("%s/.mirai/imported",root)) + asset.registry_init(&r,fmt.tprintf("%s/assets",root)) defer asset.registry_destroy(&r) testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) listed=make([dynamic]Listed,context.temp_allocator) for file in r.files { - if file.imported!=0 {append(&listed,Listed{file.id,strings.clone(file.path,context.temp_allocator)})} + if file.listed {append(&listed,Listed{file.id,strings.clone(file.path,context.temp_allocator)})} } lost:=make([dynamic]asset.Registry_Missing,context.temp_allocator) asset.registry_missing(&r,&lost) diff --git a/cook/import/sources/sync.odin b/cook/import/sources/sync.odin index 78edde95e..4d94a7875 100644 --- a/cook/import/sources/sync.odin +++ b/cook/import/sources/sync.odin @@ -95,7 +95,7 @@ sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) - imported:=project.imported_path(top,context.temp_allocator) assets:=project.folder_path(top,.Assets,context.temp_allocator) registry:asset.Registry - asset.registry_init(®istry,assets,"",context.temp_allocator,imported) + asset.registry_init(®istry,assets,"",context.temp_allocator) asset.registry_scan(®istry) import_options:=importer.Import_Options{ root=top,registry=®istry,rebuild=options.rebuild,worker=options.worker,fast_textures=options.fast_textures, diff --git a/cook/import/sources/write_test.odin b/cook/import/sources/write_test.odin index 02463ac23..844a45586 100644 --- a/cook/import/sources/write_test.odin +++ b/cook/import/sources/write_test.odin @@ -219,7 +219,7 @@ the_prefab_of_an_import_is_written_in_the_layout_of_the_game :: proc(t:^testing. if !testing.expectf(t,error.code==.None,"%s: %s",c.files[0],error.message) {return} defer importer.report_destroy(&report) p:=author.Project{name="layout",assets_root=assets,info=world} - asset.registry_init(&p.registry,assets,imported=fmt.tprintf("%s/.mirai/imported",root)) + asset.registry_init(&p.registry,assets) asset.types_init(&p.types) defer {asset.registry_destroy(&p.registry);asset.types_destroy(&p.types);author.scratch_world_free(&p.scratch)} p.assets={&p.registry,&p.types} @@ -881,7 +881,7 @@ every_fixture_imports_writes_and_reads_back_with_its_dependencies_resolved :: pr } testing.expectf(t,counts[0]==len(model.meshes) && counts[1]==len(model.materials),"%s: every mesh and material is an asset",source) registry:asset.Registry - asset.registry_init(®istry,fmt.tprintf("%s/assets",out),imported=fmt.tprintf("%s/.mirai/imported",out)) + asset.registry_init(®istry,fmt.tprintf("%s/assets",out)) testing.expect_value(t,asset.registry_scan(®istry),asset.Error.None) missing:[dynamic]asset.Registry_Missing missing.allocator=context.temp_allocator diff --git a/editor/asset_delete.odin b/editor/asset_delete.odin index 6cb7b2816..010461a33 100644 --- a/editor/asset_delete.odin +++ b/editor/asset_delete.odin @@ -78,7 +78,7 @@ asset_delete_users :: proc(s:^Session,r:^asset.Registry,files:[]u32) -> []string @(private) asset_delete_refusal :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> string { if refusal:=asset_owned_refusal(r,file);refusal!="" {return refusal} - if file.imported!=0 { + if file.listed { if sidecar,_:=import_root(s,r,file.id);sidecar=="" {return "It is built by the import of its source, which makes it again: delete or move the asset that source gives (its prefab), change the source, or duplicate it to have a copy of its own"} } if watching(s) {return "The run of an agent is being watched; stop watching it first"} diff --git a/editor/asset_move.odin b/editor/asset_move.odin index 726e1a4da..3601d9890 100644 --- a/editor/asset_move.odin +++ b/editor/asset_move.odin @@ -84,7 +84,7 @@ asset_held_refusal :: proc(r:^asset.Registry,paths:[]string,folder:="") -> strin // Why an asset cannot be renamed or moved now; "" when it can. @(private) asset_move_refusal :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> string { - if file.imported!=0 {return "It is a part of what its source gives, listed in the folder of its source: move that folder, or duplicate it to have a copy that moves"} + if file.listed {return "It is a part of what its source gives, listed in the folder of its source: move that folder, or duplicate it to have a copy that moves"} if refusal:=asset_create_refusal(s);refusal!="" {return refusal} if refusal:=asset_delete_refusal(s,r,file);refusal!="" {return refusal} files:=asset_delete_files(r,file.id) @@ -103,7 +103,7 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess file,found:=asset.registry_find(r,id) if !found {return "","The asset is not in the registry: asset.list lists them"} // What a source gives is listed where the source is: its source moves, with its sidecar, and nothing is imported. - if sidecar,_:=import_root(s,r,id);file.imported!=0 && sidecar!="" { + if sidecar,_:=import_root(s,r,id);file.listed && sidecar!="" { into,refused:=asset_folder_clean(r,folder) if refused!="" {return "",fmt.tprintf("folder: %s",refused)} if busy:=import_busy(s);busy!="" {return "",busy} @@ -130,7 +130,7 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess moves:=make([dynamic]Asset_Move,context.temp_allocator) for at,i in asset_delete_files(r,id) { // A part of the imported folder stays at the path its sidecar lists. - if i>0 && r.files[at].imported!=0 {continue} + if i>0 && r.files[at].listed {continue} from:=strings.clone(r.files[at].path,context.temp_allocator) to:=asset_create_join(into,named,extension) if i>0 {to=asset_create_join(into,from[len(source)+1:] if source!="" && asset_path_inside(from,source) else asset_path_base(from),"")} diff --git a/editor/assets_parts.odin b/editor/assets_parts.odin index b3ccce939..b940fe111 100644 --- a/editor/assets_parts.odin +++ b/editor/assets_parts.odin @@ -30,7 +30,7 @@ asset_detach :: proc(s:^Session,id:base.Asset_Id) -> (owner:base.Asset_Id,messag if !found {return {},fmt.tprintf("No asset has the id %s; asset.list lists them",base.asset_id_format(id,buffer[:]))} path:=file.path if file.owner=={} {return {},fmt.tprintf("%s is a part of no asset: it already is one of its own",path)} - if file.imported!=0 {return {},fmt.tprintf("%s is built by the import of its source, whose sidecar says it is a part: import the source with parts none, or duplicate the asset to have one of its own",path)} + if file.listed {return {},fmt.tprintf("%s is built by the import of its source, whose sidecar says it is a part: import the source with parts none, or duplicate the asset to have one of its own",path)} owner=file.owner bytes,error:=asset.registry_read(r,path,context.temp_allocator) if error==.None {error=asset.header_owner_set(bytes,{})} diff --git a/editor/import.odin b/editor/import.odin index 8990bfaa7..63c8123e6 100644 --- a/editor/import.odin +++ b/editor/import.odin @@ -421,16 +421,16 @@ import_root :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (sidecar:s if error!=.None || !imported {return} source:=base.text_get(info.source[:]) root,in_assets:=sources.source_root(full,context.temp_allocator) - if file.imported!=0 || !in_assets {root=s.project_root} + if file.listed || !in_assets {root=s.project_root} for mount in r.mounts { - if file.imported!=0 && strings.has_prefix(file.path,mount.prefix) {root=filepath.dir(mount.folder)} + if file.listed && strings.has_prefix(file.path,mount.prefix) {root=filepath.dir(mount.folder)} } source=project.path(root,source,allocator=context.temp_allocator) listed,has,load_error:=importer.sidecar_load(source,context.temp_allocator) if load_error.code!=.None || !has || len(listed.outputs)==0 || listed.outputs[0].id!=id {return} files:=make([dynamic]u32,context.temp_allocator) for output in listed.outputs { - if at,known:=r.by_id[output.id];known && r.files[at].imported!=0 {append(&files,at)} + if at,known:=r.by_id[output.id];known && r.files[at].listed {append(&files,at)} } return strings.concatenate({source,importer.SIDECAR_EXTENSION},context.temp_allocator),files[:] } @@ -581,7 +581,7 @@ import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_I library,error:=host.project_library_load(path,0) if error!=.None {return {problem=host.library_error_text(error,path)}} held.library=library - asset.registry_init(&held.registry,project.folder_path(root,.Assets,context.temp_allocator),"",allocator,project.imported_path(root,context.temp_allocator)) + asset.registry_init(&held.registry,project.folder_path(root,.Assets,context.temp_allocator),"",allocator) asset.registry_scan(&held.registry) found:[PROVIDER_CAPACITY]Import_Format_Provider held.providers=slice.clone(found[:provided_in(&held.library.info,found[:])],allocator) diff --git a/host/assets.odin b/host/assets.odin index 2cc11ebd7..29b1abbcd 100644 --- a/host/assets.odin +++ b/host/assets.odin @@ -272,10 +272,10 @@ assets_init :: proc(a:^Assets,desc:Assets_Desc) -> bool { allocator:=desc.allocator.procedure!=nil?desc.allocator:context.allocator capacity:=desc.capacity>0?desc.capacity:ASSETS_DEFAULT_CAPACITY a^={allocator=allocator,renderer=desc.renderer} - listed:=asset.registry_init(&a.registry,desc.root,desc.cache,allocator,desc.imported) + listed:=asset.registry_init(&a.registry,desc.root,desc.cache,allocator) // A copy of an asset made outside the editor gets an id of its own. a.registry.resolve_duplicates=true - for mount in desc.mounts {listed&&=asset.registry_mount(&a.registry,mount.folder,mount.prefix,mount.imported)} + for mount in desc.mounts {listed&&=asset.registry_mount(&a.registry,mount.folder,mount.prefix,mount.listed)} asset.types_init(&a.types,allocator) a.index=make(map[base.Asset_Id]Assets_Ref,allocator) a.derived=make(map[u64]u32,allocator) diff --git a/host/content.odin b/host/content.odin index 130b9805b..1681263e0 100644 --- a/host/content.odin +++ b/host/content.odin @@ -813,7 +813,7 @@ content_collect :: proc(app:^App) -> (unloaded:int) { project_asset_mounts :: proc(p:^project.Project,allocator:=context.allocator) -> []asset.Registry_Mount { mounts:=make([dynamic]asset.Registry_Mount,allocator) for folder in project.asset_folders(p,allocator) { - if folder.prefix!="" {append(&mounts,asset.Registry_Mount{folder=folder.path,prefix=folder.prefix,imported=folder.imported})} + if folder.prefix!="" {append(&mounts,asset.Registry_Mount{folder=folder.path,prefix=folder.prefix})} } // The far meshes the editor made, where the assets are not cooked: a game that runs them shows them. assets:=project.folder_path(p.root,.Assets,context.temp_allocator) diff --git a/tools/mirai/asset.odin b/tools/mirai/asset.odin index f6f475e9a..c69f654a3 100644 --- a/tools/mirai/asset.odin +++ b/tools/mirai/asset.odin @@ -251,7 +251,7 @@ asset_ls :: proc(_:string,args:[]string) -> int { } if folder=="" {return misuse("ls")} r:asset.Registry - asset.registry_init(&r,folder,cache,imported=imported_beside(folder)) + asset.registry_init(&r,folder,cache) if error:=asset.registry_scan(&r);error!=.None {return failed("ls",folder,error_text(error))} w:serial.Writer;serial.writer_init(&w) problems:=0 @@ -304,7 +304,7 @@ asset_deps :: proc(_:string,args:[]string) -> int { return 0 } r:asset.Registry - asset.registry_init(&r,folder,imported=imported_beside(folder)) + asset.registry_init(&r,folder) if error:=asset.registry_scan(&r);error!=.None {return failed("deps",folder,error_text(error))} id:=a.blob.header.id missing:=0 From 0ece3fd4e146604dc0ca5fe6c20002c58fb31fd3 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:50:33 -0300 Subject: [PATCH 010/123] import: the writer puts its outputs and its package in the machine store An import writes every output to a scratch file and puts it in the store (Import_Options.store) as a blob; nothing goes under .mirai/imported and no sidecar is written. It records a package manifest (ids from the previous manifest, origin, identity without engine versions, settings, inputs, assets with owner and dependencies) and a build manifest (blob per output, baseline documents, encoders, Final), and names them under packages/, imports/ or imports-fast/ by asset.import_key, and builds/ by build_select. Reuse is a lookup in the builds of the package; the source an asset names is relative to its package. documents_follow writes or follows the baseline documents into a folder of a project, with today's rules; an import given documents does it. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-imports/SKILL.md | 89 +-- .../mirai-imports/references/imports.md | 197 +++---- asset/kinds.odin | 2 +- cook/import/documents.odin | 203 +++++++ cook/import/font.odin | 13 +- cook/import/identity.odin | 264 +++++---- cook/import/import.odin | 2 +- cook/import/interface_test.odin | 42 +- cook/import/model.odin | 24 +- cook/import/outputs.odin | 135 +++-- cook/import/outputs_test.odin | 324 +++++++--- cook/import/package.odin | 197 +++++++ cook/import/progress.odin | 2 +- cook/import/prompts.odin | 29 +- cook/import/reader_test.odin | 18 +- cook/import/settings.odin | 9 +- cook/import/sidecar.odin | 209 ------- cook/import/sound.odin | 15 +- cook/import/sound_test.odin | 31 +- cook/import/sources/command.odin | 3 +- cook/import/sources/sync.odin | 61 +- cook/import/space_test.odin | 6 +- cook/import/write.odin | 557 ++++++++---------- cook/prefab/prefab.odin | 4 +- editor/asset_move.odin | 5 +- editor/import.odin | 13 +- tools/demo_content/imported.odin | 10 +- 27 files changed, 1391 insertions(+), 1073 deletions(-) create mode 100644 cook/import/documents.odin create mode 100644 cook/import/package.odin delete mode 100644 cook/import/sidecar.odin diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index 145a20858..42f123a62 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -20,11 +20,11 @@ the assets. ## Map - Packages, `import_source`, the imported model and CLI: [imports.md](references/imports.md#api). -- Where a source lives (in the assets, beside what it makes), its sidecar (settings, folder, ids), - what an import keeps in the imported folder and what in the assets, reimporting, a moved or - renamed source and parts with `owner`: [Identity](references/imports.md#identity-and-reimport). -- The records that take the place of the sidecar in the machine store (package and build - manifests, `packages.mirai`): [packages](../mirai-assets/references/formats.md#packages). +- The package of a source in the store (settings, ids, build of blobs), documents written or + followed in a project (`documents_follow`), reimporting and parts with `owner`: + [Identity](references/imports.md#identity-and-reimport). +- The records of the store (package and build manifests, refs, `packages.mirai`): + [packages](../mirai-assets/references/formats.md#packages). - A checkout without its imported folder, what a build imports and when a source is imported again: [Sync](references/imports.md#sync). - Adding or changing a format (`Format`, `Reader`, `Source_Space`, version): @@ -55,13 +55,11 @@ the assets. ## Workflow -1. **Importing a file.** Put it in `assets/`, in the folder of its theme beside what it makes - (the editor copies a file from outside into the folder of the browser), and - [`mirai import`](references/imports.md#what-is-written) it; review the assets with - `mirai asset verify`, `ls` and `info`. Commit the source, its `.import` and the documents it - wrote beside it; what it wrote in `.mirai/imported/` is never versioned. The editor, the CLI, - the sync of builds and the content tools import with the same `sources.import_source`: same - assets, prefab included. +1. **Importing a file.** An import takes the store (`Import_Options.store`) and puts in it the + package of the source and a build of its blobs; given `documents`, it also writes or follows + the documents in that folder of a project. Review the assets with `mirai asset verify`, `ls` + and `info`; commit the documents, never the blobs. The editor, the CLI, the sync of builds and + the content tools import with the same `sources.import_source`: same assets, prefab included. 2. **From code.** `sources.import_source(path,options)` writes everything; for the model without writing it, `importer.import_model(path,sources.FORMATS[:],options)` reads it into an `Imported_Model` (plain data, no engine components) and `importer.import_write` writes its @@ -93,37 +91,38 @@ the assets. place after the read: from the `front` its settings state, or else, for a file that is a rig, from where its body is found facing ([front](references/imports.md#front)). - **The source is not modified.** The same bytes and options give the same assets. -- **A source lives in the assets, beside what it makes, in folders by theme.** Never a folder per - asset type; the browser filters by type. One outside the assets is refused. Decided by the - owner. -- **The ids of an import live in the sidecar of its source, and nothing derives them.** The first - import gives random ids and `.import` lists them; importing again, on any machine and - in any order of sources, gives each part the asset it had by its kind and name, even if its - document was moved, and a source moved or renamed with its sidecar keeps its import +- **A source imports from anywhere.** In a project, in a plugin or in no project: what it makes + goes to the store, and its documents to the folder of the project given. In a project, it lives + beside what it makes, in folders by theme, never a folder per asset type. +- **The ids of an import live in the manifest of its package, and nothing derives them.** The + first ingest gives random ids; importing the package again, on any machine, gives each part + the asset it had by its kind and name, even if its document was moved ([identity](references/imports.md#identity-and-reimport)). A part renamed in the source is a new asset. Only what is surely part of a model belongs to its prefab (`owner`), and it is decided once ([parts](references/imports.md#parts)). - **An import never touches another.** Two models that embed the same image get a texture each, - encoded once in the content cache; no import edits the files or the sidecar of another source. - Decided by the owner. -- **What nobody edits is a cache of the source, never versioned, whatever reads it.** The meshes, - textures, sounds and fonts of every import, of an engine format or of a plugin's, are files of - the imported folder of its root (`.mirai/imported/`), named by their ids and listed by where - their sidecar is, built again from the source and its sidecar by - [sync](references/imports.md#sync); materials, prefabs, skeletons, clips, texts and prompts, - which the editor edits, are documents of `assets/`. Decided by the owner. -- **The source, its sidecar and the build decide the cache.** The same three give the same files, - byte for byte: a clone, a worktree and a branch switch are brought up to date by the same sync, - explicitly, in `mirai build`; the editor never imports by itself. + encoded once in the content cache; no import edits the documents or the package of another + source. Decided by the owner. +- **Every output is an immutable blob of the store, the same on every machine.** Nothing under + `.mirai/imported/` and no sidecar: the build lists the blobs, documents among them as their + baseline. No blob is rewritten in place (a changed owner is a new blob), and the source an asset + names is relative to its package. Materials, prefabs, skeletons, clips, texts and prompts, which + the editor edits, are also documents of a project, followed from the baseline. Decided by the + owner. +- **The identity holds no engine version; the key holds the identity and the versions.** The + identity (source, inputs, settings, plugin code) is in the manifest; the key + (`asset.import_key`) adds the writer, format, kind and texture builder versions, so any machine + with the manifest finds a build without the source. A Final build is named under `imports`, + which peers share; a Fast one only under `imports-fast`; `builds` names the one this machine + chose. The editor never imports by itself. - **Reimporting keeps what a person changed.** A material stores what the source gave and the next import rewrites only that: fields changed afterwards stay, like the overrides of a prefab ([materials](references/imports.md#what-is-written)). - **What an asset was built from is its `Import_Info`**: source (size and hash), writer and format - version, and only the settings the asset is made from. Whether a file of the cache is current - is said once, by its name (the key of what the import builds from), and a mesh or a texture - whose file under another key records what it would be built from now takes that file; the rest - is rebuilt, and a texture of the same image and settings or a mesh of the same geometry and - settings comes from the content cache. + version, and only the settings the asset is made from. Reuse is a ref lookup: a mesh or a + texture that a build of the package has, recording what it would be built from now, is taken; + the rest is rebuilt, and a texture of the same image and settings or a mesh of the same + geometry and settings comes from the content cache. - **In parallel, the same bytes.** What is expensive (levels, clusters and colliders of a mesh, levels of a texture) is built in jobs; what decides ids and files is written in a single thread, in the usual order. Importing with or without the job system gives the same files. @@ -132,8 +131,8 @@ the assets. order of the scalar loop: the tool and the dev engine (`-o:minimal`) write the same as `--release`. - **Importing is a transaction.** `import_model` leaves nothing if it fails (own arena, freed by - `model_destroy`); `import_write` removes the files it created if it fails midway. Errors name - the file, the line or the prim. + `model_destroy`); `import_write` removes the documents it created and names nothing if it fails + midway. Errors name the file, the line or the prim. - **A source does not choose what is read from disk.** A path with a control character or, on Windows, a device (`CON`, `NUL`, `COM1`) is not opened. Models only ask for files next to them or by path, never more than `MAX_SOURCE_BYTES`, and the hierarchy is bounded (`MAX_DEPTH`). @@ -156,10 +155,10 @@ the assets. `TEXCOORD_0/1`, Draco, meshopt indices and normals, GPU instancing, subdivision surfaces and time samples fail with a named error. What is ignored without breaking the source (material extensions with no equivalent, missing images) leaves a warning. -- **Settings live with the source.** `.import` decides how it is imported and wins over the - options of whoever imports (except `ignore_stored`); every import writes it. Everything that - changes an asset is a setting, also the alpha coverage of a loose image. An unreadable sidecar - is an error, not the defaults nor new ids. +- **Settings live in the package.** Its manifest decides how the source is imported again and wins + over the options of whoever imports (except `ignore_stored`). Everything that changes an asset + is a setting, also the alpha coverage of a loose image. An unreadable manifest is an error, not + the defaults nor new ids. - **What is imported is used as is.** Levels of detail are made on import and stored in the `.mesh`; loading does not recompute them. - **Collision is the source's, as authored.** The colliders the source brings are the mesh's and @@ -216,9 +215,11 @@ the assets. gets the same assets byte for byte and only what changed is imported again, the sources of a plugin format build in the cache and sync reads them with the formats of the plugins, a source outside the assets is refused, when a material is a part of a model, sync runs the importer of - a plugin for a whole source whose inputs changed), `cook/import/outputs_test.odin` (the models - of a whole source share their parts by name, names with `/` go to subfolders, inputs count in - the key), + a plugin for a whole source whose inputs changed), `cook/import/outputs_test.odin` (the outputs + of a source are the assets its package lists, the models of a whole source share their parts by + name, inputs count in the identity, importing again keeps ids and blobs, a source outside any + project imports, a setting changes the identity and an engine version only the key, a Fast + build is not shared, two stores hold the same blobs), `asset/registry_test.odin` (the listing of the cache by where the sidecar is, never by the order of a scan), `cook/import/sources/settings_test.odin` (a sidecar that is not one is refused) and `cook/import/sources/write_test.odin` (importing again keeps what was changed in a material and diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index 2415c7556..9df8aa8ad 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -18,10 +18,9 @@ Packages, all *engine-time*: - `sources.import_source` imports a file of the assets folder of a project or a plugin: a model writes its assets and, if it has nodes, its prefab; an image, its `.tex`; a sound, a font or a - prompt pack, its asset. The [sidecar](#identity-and-reimport) of the source - keeps its settings, the folder of its assets and the id of each asset: importing it again gives - the same assets. `sources.sync` imports every source of a root whose outputs are missing or - stale ([sync](#sync)). + prompt pack, its asset. The [package](#identity-and-reimport) of the source in the store keeps + its settings and the id of each asset: importing it again gives the same assets. `sources.sync` + imports every source of a root whose outputs are missing or stale ([sync](#sync)). - `sources.source_kind` says what a file is from its first bytes ([sniffing](#formats)), except what the signature does not settle: TGA by extension and prompts by `.prompts.json`. It carries the formats it searches with (`formats`, `FORMATS` by default), like `model_folder` @@ -30,23 +29,24 @@ Packages, all *engine-time*: (`sources.FORMATS[:]` or those of a test) into an `Imported_Model` in its own arena (`model_destroy`); `importer.import_write` writes its assets and returns the `Import_Report`. `prefab.model_import` does both with the prefab (`cook/prefab.model_prefab`). -- `Outputs` (`output_root`, `output_part`, `output_model`, `output_input`): the assets of an - import that is not that of one model (a scene, a source a plugin imports whole) are those the - sidecar of their source lists, like those of a model: the root by its kind (and its name when - the sidecar lists several of the kind) and every other by kind and name; `outputs_end` writes - the sidecar. `output_model` writes a model as parts of the import with the writer of - `import_write` (cooking ahead, Fast textures, kept files, the caches); its prefab is a scene of - the model's name that the caller writes with `prefab.model_prefab`. A part whose kind and name - the import already gave is that asset, written once: two models that name the same material - or texture share it. `output_input` lists a file the import reads in the sidecar and in its - key; every input comes before the first output of the cache, which fixes the key. +- `Outputs` (`output_root`, `output_part`, `output_model`, `output_input`, `output_written`): the + assets of an import that is not that of one model (a scene, a source a plugin imports whole) are + those the package of their source lists, like those of a model: the root by its kind (and its + name when the package lists several of the kind) and every other by kind and name. The caller + writes each to the file its part gives and `output_written` takes it; `outputs_end` puts the + manifest and the build in the store. `output_model` writes a model as parts of the import with + the writer of `import_write` (cooking ahead, Fast textures, taken blobs, the caches); its prefab + is a scene of the model's name that the caller writes with `prefab.model_prefab`. A part whose + kind and name the import already gave is that asset, written once: two models that name the + same material or texture share it. `output_input` lists a file the import reads in the package + and in its identity; every input comes before the first output of the cache, which fixes it. - For engine-time tools: `mesh_levels` ([levels](#levels-of-detail)) and the PBR conversions of `pbr.odin`. - `Error` is `{code,message}`; `message` lives in the caller's `temp_allocator`. `Import_Options` - carries the settings and how they apply: `ignore_stored`, `root` (the project or the plugin - whose assets folder has the source; `import_source` finds it from the path), `registry` (the - scanned assets of the root, else the import scans them), `rebuild`, the caches and `worker` - ([in parallel](#in-parallel)). + carries the settings and how they apply: `ignore_stored`, `store` (required), `manifest` (the + package to ingest again), `documents` (the folder of a project its documents go to), `root` and + `registry` (the project and its scanned assets, where documents are found by id), `rebuild`, + the caches and `worker` ([in parallel](#in-parallel)). ## Formats @@ -172,10 +172,11 @@ part ([extension](../../mirai-editor/references/extensions.md#importers-of-a-plu ## Import cache -The `Import_Info` of each asset says what it was built from: source (relative to `root`), name, +The `Import_Info` of each asset says what it was built from: source (relative to its package: the +name of its file), name, index, size and hash of what was read (the source; the image of a texture; the mesh as read, with its slots, joints and skeleton, of a mesh), version and `settings_hash`. It is not what identifies -the asset: the [sidecar](#identity-and-reimport) is. +the asset: the [package](#identity-and-reimport) is. - `importer_version` combines `IMPORTER_VERSION` (the writer) with `Format.version` (`import_version`); loose images and sounds use format version 0. @@ -184,71 +185,63 @@ the asset: the [sidecar](#identity-and-reimport) is. (`mesh_build_hash`); for a texture, its `Build_Options` with `BUILD_VERSION` and the block encoders (`textures.encoders_id`); for a sound, whether it is `Stream`. Materials, skeletons and clips store 0. Changing a texture setting keeps the meshes, and one of the meshes, the textures. -- What is current is decided once, by the name of a file of the cache: `import_write` keeps a mesh - or a texture whose file under the key of what the import builds from now exists - (`Written.kept`) and builds nothing ahead when every output of the cache is kept; if only the - `owner` changes, it rewrites the header. A mesh or a texture with no file under that key takes - the file of its id under another key, renamed to it, when that file is of the version its kind - is written in and its `Import_Info` is what the import records now: a change of the key that - does not reach it (the code of a plugin, a setting of other assets) neither builds nor writes - it. A texture of an import of fast textures takes one built Fast only while the texture cache - holds no Final of it. Materials, skeletons, clips and prefabs, which are - cheap, are always written, with what a person changed in them ([what is - written](#what-is-written)), also with `rebuild`. -- A change of writer or format version, of the code of the plugin that imports a source whole, - of the settings or of `kind_version` of a kind of the cache - ([version rule](../../mirai-assets/references/formats.md#versions)) changes the key and - rebuilds; `rebuild` (`mirai import --rebuild`) rebuilds everything. +- What is current is a ref lookup: `import_write` takes the blob a build of the package has for a + mesh or a texture (`Written.kept`), from the build of the key of what the import is made from + now, else from the build this machine chose for the previous manifest, when the blob is of the + version its kind is written in, has the `owner` decided now and its `Import_Info` is what the + import records now: a change that does not reach it (the code of a plugin, a setting of other + assets) neither builds nor writes it. It builds nothing ahead when the build of the key has + every output of the cache. A texture of an import of fast textures takes one built Fast only + while the texture cache holds no Final of it. Materials, skeletons, clips and prefabs, which + are cheap, are always written, also with `rebuild`. +- A change of writer or format version, of a kind version or of the builder of textures changes + the key (`asset.import_key` of the identity and `import_versions`); one of the source, an + input, the settings or the code of the plugin that imports it whole changes the identity too; + `rebuild` (`mirai import --rebuild`) rebuilds everything. ## Identity and reimport -A source lives in the assets folder of a project or a plugin, beside what it makes, in the folder -of its theme (never a folder per asset type: the browser filters by type). `.import`, the -sidecar, is what is versioned of an import with its source (`asset/sidecar.odin`, read by the -registry and by the importer alike): a text with the settings, the `folder` beside the source its -assets go to (a model's: its stem, or what `Format.folder` says; an image's, a sound's: none, they -go beside it), the files the import read besides the source (`inputs`) and one output per asset -the source gives, by kind, name and index, with its id. Nothing else decides an id: the first -import gives each part a random one and the sidecar keeps it; importing again, on any machine and -in any order of sources, gives each part the asset it had. A sidecar that does not parse is an -error that names it, never the defaults nor new ids. A source in no assets folder is refused, -saying where it goes; `import_source` finds its root (the project or plugin) from its path. - -What an import writes is of two sorts (`asset.import_cached`), whatever the format that reads the -source, the engine's or a plugin's: - -- **The cache**: meshes, textures, sounds and fonts. Nothing edits them after the import, so the - source and the sidecar give them back whole. Each is a file of the imported folder of the root - (`project.imported_path`: `.mirai/imported/`), named by its id and by the key of what it was - built from (`import_key`), never versioned. Nothing stores where it is listed: the - [registry](../../mirai-assets/references/formats.md#registry) lists it at - `//`. [Sync](#sync) builds it again where it is - missing. -- **Documents**: materials, prefabs, skeletons, clips, texts and prompts. The editor edits them, - so they are files of the assets folder, versioned. An import finds each by its id in the - registry of the root, wherever a person moved it, writes again what the source gives and keeps - what was changed ([what is written](#what-is-written)); a new one goes to the folder of the - import. -- **A name with `/`** is a folder under that of the import, for the cache as for documents: - each part made safe (`asset.safe_path`), without empty, `.` and `..` parts. +An import ingests its source as a package of the store +([packages](../../mirai-assets/references/formats.md#packages)), from wherever the source is: in +the assets of a project, of a plugin or in no project. Its manifest is what identifies the +import: the origin (where the source was, a hint), the identity (`identity_of`: the bytes of the +source and its inputs, its settings and the code of a plugin that imports it whole, no engine +version), the settings, the inputs and one asset per thing the source gives, by kind, name and +index, with its id, owner and dependencies. Nothing else decides an id: the first ingest gives +each part a random one; importing again (`Import_Options.manifest`, else the manifest the ref +`packages` of this machine names for the path of the source) gives each part the asset it had. A +manifest that does not read is an error that names it, never the defaults nor new ids. + +What an import writes is a blob of its build (`writer_put`), whatever the format that reads the +source, the engine's or a plugin's; each part is written to a scratch file first, and the store +dedupes. Blobs never change in place and are the same bytes on every machine: the source an +asset names is relative to the package, a changed owner is a new blob. The build lists the blob of +each output, which encoders built it and whether it is Final; it is named under `imports/` +when Final, else under `imports-fast/` (never shared), and as the build this machine chose +(`builds/`), which a program without importers finds. Of two sorts +(`asset.import_cached`): + +- **The cache**: meshes, textures, sounds and fonts. Nothing edits them after the import: the + project lists them from its packages. +- **Documents**: materials, prefabs, skeletons, clips, texts and prompts. The build keeps them as + the source gives them (their baseline); a project edits a file of each, written or followed in + a folder of its assets by `documents_follow` (`cook/import/documents.odin`), or by the import + itself when given `documents`: found by its id in the registry of the root wherever a person + moved it, else a new file there ([what is written](#what-is-written)). The prefab of a model is + written in the types of the game, by the caller, at `Import_Report.prefab_path`. +- **A name with `/`** is a folder under that of the documents: each part made safe + (`asset.safe_path`), without empty, `.` and `..` parts. - **The root** of an import is the prefab of a model, the texture of an image or the sound of a - sound, without `owner`: the output of its kind the sidecar lists. A model with nodes that has + sound, without `owner`: the asset of its kind the package lists. A model with nodes that has none receives a new prefab. -- **A part** is the output of its kind and name (among equal names, the one of its index); one the - sidecar does not list is new. A part renamed in the source is a new asset: what named the old - one does not silently get another. An output the source no longer gives leaves the sidecar with - a warning: its file goes with it when it is of the cache, and stays as a plain asset when it is - a document. -- **A source moved or renamed** keeps its import as long as its sidecar goes with it, and takes - what it has in the cache with it without an import: the registry lists those assets where the - sidecar is at its next scan, which a refresh of a sidecar or of its source triggers (the editor - moves a source with its sidecar, and a folder with its sources). A source that is deleted takes - them out of the listing at once. A copy of the source without its sidecar is another import, - with ids of its own. +- **A part** is the asset of its kind and name (among equal names, the one of its index); one the + package does not list is new. A part renamed in the source is a new asset: what named the old + one does not silently get another. An asset the source no longer gives leaves the package with + a warning; a document of it stays in a project as a plain asset. - **An import never touches another.** Two models that embed the same image get a texture asset - each; the [texture cache](#texture-cache) keeps one encoded copy. No import writes a file or a - sidecar of another source. Decided by the owner. + each; the [texture cache](#texture-cache) keeps one encoded copy. No import writes a document + or a package of another source. Decided by the owner. - **The textures of a model** name the model as source and the image as name, with the size and hash of the image. @@ -275,8 +268,8 @@ what surely belongs to that model is a part; what is doubtful stays independent edges do not tile` or `of its own: `. - **It is decided once.** A document keeps the `owner` its file has: a part taken out of its owner (`asset.detach` in the editor, which only clears `owner`) is still an output of the import, which - writes it and leaves it outside. An output of the cache keeps what its `part` says in the - sidecar. Only `parts` decides it again. + writes it and leaves it outside. An asset keeps what its package says (`part` of the cache, + `owner` of a document). Only `parts` decides it again. - `Import_Settings.parts` (`mirai import --parts auto|all|none`): `Auto` the above; `All` makes parts of all the materials and textures of the model, and `None` of none, also of those that already existed. @@ -414,7 +407,7 @@ pass), one `.mesh` per key with its levels, clusters and colliders, without mate ## Settings of a source `Import_Settings` is how a source becomes assets; its zero value is the default of each setting and -its fields are the options of `mirai import` (`IMPORT_USAGE`). The sidecar writes only the settings +its fields are the options of `mirai import` (`IMPORT_USAGE`). The package keeps only the settings that differ from their default, so it shows what was chosen. What is not obvious: - `image` says how a loose image is read; `Normal_Green_Down` is a normal map with green pointing @@ -436,16 +429,16 @@ that differ from their default, so it shows what was chosen. What is not obvious materials, skeletons, meshes or prefab. It is how the motion of another file (a set of clips exported with a body) joins a character imported before. In the editor it is `skeleton` of `asset.import`, the way to give it to a [plugin format](#plugin-formats). The reader sees it in - `r.options.settings` (the settings the import applies, those of `.import` when it has - one): a format whose clips keep no skeleton of their own (Cast) reads that `.skeleton` to bring - its keys to the frames of its bones, and adds its bones as a skeleton of the model, which is - not written. + `r.options.settings` (the settings the import applies, those of its package when it was + ingested before): a format whose clips keep no skeleton of their own (Cast) reads that + `.skeleton` to bring its keys to the frames of its bones, and adds its bones as a skeleton of + the model, which is not written. -Settings live in the [sidecar](#identity-and-reimport) of the source, one entry per field name (an +Settings live in the [package](#identity-and-reimport) of the source, one entry per field name (an absent field is its default), and win over the options except with `ignore_stored`: importing a -source again gives the same assets whoever does it. An unreadable file (unknown field or value, -negative number, coverage outside 0 to 1) is `Malformed`, never the defaults. Every import writes -its sidecar when it ends, and does not touch it if it already says the same. +source again gives the same assets whoever does it. Unreadable settings (unknown field or value, +negative number, coverage outside 0 to 1) are `Malformed`, never the defaults. Every import puts +its manifest in the store when it ends. ## The imported model @@ -458,13 +451,13 @@ those a mesh uses; repeated names are told apart with `name (n)`. ## What is written -What is new gets a flat file in the folder of the import (`.mat`, `.skeleton`, `.anim` and, -with `import_source`, the model's `.scene`), with a valid file name that does not overwrite -another asset; what is of the cache is a file of the imported folder named by its id. What already -existed is written where it is ([identity](#identity-and-reimport)). +Every output is a blob of the build. In the folder of the documents, a new document gets a flat +file (`.mat`, `.skeleton`, `.anim` and, with `import_source`, the model's `.scene`), with a +valid file name that does not overwrite another asset; one that already existed is followed where +it is ([identity](#identity-and-reimport)). Each imported asset carries the `Cooked` flag and its `Import_Info` ([cache](#import-cache)). -- **Ids and owner.** Random the first time and those of the sidecar afterwards; the +- **Ids and owner.** Random the first time and those of the package afterwards; the [parts](#parts) name their prefab as `owner`. Dependencies come from the `Asset_Id`s that each asset names. - **Compression.** Texels go in BC blocks ([textures](#textures-cooktextures)); the bulk columns of @@ -475,11 +468,13 @@ Each imported asset carries the `Cooked` flag and its `Import_Info` ([cache](#im previous import it rewrites `imported` and follows the source except in the fields that differ from what it gave before (`asset.material_follow`, the algorithm of prefab overrides); the parameters of a program stay the same by name, and `overlays` and `hidden` are kept: what was - changed after importing stays and a new texture of the source arrives. The skeleton likewise - keeps the body parts set by hand. + changed after importing stays and a new texture of the source arrives. A skeleton keeps the body + parts set by hand and the sockets added on bones it still has, and a clip the root set by hand + while its bone is in its skeleton. A document keeps the `owner` its file has. The follow starts + from the baseline: the grips of a hand are those of the body found on the bones. - **Images that cannot be used** leave a warning and the map without a texture; a loose image that - does not decode fails. If `import_write` fails, it deletes what it created in that call; what - already existed is replaced one at a time with atomic writes. + does not decode fails. If `import_write` fails, it deletes the documents it created in that call + and names nothing; what already existed is replaced one at a time with atomic writes. - `Written.detail` says what each asset has (triangles per level and colliders of a mesh; format, size and mips of a texture; entities of a prefab), also for a kept one. @@ -489,11 +484,11 @@ file is in the assets folder of a project or a plugin; it imports where it is. - It first builds the engine if it is not up to date, in dev unless `--release` is given; what that build says goes to stderr and stdout is left with the report. -- It starts from the settings of the sidecar, applies the given options (`IMPORT_USAGE`) and writes - the sidecar again: repeating the command without options gives the same assets. A source a format +- It starts from the settings of the package of the source, applies the given options + (`IMPORT_USAGE`): repeating the command without options gives the same assets. A source a format of a plugin reads takes the formats of the editor library of its project. - The report gives one line per asset (`id kind bytes path (detail)`), how many were already up to - date, the sidecar and the warnings. Exit codes: 0, 1 (failed) and 2 (usage). + date and the warnings. Exit codes: 0, 1 (failed) and 2 (usage). ## Template rigs (`cook/rig`) diff --git a/asset/kinds.odin b/asset/kinds.odin index dd99b4c82..3b8b68f95 100644 --- a/asset/kinds.odin +++ b/asset/kinds.odin @@ -141,7 +141,7 @@ version_message :: proc(a:^Asset,types:^Types,path:string,current:u16=0,allocato // Where an asset built from a source file came from: what an importer needs // to know whether it is out of date. It is the row of the table "import". Import_Info :: struct { - source:[260]u8 `mirai:"text"`, // path of the source file, relative to the project + source:[260]u8 `mirai:"text"`, // path of the source file, relative to its package name:[64]u8 `mirai:"text"`, // what was taken from the source: a mesh, a material, an image index:i32, // its position among those of its kind in the source source_size:u64, // bytes diff --git a/cook/import/documents.odin b/cook/import/documents.odin new file mode 100644 index 000000000..11f20ad6d --- /dev/null +++ b/cook/import/documents.odin @@ -0,0 +1,203 @@ +package importer + +import "core:fmt" +import "core:os" +import "core:path/filepath" +import "core:strings" +import "mirai:asset" +import anim "mirai:std/anim" +import "mirai:std/base" +import "mirai:std/serial" +import "mirai:store" + +// The documents of a package (materials, skeletons, clips, scenes) are edited +// in each project that uses it. Its build keeps them as the source gives them, +// their baseline; a project gets a file of each, written from the baseline the +// first time and followed on every later import: the file keeps what was +// changed in it after the last import. A material keeps the fields changed +// over what its source gave, its overlays and hidden names; a skeleton keeps +// the body parts set by hand and the sockets added to it on bones it still +// has; a clip keeps the root set by hand while its bone is still in its +// skeleton. Any document keeps the owner its file has, so one taken out of the +// package stays out. + +// The baseline of a skeleton the clips followed with it play on. +Followed_Skeleton :: struct { + id:base.Asset_Id, + path:string, +} + +// A document documents_follow wrote. +Followed :: struct { + id:base.Asset_Id, + kind:asset.Fourcc, + path:string, + warning:string, // what of the file could not be kept; "" when all +} + +// Writes into folder the documents of the build b of the package p, or +// follows those whose ids registry has where it has them. A new file is +// named as the package names its document, under folder. What it wrote, in +// allocator. B4: the per-project step of an ingest calls it. +documents_follow :: proc(s:^store.Store,p:^asset.Package,b:^asset.Build,folder:string,registry:^asset.Registry,allocator:=context.allocator) -> (written:[dynamic]Followed,error:Error) { + written=make([dynamic]Followed,allocator) + types:asset.Types + asset.types_init(&types,context.temp_allocator) + defer asset.types_destroy(&types) + skeletons:=make([dynamic]Followed_Skeleton,context.temp_allocator) + for o in b.outputs { + if o.kind!=asset.KIND_SKELETON {continue} + if blob,path_error:=store.blob_path(s,o.blob,context.temp_allocator);path_error==.None {append(&skeletons,Followed_Skeleton{o.id,blob})} + } + taken:=make(map[string]bool,context.temp_allocator) + for o in b.outputs { + if asset.import_cached(o.kind) {continue} + blob,path_error:=store.blob_path(s,o.blob,context.temp_allocator) + if path_error!=.None {return written,error_make(.Not_Found,"the store at %s has no blob of the document %v of the package %v: fetch it from a peer that has it",s.root,o.id,p.id,allocator=context.temp_allocator)} + target:string + if registry!=nil { + if file,found:=asset.registry_find(registry,o.id);found && file.kind==o.kind {target=asset.registry_file_path(registry,file.path,context.temp_allocator)} + } + if target=="" { + name:="unnamed" + for &a in p.assets { + if a.id==o.id {name=asset.safe_path(a.name)} + } + info,_:=asset.kind_info(o.kind) + for at:=1;;at+=1 { + candidate:=fmt.tprintf("%s/%s%s",folder,name,info.extension) if at==1 else fmt.tprintf("%s/%s (%d)%s",folder,name,at,info.extension) + key:=strings.to_lower(candidate,context.temp_allocator) + if taken[key] || os.exists(candidate) {continue} + taken[key]=true + target=candidate + break + } + if under:=filepath.dir(target);!os.is_dir(under) {_=os.make_directory_all(under)} + } + warning,follow_error:=document_follow(&types,o.kind,blob,target,skeletons[:],registry) + if follow_error!=.None {return written,error_make(.Write,"%s could not be written: %v",target,follow_error,allocator=context.temp_allocator)} + append(&written,Followed{id=o.id,kind=o.kind,path=strings.clone(target,allocator),warning=strings.clone(warning,allocator)}) + } + return written,{} +} + +// Writes the document whose baseline is the file at baseline to target, or +// follows the file target has. skeletons: the baselines the clips of an +// animation may play on, else registry has them. The warning, in the temp +// allocator, says what of the file could not be kept. +document_follow :: proc(types:^asset.Types,kind:asset.Fourcc,baseline,target:string,skeletons:[]Followed_Skeleton,registry:^asset.Registry) -> (warning:string,error:asset.Error) { + given:asset.Asset + asset.asset_load(&given,baseline,context.temp_allocator) or_return + defer asset.asset_close(&given) + previous:asset.Asset + existed:=os.exists(target) && asset.asset_load(&previous,target,context.temp_allocator)==.None + defer if existed {asset.asset_close(&previous)} + owner:=previous.blob.header.owner if existed else given.blob.header.owner + id:=given.blob.header.id + switch kind { + case asset.KIND_MATERIAL: + view:=asset.material_view(&given,types) or_return + desc:=asset.Material_Desc{id=id,owner=owner,info=view.info^,imported=view.imported,parameters=view.parameters,imported_parameters=view.imported_parameters,source=view.source} + if existed && previous.blob.header.id==id { + if kept,kept_error:=asset.material_view(&previous,types);kept_error==.None { + desc.info=asset.material_follow(types,kept,view.imported^ if view.imported!=nil else view.info^) + desc.parameters=asset.material_parameters_follow(kept,view.imported_parameters,context.temp_allocator) + desc.overlays,desc.hidden=kept.overlays,kept.hidden + } + } + return "",asset.material_save(target,types,desc) + case asset.KIND_SKELETON: + view:=asset.skeleton_view(&given,types) or_return + kept:^asset.Skeleton_View + kept_view:asset.Skeleton_View + if existed { + view_error:asset.Error + kept_view,view_error=asset.skeleton_view(&previous,types) + if view_error==.None {kept=&kept_view} + if view_error==.Version {warning=fmt.tprintf("%s is of a version this build does not read: the body parts, axes and sockets set by hand on it were not kept; set them again",target)} + } + body:asset.Skeleton_Body + body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part + count,_:=skeleton_body(kept,view.bones,&body,&body_parts) + sockets:=make([dynamic]asset.Skeleton_Socket,context.temp_allocator) + append(&sockets,..view.sockets) + skeleton_sockets_kept(kept,view.bones,&sockets) + return warning,asset.skeleton_save(target,types,{id=id,owner=owner,bones=view.bones,sockets=sockets[:],body=&body if count>0 else nil,body_parts=body_parts[:count],source=view.source}) + case asset.KIND_ANIMATION: + view:=asset.animation_view(&given,types) or_return + clips,editable:=asset.animation_descs(view,context.temp_allocator) + if !editable {break} + if existed { + kept,view_error:=asset.animation_view(&previous,types) + if view_error==.Version {warning=fmt.tprintf("%s is of a version this build does not read: the root bones set by hand on its clips were not kept; set them again",target)} + for &clip in clips { + for &old in kept.clips { + if base.text_get(old.name[:])!=clip.name || !old.root.translation && !old.root.rotation {continue} + bones,found:=skeleton_bones(types,clip.skeleton,skeletons,registry) + if !found || asset.skeleton_bone_index(bones,old.root.bone)>=0 {clip.root=old.root} + break + } + } + } + return warning,asset.animation_save(target,types,{id=id,owner=owner,clips=clips,source=view.source}) + } + // Any other document is the baseline, with the owner of its file. + data,read_error:=serial.file_read(baseline,context.temp_allocator) + if read_error!=.None {return "",.IO} + if owner!=given.blob.header.owner {asset.header_owner_set(data,owner) or_return} + if serial.file_write_atomic(target,data)!=.None {return "",.IO} + return "",.None +} + +// The bones of the skeleton id: a baseline of skeletons, else the file +// registry has. In the temp allocator. +@(private="file") +skeleton_bones :: proc(types:^asset.Types,id:base.Asset_Id,skeletons:[]Followed_Skeleton,registry:^asset.Registry) -> (bones:[]asset.Skeleton_Bone,found:bool) { + path:string + for s in skeletons { + if s.id==id {path=s.path} + } + if path=="" && registry!=nil { + if file,known:=asset.registry_find(registry,id);known {path=asset.registry_file_path(registry,file.path,context.temp_allocator)} + } + if path=="" {return} + a:asset.Asset + if asset.asset_load(&a,path,context.temp_allocator)!=.None {return} + view,error:=asset.skeleton_view(&a,types) + if error!=.None {return} + return view.bones,true +} + +// The sockets of the file an import replaces that it does not make again (one a person or an agent +// added, such as where a weapon hangs), on a bone it still has. Those it makes (named by the +// source, bone.*, grips) are made again from the source. +@(private) +skeleton_sockets_kept :: proc(previous:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,sockets:^[dynamic]asset.Skeleton_Socket) { + if previous==nil {return} + made:=len(sockets) + for &socket in previous.sockets { + name:=base.text_get(socket.name[:]) + if strings.has_prefix(name,"bone.") {continue} + again:=false + for &other in sockets[:made] {if other.name==socket.name {again=true;break}} + if again {continue} + for &bone in bones {if bone.path==socket.bone {append(sockets,socket);break}} + } +} + +// The body of an imported skeleton: found on its bones, keeping the parts a +// person set in the file it replaces when their bones are still there. +// Returns how many parts it wrote, 0 for a skeleton without a body, and the map they come from. +@(private) +skeleton_body :: proc(previous_file:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,body:^asset.Skeleton_Body,parts:^[len(anim.Body_Part)]asset.Skeleton_Body_Part) -> (count:int,map_:anim.Body_Map) { + previous:=anim.body_map_empty() + if previous_file!=nil { + previous=asset.skeleton_body_map(bones,previous_file.body,previous_file.body_parts) + for part in anim.Body_Part {if part not_in previous.manual {previous.bones[part],previous.confidence[part]=-1,0}} + } + found:bool + map_,found=asset.skeleton_body_detect(bones,previous) + if !found {return 0,map_} + count,_=asset.skeleton_body_rows(bones,map_,body,parts) + return count,map_ +} diff --git a/cook/import/font.odin b/cook/import/font.odin index ab1937e6e..f7d79f765 100644 --- a/cook/import/font.odin +++ b/cook/import/font.odin @@ -2,7 +2,6 @@ package importer import "core:fmt" import "core:mem" -import "core:os" import "core:path/filepath" import "mirai:asset" import "mirai:std/base" @@ -49,20 +48,20 @@ import_font :: proc(path:string,options:Import_Options={},allocator:=context.all report_init(&report,allocator) w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { source:=w.source name:=filepath.stem(path) info:=import_info(source,name,0,u64(len(bytes)),base.hash64(bytes),0,import_version(0)) desc:=asset.Font_Desc{file=bytes,source=&info,compression=.Stream} target:string - desc.id,target=root_take(&w,asset.KIND_FONT,name,".font") - existed:=os.exists(target) + output:int + desc.id,target,output=root_take(&w,asset.KIND_FONT,name,".font") if save_error:=asset.font_save(target,&w.types,desc,context.temp_allocator);save_error!=.None { ok=save_failed(&w,save_error,target) - } else { - record(&w,desc.id,asset.KIND_FONT,name,target,source,existed,fmt.tprintf("%v outlines, %d glyphs of the atlas",outlines,glyphs)) - ok=writer_finish(&w) + } else if ok=writer_put(&w,output,target);ok { + record(&w,output,name,fmt.tprintf("%v outlines, %d glyphs of the atlas",outlines,glyphs)) + ok=writer_finish(&w,name,"font") } } if !ok { diff --git a/cook/import/identity.odin b/cook/import/identity.odin index 72984590e..dc7c318d2 100644 --- a/cook/import/identity.odin +++ b/cook/import/identity.odin @@ -8,13 +8,16 @@ import "core:strings" import "mirai:asset" import "mirai:project" import "mirai:std/base" +import "mirai:store" -// Which asset each part of a source is and where its file is. The sidecar of -// the source says the asset: the output of its kind and name (of its kind for -// the root of the import), else a new one. An asset of the cache is a file of -// the imported folder of the root named by its id and the key of what the -// import builds from; a document is where the registry of the root has its -// id, else a new file in the folder of the import. +// Which asset each part of a source is and where it is written. The manifest +// of the package the source was ingested as (package_previous) says the +// asset: the one of its kind and name (of its kind for the root of the +// import), else a new one. Every part is written to a file of the scratch +// folder of the import and put in the store as a blob of its build; a +// document is also written, or followed, in the folder of the documents +// (Import_Options.documents): where the registry of the root has its id, else +// a new file there. // What an import writes, before its file and id are known. @(private) @@ -24,187 +27,210 @@ Part :: struct { index:int, // among those of its kind in the source extension:string, only:bool, // the one of its kind, whatever its name, as the root (the clips of a model, named after it) - output:int, // what the sidecar the import writes lists for it (Writer.outputs) + output:int, // what the manifest the import writes lists for it (Writer.outputs) given:bool, // an output this import gave before (Writer.given), written then decided:bool, // whether it is a part of the root was decided by an import before: part says it part:bool, id:base.Asset_Id, owner:base.Asset_Id, - path:string, + path:string, // where it is written, in the scratch folder of the import + target:string, // of a document: its file in the folder of the documents; "": none } -// Reads the sidecar of the source at path, which says which asset each thing -// of the source is, and the folder of the source new documents go to: the -// one its sidecar keeps, else folder. The source is in the assets of the -// project or the plugin at options.root. +// Starts the import of the source at path: the manifest it was ingested as, +// the scratch folder its parts are written in and, with +// options.documents, the folder of the documents (folder under it: where +// those of a first import go, "" in it) and the assets of the root they are +// found among. @(private) writer_source :: proc(w:^Writer,path,folder:string) -> bool { w.path=path w.keep=context.temp_allocator - w.source=source_path(w.options,path) + w.origin=path_key(path) + w.source=filepath.base(w.origin) w.inputs.allocator=context.temp_allocator - names:=project.FOLDER_NAMES - if w.options.root=="" {return writer_fail(w,.Not_Found,"%s: an import names the project or the plugin whose assets the source is in (root)",path)} - if !strings.has_prefix(w.source,names[.Assets]) || !strings.has_prefix(w.source[len(names[.Assets]):],"/") { - return writer_fail(w,.Not_Found,"%s is not in the %s folder of %s: move it there, with its %s when it has one, beside what it makes",path,names[.Assets],w.options.root,SIDECAR_EXTENSION) - } - // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). - sidecar,found,error:=sidecar_load(path,context.temp_allocator) + w.outputs.allocator=context.temp_allocator + w.targets.allocator=context.temp_allocator + w.paths.allocator=context.temp_allocator + w.blobs.allocator=context.temp_allocator + w.fixed=false + s:=w.options.store + if s==nil {return writer_fail(w,.Write,"%s: an import puts what it builds in a store: give it one (Import_Options.store)",path)} + previous,manifest,found,error:=package_previous(s,w.options.manifest,w.origin,context.temp_allocator) if error.code!=.None {return writer_fail(w,error.code,"%s",error.message)} - w.sidecar=sidecar - w.stored=found - if !found {w.sidecar.folder=folder} - w.folder=filepath.dir(path) if w.sidecar.folder=="" else fmt.tprintf("%s/%s",filepath.dir(path),w.sidecar.folder) - w.taken=make([]bool,len(w.sidecar.outputs),context.temp_allocator) - w.outputs=make([dynamic]Output,context.temp_allocator) - w.paths=make([dynamic]string,context.temp_allocator) - w.imported=project.imported_path(w.options.root,context.temp_allocator) - w.registry=w.options.registry - if w.registry==nil { - w.registry=new(asset.Registry,context.temp_allocator) - asset.registry_init(w.registry,project.folder_path(w.options.root,.Assets,context.temp_allocator),"",context.temp_allocator,w.imported) - asset.registry_scan(w.registry) + w.previous,w.previous_manifest,w.stored=previous,manifest,found + w.package_id=previous.id if found else writer_id(w) + w.taken=make([]bool,len(previous.assets),context.temp_allocator) + if documents:=w.options.documents;documents!="" { + w.folder=documents if folder=="" else fmt.tprintf("%s/%s",documents,folder) + w.registry=w.options.registry + if w.registry==nil && w.options.root!="" { + w.registry=new(asset.Registry,context.temp_allocator) + asset.registry_init(w.registry,project.folder_path(w.options.root,.Assets,context.temp_allocator),"",context.temp_allocator) + asset.registry_scan(w.registry) + } } + scratch,scratch_error:=os.make_directory_temp("","mirai-import-*",w.keep) + if scratch_error!=nil {return writer_fail(w,.Write,"%s: the folder its parts are written in before the store takes them could not be made: %v",path,scratch_error)} + w.scratch=scratch return true } -// The file of an output of the cache: in the imported folder, by its id and -// the key of what the import builds from (import_key). +// The file a part is written to before the store takes it. @(private) -imported_file :: proc(w:^Writer,id:base.Asset_Id,extension:string) -> string { - if w.key==0 { - slice.sort(w.inputs[:]) - w.key=import_key(w.path,{settings=w.report.settings,inputs=w.inputs},w.version,w.options.importer_code,w.options.fast_textures) - } - return asset.imported_file(w.imported,id,w.key,extension,context.temp_allocator) +scratch_path :: proc(w:^Writer,id:base.Asset_Id,extension:string) -> string { + text:[base.ASSET_ID_TEXT]u8 + return fmt.tprintf("%s/%s%s",w.scratch,base.asset_id_format(id,text[:]),extension) } -// Notes a file the import reads besides its source, as the sidecar lists it. -// False for a new one once the key of the import is fixed: the files of the -// cache are named without it. +// Notes a file the import reads besides its source, as the manifest lists it. +// False for a new one once what the import is made from is fixed +// (writer_identity). @(private) writer_input :: proc(w:^Writer,file:string) -> bool { - relative,error:=filepath.rel(filepath.dir(path_key(w.path)),path_key(file),context.temp_allocator) + relative,error:=filepath.rel(filepath.dir(w.origin),path_key(file),context.temp_allocator) if error!=.None {return true} slashed,_:=strings.replace_all(relative,"\\","/",context.temp_allocator) if slice.contains(w.inputs[:],slashed) {return true} - if w.key!=0 {return false} + if w.fixed {return false} append(&w.inputs,strings.clone(slashed,w.keep)) return true } -// Gives each part its asset and its file: the output of the sidecar of its -// kind and name (the one of its index first), or of its kind for the only one -// of its kind, else a new id. What the import gives is listed in w.outputs, the sidecar it -// writes when it ends. A part of the kind and name of one of the first -// w.given outputs, or of the kind, name and index of a later one, is that one -// (given): an output is listed once however many times it is asked for. +// Gives each part its asset and its files: the asset of the previous manifest +// of its kind and name (the one of its index first), or of its kind for the +// only one of its kind, else a new id. What the import gives is listed in +// w.outputs, the assets of the manifest it writes. A part of the kind and name +// of one of the first w.given outputs, or of the kind, name and index of a +// later one, is that one (given): an output is listed once however many times +// it is asked for. Planning a part of the cache fixes what the import is made +// from. @(private) parts_plan :: proc(w:^Writer,parts:[]Part) { each: for &p in parts { if !p.only { for output,at in w.outputs { if output.kind!=p.kind || output.name!=p.name || at>=w.given && output.index!=p.index {continue} - p.id,p.output,p.path,p.given=output.id,at,w.paths[at],true + p.id,p.output,p.path,p.target,p.given=output.id,at,w.paths[at],w.targets[at],true continue each } } entry:=-1 - for output,at in w.sidecar.outputs { - if w.taken[at] || output.kind!=p.kind || !p.only && output.name!=p.name {continue} + for a,at in w.previous.assets { + if w.taken[at] || a.kind!=p.kind || !p.only && a.name!=p.name {continue} entry=at - if p.only || output.index==p.index {break} + if p.only || a.index==p.index {break} } p.id=writer_id(w) if entry>=0 { w.taken[entry]=true - p.id=w.sidecar.outputs[entry].id + a:=w.previous.assets[entry] + p.id,p.decided=a.id,true + p.part=a.part if asset.import_cached(p.kind) else a.owner!={} } - switch { - case asset.import_cached(p.kind): - p.decided=entry>=0 - p.part=p.decided && w.sidecar.outputs[entry].part - p.path=imported_file(w,p.id,p.extension) - case: - if file,found:=asset.registry_find(w.registry,p.id);found && file.kind==p.kind && file.imported==0 { - p.path=asset.registry_file_path(w.registry,file.path,context.temp_allocator) - p.decided,p.part=true,file.owner!={} - } else { - p.path=fresh_path(w,fmt.tprintf("%s%s",asset.safe_path(p.name),p.extension)) + p.path=scratch_path(w,p.id,p.extension) + p.target="" + if asset.import_cached(p.kind) { + writer_identity(w) + } else if w.folder!="" { + if w.registry!=nil { + if file,found:=asset.registry_find(w.registry,p.id);found && file.kind==p.kind {p.target=asset.registry_file_path(w.registry,file.path,context.temp_allocator)} } + if p.target=="" {p.target=fresh_path(w,fmt.tprintf("%s%s",asset.safe_path(p.name),p.extension))} + if folder:=filepath.dir(p.target);!os.is_dir(folder) {_=os.make_directory_all(folder)} } - if folder:=filepath.dir(p.path);!os.is_dir(folder) {_=os.make_directory_all(folder)} p.output=len(w.outputs) - append(&w.outputs,Output{kind=p.kind,name=strings.clone(p.name,w.keep),index=p.index,id=p.id}) + append(&w.outputs,asset.Package_Asset{kind=p.kind,name=strings.clone(p.name,w.keep),index=p.index,id=p.id}) append(&w.paths,strings.clone(p.path,w.keep)) + append(&w.targets,strings.clone(p.target,w.keep)) + append(&w.blobs,store.Hash{}) } } -// The id and the file of the root of an import: the output of kind its -// sidecar lists, else a new one named name. +// The id, the file and the output of the root of an import: the asset of +// kind the previous manifest lists, else a new one named name. @(private) -root_take :: proc(w:^Writer,kind:asset.Fourcc,name,extension:string) -> (id:base.Asset_Id,path:string) { +root_take :: proc(w:^Writer,kind:asset.Fourcc,name,extension:string) -> (id:base.Asset_Id,path:string,output:int) { parts:=[1]Part{{kind=kind,name=name,extension=extension,only=true}} parts_plan(w,parts[:]) - return parts[0].id,parts[0].path + return parts[0].id,parts[0].path,parts[0].output } -// Records which parts of the cache name the root of the import as their -// owner, once that is decided: what the sidecar keeps for the next import. +// Takes the file of an output written at file as a blob of the build, with the +// owner and the dependencies its header gives, and its document written or +// followed at its target (document_follow). False, with the writer failed, +// when the store does not take it or the target cannot be written. @(private) -parts_record :: proc(w:^Writer,parts:[]Part) { - for &p in parts { - if asset.import_cached(p.kind) && !p.given {w.outputs[p.output].part=p.owner!={}} - } +writer_put :: proc(w:^Writer,output:int,file:string) -> bool { + s:=w.options.store + hash,error:=store.put_file(s,file) + if error!=.None {return writer_fail(w,.Write,"%s could not be put in the store at %s: %v",file,s.root,error)} + return writer_take(w,output,hash) } -// Ends the outputs of an import that wrote its assets: what its sidecar -// listed and the source no longer gives leaves it (a document stays, a plain -// asset from then on), the files of the imported folder its outputs had under -// another key go, and the sidecar is written with the settings the source was -// imported with. An import of documents alone leaves its stamp (STAMP_EXTENSION). +// Takes the blob hash as the output of the build, from the store: its owner and +// dependencies, and its document at its target. @(private) -writer_finish :: proc(w:^Writer) -> bool { - current:=make(map[base.Asset_Id]string,context.temp_allocator) - for output in w.outputs { - info,known:=asset.kind_info(output.kind) - if known && asset.import_cached(output.kind) {current[output.id]=filepath.base(imported_file(w,output.id,info.extension))} +writer_take :: proc(w:^Writer,output:int,hash:store.Hash) -> bool { + s:=w.options.store + blob,path_error:=store.blob_path(s,hash,context.temp_allocator) + if path_error!=.None {return writer_fail(w,.Write,"the store at %s does not have the blob it took: %v",s.root,path_error)} + dependencies:=make([dynamic]base.Asset_Id,context.temp_allocator) + header,_,_,peek_error:=asset.import_peek(blob,&w.types,&dependencies) + if peek_error!=.None {return writer_fail(w,.Write,"%s, written for %s, does not read: %v",blob,w.outputs[output].name,peek_error)} + o:=&w.outputs[output] + w.blobs[output]=hash + o.owner=header.owner + o.part=asset.import_cached(o.kind) && header.owner!={} + o.dependencies=make([dynamic]base.Asset_Id,w.keep) + for d in dependencies { + if d!={} && d!=o.id && !slice.contains(o.dependencies[:],d) {append(&o.dependencies,d)} } - stamp:="" - if len(current)==0 && len(w.outputs)>0 { - stamp=imported_file(w,w.outputs[0].id,STAMP_EXTENSION) - current[w.outputs[0].id]=filepath.base(stamp) + if target:=w.targets[output];target!="" { + existed:=os.exists(target) + warning,follow_error:=document_follow(&w.types,o.kind,blob,target,w.followed[:],w.registry) + if follow_error!=.None {return save_failed(w,follow_error,target)} + if warning!="" {writer_warn(w,"%s",warning)} + if !existed {append(&w.created,strings.clone(target,w.keep))} } - for output,at in w.sidecar.outputs { + if o.kind==asset.KIND_SKELETON {append(&w.followed,Followed_Skeleton{o.id,strings.clone(blob,w.keep)})} + return true +} + +// Ends the outputs of an import that wrote its assets: what the previous +// manifest listed and the source no longer gives leaves the package (a +// document stays, a plain asset from then on), and the manifest and the build +// are put in the store and named (writer_publish). +@(private) +writer_finish :: proc(w:^Writer,name,importer:string) -> bool { + for a,at in w.previous.assets { if w.taken[at] {continue} - if asset.import_cached(output.kind) { - current[output.id]="" - writer_warn(w,"%s %s is no longer in the source: its asset is gone, and what named it names nothing",kind_name(output.kind),output.name) - } else if file,found:=asset.registry_find(w.registry,output.id);found { - writer_warn(w,"%s is no longer in the source: it was left as it is",file.path) + if asset.import_cached(a.kind) { + writer_warn(w,"%s %s is no longer in the source: its asset is gone, and what named it names nothing",kind_name(a.kind),a.name) + } else if w.registry!=nil { + if file,found:=asset.registry_find(w.registry,a.id);found {writer_warn(w,"%s is no longer in the source: it was left as it is",file.path)} } } - entries,_:=os.read_all_directory_by_path(w.imported,context.temp_allocator) - for entry in entries { - id,_,named:=asset.imported_name(entry.name) - if name,listed:=current[id];named && listed && name!=entry.name {os.remove(entry.fullpath)} - } - slice.sort(w.inputs[:]) - if !sidecar_save(w.path,{settings=w.report.settings,folder=w.sidecar.folder,inputs=w.inputs,outputs=w.outputs}) { - return writer_fail(w,.Write,"%s%s, which keeps the ids of the import, could not be written",w.path,SIDECAR_EXTENSION) - } - w.report.saved=strings.concatenate({w.path,SIDECAR_EXTENSION},w.allocator) - // Without it the next sync imports the source again: nothing is lost. - if stamp!="" { - _=os.make_directory_all(w.imported) - _=os.write_entire_file(stamp,nil) - } - return true + return writer_publish(w,name,importer) +} + +// Removes the documents the import created and the scratch folder; the blobs it +// put stay, named by nothing. +@(private) +writer_rollback :: proc(w:^Writer) { + for path in w.created {os.remove(path)} +} + +// Lets the writer go: its types and its scratch folder. +@(private) +writer_close :: proc(w:^Writer) { + if w.scratch!="" {_=os.remove_all(w.scratch)} + asset.types_destroy(&w.types) } -// The first of file (a path under the folder of the import) there, then file -// with " (2)" and on before its extension, that no part of this write took -// and no file has. +// The first of file (a path under the folder of the documents) there, then +// file with " (2)" and on before its extension, that no part of this write +// took and no file has. @(private) fresh_path :: proc(w:^Writer,file:string) -> string { slash:=strings.last_index_byte(file,'/') diff --git a/cook/import/import.odin b/cook/import/import.odin index 9e29e680b..57268be1b 100644 --- a/cook/import/import.odin +++ b/cook/import/import.odin @@ -20,7 +20,7 @@ import_model :: proc(path:string,formats:[]Format,options:Import_Options={},allo caller:=context.temp_allocator settings,stored:=options.settings,false if !options.ignore_stored { - kept,found,settings_error:=settings_load(path) + kept,found,settings_error:=settings_load(options,path) if settings_error.code!=.None {return {},settings_error} if found {settings,stored=kept,true} } diff --git a/cook/import/interface_test.odin b/cook/import/interface_test.odin index 0c10792b3..41994e623 100644 --- a/cook/import/interface_test.odin +++ b/cook/import/interface_test.odin @@ -8,31 +8,44 @@ import "mirai:asset" import cimage "mirai:cook/image" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" @(private="file") CFF_OTF :: #load("../../assets/fixtures/cff.otf") @(private="file") INTER :: #load("../../assets/fonts/Inter-Regular.ttf") // A folder of a test as the root of a project: its assets folder, where the -// sources of an import are. "" when there is none. -test_assets :: proc(t:^testing.T,name:string) -> (root,assets:string) { +// sources of an import are, and a store of its own (Import_Options.store), +// in the temp allocator. "" when there is none. +test_assets :: proc(t:^testing.T,name:string) -> (root,assets:string,s:^store.Store) { root=base.test_folder(t,name) if root=="" {return} assets=fmt.tprintf("%s/assets",root) os.make_directory_all(assets) + s=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return "","",nil} return } +// A store at folder, in the temp allocator; nil when it does not open. +test_store :: proc(t:^testing.T,folder:string) -> ^store.Store { + s:=new(store.Store,context.temp_allocator) + opened,error:=store.open(folder,context.temp_allocator) + if !testing.expect_value(t,error,store.Error.None) {return nil} + s^=opened + return s +} + @(test) fonts_import_as_the_file_they_are_and_collections_are_refused :: proc(t:^testing.T) { - root,folder:=test_assets(t,"fonts") + _,folder,s:=test_assets(t,"fonts") if folder=="" {return} for file in ([2]string{"face.otf","inter.ttf"}) { source:=fmt.tprintf("%s/%s",folder,file) _=os.write_entire_file(source,CFF_OTF if file=="face.otf" else INTER) - report,error:=import_font(source,{root=root}) + report,error:=import_font(source,{store=s}) if !testing.expectf(t,error.code==.None,"%s imports: %s",file,error.message) {continue} defer report_destroy(&report) - testing.expect(t,len(report.assets)==1 && report.assets[0].kind==asset.KIND_FONT && strings.has_suffix(report.assets[0].path,".font")) + testing.expect(t,len(report.assets)==1 && report.assets[0].kind==asset.KIND_FONT && report.assets[0].blob!={} && strings.has_prefix(report.assets[0].path,s.root)) types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) a:asset.Asset if !testing.expect_value(t,asset.asset_load(&a,report.assets[0].path),asset.Error.None) {continue} @@ -41,19 +54,20 @@ fonts_import_as_the_file_they_are_and_collections_are_refused :: proc(t:^testing testing.expect_value(t,view_error,asset.Error.None) testing.expect(t,view.info.outlines==(.CFF if file=="face.otf" else .TrueType) && len(view.file)==len(CFF_OTF if file=="face.otf" else INTER)) // Again: the same asset, with the same id. - again,again_error:=import_font(source,{root=root}) + again,again_error:=import_font(source,{store=s}) testing.expect(t,again_error.code==.None && len(again.assets)==1 && again.assets[0].id==report.assets[0].id && again.assets[0].path==report.assets[0].path) report_destroy(&again) } collection:=fmt.tprintf("%s/family.ttc",folder) _=os.write_entire_file(collection,transmute([]u8)string("ttcf\x00\x02\x00\x00\x00\x00\x00\x02............")) - _,error:=import_font(collection,{root=root}) + _,error:=import_font(collection,{store=s}) testing.expect(t,error.code==.Unsupported && strings.contains(error.message,"collection") && strings.contains(error.message,"fonttools ttLib -y 0"),error.message) broken:=fmt.tprintf("%s/broken.otf",folder) _=os.write_entire_file(broken,transmute([]u8)string("OTTO\x00\x09\x00\x80\x00\x03\x00\x10 not tables at all")) - _,error=import_font(broken,{root=root}) + _,error=import_font(broken,{store=s}) testing.expect(t,error.code==.Malformed,error.message) - testing.expect(t,!os.exists(fmt.tprintf("%s/broken.font",folder)),"A refused font writes nothing") + refs,_:=store.ref_list(s,asset.REF_PACKAGES,context.temp_allocator) + testing.expect(t,len(refs)==2,"A refused font records nothing") } @(test) @@ -155,7 +169,7 @@ prompt_images_pack_into_power_of_two_atlases :: proc(t:^testing.T) { @(test) prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { - root,folder:=test_assets(t,"prompts") + root,folder,s:=test_assets(t,"prompts") if folder=="" {return} os.make_directory_all(fmt.tprintf("%s/images",folder)) paint :: proc(path:string,width,height:int,color:[4]u8) { @@ -172,12 +186,12 @@ prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { {"family": "PlayStation", "input": "South", "image": "images/a.png"}, {"family": "Keyboard", "input": "Key", "key": "Space", "image": "images/space.png"} ]}`)) - report,error:=import_prompts(manifest,{root=root}) + report,error:=import_prompts(manifest,{store=s,root=root,documents=folder}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} defer report_destroy(&report) if !testing.expect_value(t,len(report.assets),2) {return} atlas,pack:=report.assets[0],report.assets[1] - testing.expect(t,atlas.kind==asset.KIND_TEXTURE && pack.kind==asset.KIND_PROMPTS && pack.path==fmt.tprintf("%s/pad.prompts",folder) && strings.contains(atlas.path,"/.mirai/imported/")) + testing.expect(t,atlas.kind==asset.KIND_TEXTURE && pack.kind==asset.KIND_PROMPTS && pack.path==fmt.tprintf("%s/pad.prompts",folder) && strings.has_prefix(atlas.path,s.root)) types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) a,b:asset.Asset if !testing.expect_value(t,asset.asset_load(&a,pack.path),asset.Error.None) {return} @@ -199,7 +213,7 @@ prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { testing.expect_value(t,texel(level,width,x,y),[4]u8{255,255,255,128}) testing.expect_value(t,texel(level,width,x-PROMPTS_PADDING,y-PROMPTS_PADDING),[4]u8{255,255,255,128}) // the padding repeats the edge, alpha straight // Again: the same files and ids. - again,again_error:=import_prompts(manifest,{root=root}) + again,again_error:=import_prompts(manifest,{store=s,root=root,documents=folder}) testing.expect(t,again_error.code==.None && len(again.assets)==2 && again.assets[0].id==atlas.id && again.assets[1].id==pack.id && again.assets[0].kept) report_destroy(&again) @@ -215,7 +229,7 @@ prompts_import_packs_the_images_of_a_manifest :: proc(t:^testing.T) { }) { bad:=fmt.tprintf("%s/bad.prompts.json",folder) _=os.write_entire_file(bad,transmute([]u8)c.json) - _,refused:=import_prompts(bad,{root=root}) + _,refused:=import_prompts(bad,{store=s,root=root,documents=folder}) testing.expectf(t,refused.code!=.None && strings.contains(refused.message,c.says),"%s: %s",c.json,refused.message) } } diff --git a/cook/import/model.odin b/cook/import/model.odin index bc2baf0ab..c6ba2f25d 100644 --- a/cook/import/model.odin +++ b/cook/import/model.odin @@ -7,6 +7,7 @@ import "mirai:core" import "mirai:cook/textures" import "mirai:std/base" import "mirai:std/jobs" +import mstore "mirai:store" import mm "mirai:std/math" // What an importer reads from a source file, as plain data that depends on no @@ -47,14 +48,21 @@ Error :: struct { } Import_Options :: struct { - // Replaced by the settings the sidecar of the source keeps when it has - // one, unless ignore_stored. + // Replaced by the settings the package of the source keeps when it was + // ingested before, unless ignore_stored. using settings:Import_Settings, ignore_stored:bool, - // The project or the plugin whose assets folder the source is in: sources - // are named relative to it in Import_Info, its documents are found among - // its assets, and what it builds of the cache kinds goes to its imported - // folder (sidecar.odin). + store:^mstore.Store, // where the import puts the blobs of its build and the records of its package (package.odin) + // The manifest of the package the source was ingested as, whose ids and + // settings the import keeps; zero: the one the store names for the path + // of the source (package_previous), else a new package. + manifest:mstore.Hash, + // The folder of the assets of a project the documents of the import are + // written or followed in (documents.odin), and the prefab of a model; + // "": they are blobs of the build alone. + documents:string, + // The project or the plugin the import is for: the codecs of its + // assets, and the assets its documents are found among by id. root:string, registry:^asset.Registry, // the scanned assets of root, which the import does not change; nil: it scans them world:^core.World_Info, // the types of the game of root, which the prefab of a model is written in @@ -67,7 +75,7 @@ Import_Options :: struct { mesh_cache:string, // folder of meshes built before, kept by geometry and settings (mesh_cook); "": none // The identity of the code of the plugin that imports the source whole // (sources.code_identity), 0 for the engine's: part of what its outputs - // are built from (import_key). + // are made from (identity_of). importer_code:u64, material_index:int, // the index of the first material among those of the source: imports that write its materials apart give each its own // Where the import cooks: the worker that calls it (the owner of its @@ -198,7 +206,7 @@ Imported_Model :: struct { warnings:[]string, // what the importer left out or could not honor programs:[]string, // the Surface programs reading it wrote to the shaders of the project, their files files:[]string, // the files reading it read besides the source and its images (buffers, material libraries) - settings:Import_Settings, // those of the sidecar of the source when stored is set, else those of the options it was read with + settings:Import_Settings, // those of the package of the source when stored is set, else those of the options it was read with stored:bool, arena:^mem.Dynamic_Arena, allocator:mem.Allocator, diff --git a/cook/import/outputs.odin b/cook/import/outputs.odin index 65f233e61..7d5a0287a 100644 --- a/cook/import/outputs.odin +++ b/cook/import/outputs.odin @@ -2,75 +2,83 @@ package importer import "core:fmt" import "core:mem" +import "core:path/filepath" import "core:strings" import "mirai:asset" import "mirai:std/base" // The assets of an import that is not that of one model: the scene of a // level, the source a plugin imports whole with the models it reads -// (output_model). They are the assets the sidecar of their source lists as -// those of a model are (identity.odin): the root by output_root, each other -// one by kind and name (then index). A new one gets a random id and a file of -// its own, a name with '/' in that folder under the one of the import, and -// outputs_end writes the sidecar. Outputs does not move while it is open: its +// (output_model). They are the assets of the package of their source as those +// of a model are (identity.odin): the root by output_root, each other one by +// kind and name (then index). Each is written to the file its part gives and +// taken by output_written: a blob of the build, and a document also written +// or followed in the folder of the documents. outputs_end puts the manifest +// and the build in the store. Outputs does not move while it is open: its // search keeps memory of its own. One thread at a time. Outputs :: struct { w:Writer, report:Import_Report, source:string, // the source as imports name it + importer:string, size,hash:u64, version:u32, - current:Maybe(bool), // every output of the cache the sidecar lists is current: models cook nothing ahead + current:Maybe(bool), // the store has every output of the cache of the package, built from what it is made from now: models cook nothing ahead scratch:mem.Dynamic_Arena, // the temp memory of the search } // Starts the outputs of an import of the file at source, whose bytes have // size and hash, read by a writer of version (import_version), with the -// settings its sidecar keeps unless options.ignore_stored. folder: under -// the folder of the source, where the new documents of a first import go -// ("" beside it), unless output_part names another. -outputs_begin :: proc(o:^Outputs,source,folder:string,size,hash:u64,version:u32,options:Import_Options,allocator:=context.allocator) -> Error { +// settings its package keeps unless options.ignore_stored. folder: under +// the folder of the documents, where the new documents of a first import go +// ("" in it), unless output_part names another. importer: what reads it, as +// its package records it (asset.Package_Origin.importer). +outputs_begin :: proc(o:^Outputs,source,folder:string,size,hash:u64,version:u32,options:Import_Options,importer:="",allocator:=context.allocator) -> Error { o^={size=size,hash=hash,version=version} caller:=context.temp_allocator mem.dynamic_arena_init(&o.scratch,allocator,allocator,block_size=1<<20,out_band_size=1<<20) context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) report_init(&o.report,allocator) - o.source=source_path(options,source) + o.source=source_path(source) + o.importer=strings.clone(importer,context.temp_allocator) if !writer_init(&o.w,source,folder,options,&o.report,allocator) { - error:=outputs_error(o,caller) - outputs_end(o,true) + error:=writer_error(&o.w,caller) + _,_=outputs_end(o,true) return error } - // The settings kept with the source are those of every import of it. - if o.w.stored && !options.ignore_stored {o.w.options.settings=o.w.sidecar.settings} + o.w.version=version + // The settings its package keeps are those of every import of it. + if o.w.stored && !options.ignore_stored { + settings,valid:=settings_parse(o.w.previous.settings) + if !valid { + writer_fail(&o.w,.Malformed,"the settings the package of %s keeps (%s) are not those of this build: import it as a new package",source,o.w.previous.settings) + error:=writer_error(&o.w,caller) + _,_=outputs_end(o,true) + return error + } + o.w.options.settings=settings + } o.report.settings=o.w.options.settings return {} } -// The error of the writer, its message in allocator. -@(private="file") -outputs_error :: proc(o:^Outputs,allocator:mem.Allocator) -> Error { - return writer_error(&o.w,allocator) -} - // The file (in the temp allocator) and id of the root of the import: the -// output of kind the sidecar of the source lists, of that name when it lists +// asset of kind the package of the source lists, of that name when it lists // several of the kind, else a new one. output_root :: proc(o:^Outputs,kind:asset.Fourcc,name,extension:string) -> (id:base.Asset_Id,path:string,error:Error) { caller:=context.temp_allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) listed:=0 - for output in o.w.sidecar.outputs {listed+=int(output.kind==kind)} + for a in o.w.previous.assets {listed+=int(a.kind==kind)} parts:=[1]Part{{kind=kind,name=name,extension=extension,only=listed<=1}} parts_plan(&o.w,parts[:]) return parts[0].id,strings.clone(parts[0].path,caller),{} } -// A file the import reads besides its source: the sidecar lists it, and its -// bytes are part of what the outputs of the cache are built from -// (import_key), so sync imports the source again when it changes. Every -// input is given before the first output of the cache is planned, when that -// key is fixed: a new one after it is an error. +// A file the import reads besides its source: the package lists it, and its +// bytes are part of what the import is made from (identity_of). Every input +// is given before the first output of the cache is planned, when that is +// fixed: a new one after it is an error. output_input :: proc(o:^Outputs,file:string) -> Error { caller:=context.temp_allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) @@ -78,23 +86,13 @@ output_input :: proc(o:^Outputs,file:string) -> Error { return input_late(file,caller) } -// The file and id of another output of kind: the one the sidecar of the -// source lists with that name (the same index first), else a new one, a -// document in folder ("": that of the import). Asked for again with the same -// name and index, it is the same. +// The file to write and the id of another output of kind: the asset the +// package of the source lists with that name (the same index first), else a +// new one, a document of folder ("": that of the import) in the folder of the +// documents. Asked for again with the same name and index, it is the same. output_part :: proc(o:^Outputs,kind:asset.Fourcc,name:string,index:int,extension:string,folder:="") -> (id:base.Asset_Id,path:string) { - caller:=context.temp_allocator - context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) - parts:=[1]Part{{kind=kind,name=name,index=index,extension=extension}} - if folder!="" { - kept:=o.w.folder - o.w.folder=folder - defer o.w.folder=kept - parts_plan(&o.w,parts[:]) - } else { - parts_plan(&o.w,parts[:]) - } - return parts[0].id,strings.clone(parts[0].path,caller) + ids,paths:=output_parts(o,kind,{name},index,extension,folder) + return ids[0],paths[0] } // The files and ids of other outputs of kind, by name, as output_part gives @@ -105,7 +103,7 @@ output_parts :: proc(o:^Outputs,kind:asset.Fourcc,names:[]string,index:int,exten parts:=make([]Part,len(names),context.temp_allocator) for name,i in names {parts[i]={kind=kind,name=name,index=index,extension=extension}} kept:=o.w.folder - if folder!="" {o.w.folder=folder} + if folder!="" && o.w.folder!="" {o.w.folder=folder} parts_plan(&o.w,parts) o.w.folder=kept ids=make([]base.Asset_Id,len(parts),caller) @@ -120,42 +118,55 @@ output_info :: proc(o:^Outputs,name:string,index:int,settings:u64=0) -> asset.Im return import_info(o.source,name,index,o.size,o.hash,settings,o.version) } -// Records an output written at path: the report lists it, and a failure of -// the import removes it when it is new. -output_written :: proc(o:^Outputs,id:base.Asset_Id,kind:asset.Fourcc,name,path:string,existed:bool,detail:="") { +// Takes the output id, written to the file its part gave: a blob of the +// build, a document also written or followed in the folder of the documents, +// and the report lists it as name. +output_written :: proc(o:^Outputs,id:base.Asset_Id,name:string,detail:="") -> Error { + caller:=context.temp_allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) - record(&o.w,id,kind,name,path,o.source,existed,detail) + for output,at in o.w.outputs { + if output.id!=id {continue} + if !writer_put(&o.w,at,o.w.paths[at]) { + error:=writer_error(&o.w,caller) + o.w.error={} + return error + } + record(&o.w,at,name,detail) + return {} + } + return error_make(.Malformed,"%s: no part of the import is the asset %v: write the outputs output_root and output_part give",o.source,id,allocator=caller) } output_warn :: proc(o:^Outputs,format:string,args:..any) { append(&o.report.warnings,fmt.aprintf(format,..args,allocator=o.report.allocator)) } -// Whether every output of the cache the sidecar lists has its file under the -// key of what the import builds from, or one under another key it may take: -// then the writer keeps each or builds it, and a model read ahead for it needs -// nothing cooked (model_cook). It fixes that key: -// every input is given (output_input) before. +// Whether the store has every output of the cache of the package, built from +// what the import is made from now: then the writer takes each, and a model +// read ahead for it needs nothing cooked (model_cook). It fixes what the +// import is made from: every input is given (output_input) before. outputs_current :: proc(o:^Outputs) -> bool { if current,known:=o.current.?;known {return current} + context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) o.current=writer_current(&o.w) return o.current.? } -// Ends the outputs: writes the sidecar of the source and gives the report, -// with a warning for each output the sidecar listed that none took. failed -// removes the files the outputs created, leaves the sidecar as it was and -// gives no report. -outputs_end :: proc(o:^Outputs,failed:bool) -> (report:Import_Report) { +// Ends the outputs: puts the manifest of the package and its build in the +// store, names them, and gives the report, with a warning for each asset the +// package listed that none took. failed removes the documents the outputs +// created, records nothing and gives no report. +outputs_end :: proc(o:^Outputs,failed:bool,allocator:=context.allocator) -> (report:Import_Report,error:Error) { + caller:=allocator context.temp_allocator=base.dynamic_arena_allocator(&o.scratch) - if failed { + if failed || !writer_finish(&o.w,filepath.stem(o.source),o.importer if o.importer!="" else "whole") { + if !failed {error=writer_error(&o.w,caller)} writer_rollback(&o.w) report_destroy(&o.report) } else { - if !writer_finish(&o.w) {writer_warn(&o.w,"%s",o.w.error.message)} report=o.report } - asset.types_destroy(&o.w.types) + writer_close(&o.w) delete(o.w.error.message,o.w.allocator) mem.dynamic_arena_destroy(&o.scratch) o^={} diff --git a/cook/import/outputs_test.odin b/cook/import/outputs_test.odin index dda99e784..68cccffa5 100644 --- a/cook/import/outputs_test.odin +++ b/cook/import/outputs_test.odin @@ -3,75 +3,103 @@ package importer import "core:fmt" import "core:mem" import "core:os" +import "core:slice" import "core:strings" import "core:testing" import "mirai:asset" import "mirai:cook/textures" import "mirai:std/base" import mm "mirai:std/math" +import "mirai:store" + +// The package an import wrote, from the store. +@(private="file") +test_package :: proc(t:^testing.T,s:^store.Store,report:Import_Report) -> (p:asset.Package,b:asset.Build,ok:bool) { + error:asset.Error + if p,error=asset.package_get(s,report.manifest,context.temp_allocator);!testing.expect_value(t,error,asset.Error.None) {return} + if b,error=asset.build_get(s,report.build,context.temp_allocator);!testing.expect_value(t,error,asset.Error.None) {return} + return p,b,true +} // What a plugin that imports a source whole writes with Outputs: the root and -// the parts are the assets the sidecar of the source lists by kind and name, -// each in the folder its kind goes to, a part asked for twice is one, a part -// renamed in the source is another, and a run that fails leaves the sidecar as -// it was. +// the parts are the assets the package of the source lists by kind and name, +// a document also in the folder its kind goes to, a part asked for twice is +// one, a part renamed in the source is another, and a run that fails records +// nothing. @(test) -the_outputs_of_a_source_are_the_assets_its_sidecar_lists :: proc(t:^testing.T) { - root,assets:=test_assets(t,"outputs") +the_outputs_of_a_source_are_the_assets_its_package_lists :: proc(t:^testing.T) { + root,assets,s:=test_assets(t,"outputs") if root=="" {return} source:=fmt.tprintf("%s/level.map",assets) bytes:="a level" testing.expect(t,os.write_entire_file(source,bytes)==nil) - Pass :: struct {scene,floor:base.Asset_Id,materials:[2]base.Asset_Id,scene_path,floor_path:string,warnings:int} - pass :: proc(t:^testing.T,root,source,bytes:string,names:[]string,failed:=false) -> (p:Pass,ok:bool) { + Pass :: struct {scene,floor:base.Asset_Id,materials:[2]base.Asset_Id,floor_blob:store.Hash,report:Import_Report,warnings:int} + pass :: proc(t:^testing.T,s:^store.Store,root,source,bytes:string,names:[]string,failed:=false) -> (p:Pass,ok:bool) { o:Outputs - error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{root=root}) + error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{store=s,root=root,documents=fmt.tprintf("%s/assets",root)},allocator=context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} root_error:Error - p.scene,p.scene_path,root_error=output_root(&o,asset.KIND_SCENE,"level",".scene") + p.scene,_,root_error=output_root(&o,asset.KIND_SCENE,"level",".scene") testing.expect_value(t,root_error.code,Error_Code.None) - p.floor,p.floor_path=output_part(&o,asset.KIND_MESH,"Floor",0,".mesh") + floor_path:string + p.floor,floor_path=output_part(&o,asset.KIND_TEXTURE,"Floor",0,".tex") ids,paths:=output_parts(&o,asset.KIND_MATERIAL,names,0,".mat",fmt.tprintf("%s/assets/level/materials",root)) for id,at in ids { if at0 && string(after)==string(kept),"a failed import leaves the sidecar as it was") - - // A part renamed in the source is a new one, and the one that is gone leaves the sidecar. - renamed,renamed_ok:=pass(t,root,source,bytes,{"Wood","Granite"}) + manifest,build,read:=test_package(t,s,first.report) + if !read {return} + testing.expect(t,len(manifest.assets)==4 && len(build.outputs)==3,"the scene planned is listed; what was written is built") + for o in build.outputs { + listed:=false + for a in manifest.assets {listed||=a.id==o.id && a.kind==o.kind} + testing.expect(t,listed && store.has(s,o.blob),"each output of the build is an asset of its package, in the store") + } + testing.expect(t,os.exists(fmt.tprintf("%s/level/materials/Stone.mat",assets)),"a document is written in the folder of the documents") + + // Again: the same assets. A run that fails records nothing. + again,again_ok:=pass(t,s,root,source,bytes,{"Wood","Stone"}) + testing.expect(t,again_ok && again.scene==first.scene && again.floor==first.floor && again.floor_blob==first.floor_blob && again.materials=={first.materials[1],first.materials[0]} && again.warnings==0,"each output is found by its kind and name") + id_text:[base.ASSET_ID_TEXT]u8 + named,_:=store.ref_get(s,asset.REF_PACKAGES,base.asset_id_format(manifest.id,id_text[:])) + _,_=pass(t,s,root,source,bytes,{"Clay"},failed=true) + after,_:=store.ref_get(s,asset.REF_PACKAGES,base.asset_id_format(manifest.id,id_text[:])) + testing.expect(t,named==again.report.manifest && after==named,"a failed import leaves the package as it was") + + // A part renamed in the source is a new one, and the one that is gone leaves the package. + renamed,renamed_ok:=pass(t,s,root,source,bytes,{"Wood","Granite"}) testing.expect(t,renamed_ok && renamed.materials[0]==first.materials[1] && renamed.materials[1]!=first.materials[0],"Granite is not what Stone was") - _,gone_ok:=pass(t,root,source,bytes,{"Granite"}) - sidecar,found,_=sidecar_load(source,context.temp_allocator) - testing.expect(t,gone_ok && found && len(sidecar.outputs)==3,"Wood is no longer listed") + gone,gone_ok:=pass(t,s,root,source,bytes,{"Granite"}) + left,_,_:=test_package(t,s,gone.report) + testing.expect(t,gone_ok && len(left.assets)==3,"Wood is no longer listed") } // A quad of one material, which a model of the test draws. @@ -113,20 +141,20 @@ model_of :: proc(name,mesh,image:string,materials:..string) -> Imported_Model { // import: one part of the kind and name another model gave is that asset, // written once, and every part keeps its asset when the source is imported // again. A name with '/' goes to that folder, made safe; the files the import -// reads are what its cache is built from, given before the cache is. +// reads are what it is made from, given before the cache is planned. @(test) the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { - root,assets:=test_assets(t,"outputs-models") + root,assets,s:=test_assets(t,"outputs-models") if root=="" {return} source:=fmt.tprintf("%s/level.map",assets) bytes:="a level of two props" testing.expect(t,os.write_entire_file(source,bytes)==nil) package_file:=fmt.tprintf("%s/props.pkg",assets) testing.expect(t,os.write_entire_file(package_file,"the props")==nil) - Pass :: struct {level:base.Asset_Id,crate,barrel:Import_Report,textures,materials,meshes:int,wood,stone:string} - pass :: proc(t:^testing.T,root,source,bytes,package_file:string,root_first:bool) -> (p:Pass,ok:bool) { + Pass :: struct {level:base.Asset_Id,crate,barrel,report:Import_Report,textures,materials,meshes:int,wood,stone:string} + pass :: proc(t:^testing.T,s:^store.Store,root,source,bytes,package_file:string,root_first:bool) -> (p:Pass,ok:bool) { o:Outputs - error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{root=root,textures=.Uncompressed}) + error:=outputs_begin(&o,source,"level",u64(len(bytes)),base.hash64(transmute([]u8)bytes),import_version(0),{store=s,root=root,documents=fmt.tprintf("%s/assets",root),textures=.Uncompressed},allocator=context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} testing.expect_value(t,output_input(&o,package_file).code,Error_Code.None) if root_first {p.level,_,_=output_root(&o,asset.KIND_SCENE,"level",".scene")} @@ -139,8 +167,8 @@ the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { late:=output_input(&o,fmt.tprintf("%s.more",package_file)) testing.expect(t,late.code!=.None && output_input(&o,package_file).code==.None,"a new input once the cache is planned is refused, a known one is not") if !root_first {p.level,_,_=output_root(&o,asset.KIND_SCENE,"level",".scene")} - report:=outputs_end(&o,false) - defer report_destroy(&report) + report,end_error:=outputs_end(&o,false,context.temp_allocator) + testing.expectf(t,end_error.code==.None,"%s",end_error.message) for written in report.assets { switch written.kind { case asset.KIND_TEXTURE: p.textures+=1 @@ -150,42 +178,46 @@ the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { if written.name=="Materials/M_Wood" {p.wood=strings.clone(written.path,context.temp_allocator)} if written.name=="../Materials/./M_Stone" {p.stone=strings.clone(written.path,context.temp_allocator)} } + p.report=report return p,true } - first,ok:=pass(t,root,source,bytes,package_file,false) + first,ok:=pass(t,s,root,source,bytes,package_file,false) if !ok {return} crate,barrel:=first.crate,first.barrel testing.expect(t,crate.material_ids[0]==barrel.material_ids[0] && crate.image_ids[0]==barrel.image_ids[0],"the material and the texture of both models are one asset") testing.expect(t,crate.mesh_ids[0]!=barrel.mesh_ids[0] && crate.prefab!=barrel.prefab && crate.prefab!={} && barrel.material_ids[1]!={}) testing.expectf(t,first.textures==1 && first.materials==2 && first.meshes==2,"what is shared is written once: %v",first) - testing.expectf(t,strings.has_suffix(crate.prefab_path,"/assets/level/Crate.scene"),"%s",crate.prefab_path) testing.expectf(t,strings.has_suffix(first.wood,"/assets/level/Materials/M_Wood.mat"),"%s",first.wood) testing.expectf(t,strings.has_suffix(first.stone,"/assets/level/Materials/M_Stone.mat"),"%s goes nowhere else",first.stone) - // The registry lists the cache where the names say. - r:asset.Registry - asset.registry_init(&r,assets,allocator=context.temp_allocator,imported=fmt.tprintf("%s/.mirai/imported",root)) - testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) - listed:=make(map[string]base.Asset_Id,context.temp_allocator) - for file in r.files {listed[file.path]=file.id} - testing.expectf(t,listed["level/Meshes/SM_Crate.mesh"]==crate.mesh_ids[0] && listed["level/Textures/T_Wood.tex"]==crate.image_ids[0],"%v",listed) + // The package lists the parts by the names the models give them, the prefab of each among them. + manifest,build,read:=test_package(t,s,first.report) + if !read {return} + named:=make(map[string]base.Asset_Id,context.temp_allocator) + for a in manifest.assets {named[a.name]=a.id} + testing.expectf(t,named["Meshes/SM_Crate"]==crate.mesh_ids[0] && named["Textures/T_Wood"]==crate.image_ids[0] && named["Crate"]==crate.prefab,"%v",named) + built:=0 + for o in build.outputs {built+=int(o.id==crate.mesh_ids[0] || o.id==crate.image_ids[0])} + testing.expect(t,built==2,"the build has the mesh and the texture") - // Again, with the root asked for first among the scenes its sidecar lists: the same assets. - again,again_ok:=pass(t,root,source,bytes,package_file,true) + // Again, with the root asked for first among the scenes its package lists: the same assets. + again,again_ok:=pass(t,s,root,source,bytes,package_file,true) testing.expect(t,again_ok && again.level==first.level && again.crate.mesh_ids[0]==crate.mesh_ids[0] && again.barrel.material_ids[1]==barrel.material_ids[1] && again.crate.image_ids[0]==crate.image_ids[0] && again.crate.prefab==crate.prefab,"each part keeps its asset") - // An input that changed changes what the cache is built from. - sidecar,_,_:=sidecar_load(source,context.temp_allocator) - before:=import_key(source,sidecar,import_version(0),0,false) + // An input that changed changes what the import is made from. + testing.expect(t,len(manifest.inputs)==1 && manifest.inputs[0]=="props.pkg","the input is listed") + settings,_:=settings_parse(manifest.settings) + before:=identity_of(path_key(source),manifest.inputs[:],settings,0) + testing.expect(t,before==manifest.identity,"the identity is that of the source, its input and its settings") testing.expect(t,os.write_entire_file(package_file,"other props")==nil) - testing.expect(t,len(sidecar.inputs)==1 && sidecar.inputs[0]=="props.pkg" && import_key(source,sidecar,import_version(0),0,false)!=before,"the input is listed and counts in the key") + testing.expect(t,identity_of(path_key(source),manifest.inputs[:],settings,0)!=before,"the input counts in the identity") } // An image only its format decodes is keyed by its data: a texture the // texture cache keeps for that data is written without making its texels. @(test) an_image_only_its_format_decodes_is_taken_from_the_cache_without_decoding :: proc(t:^testing.T) { - root,assets:=test_assets(t,"outputs-made") + root,assets,s:=test_assets(t,"outputs-made") if root=="" {return} cache:=fmt.tprintf("%s/cache",root) texels_of :: proc(data:[]u8,allocator:mem.Allocator) -> (made:textures.Source,ok:bool) { @@ -194,18 +226,17 @@ an_image_only_its_format_decodes_is_taken_from_the_cache_without_decoding :: pro return {texels=texels,width=4,height=4,format=.RGBA8},true } refused :: proc(data:[]u8,allocator:mem.Allocator) -> (made:textures.Source,ok:bool) {return} - written :: proc(t:^testing.T,root,assets,name:string,make:Image_Make,cache:string) -> (texels:string) { + written :: proc(t:^testing.T,s:^store.Store,root,assets,name:string,make:Image_Make,cache:string) -> (texels:string,report:Import_Report) { source:=fmt.tprintf("%s/%s.map",assets,name) testing.expect(t,os.write_entire_file(source,name)==nil) o:Outputs - error:=outputs_begin(&o,source,name,u64(len(name)),base.hash64(transmute([]u8)name),import_version(0),{root=root,textures=.Uncompressed,texture_cache=cache}) + error:=outputs_begin(&o,source,name,u64(len(name)),base.hash64(transmute([]u8)name),import_version(0),{store=s,root=root,textures=.Uncompressed,texture_cache=cache},allocator=context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model:=model_of("Crate","SM_Crate","T_Wood","M_Wood") model.images[0]={name="T_Wood",data=transmute([]u8)string("packed"),make=make,srgb=true} _,error=output_model(&o,&model,context.temp_allocator) testing.expectf(t,error.code==.None,"%s",error.message) - report:=outputs_end(&o,false) - defer report_destroy(&report) + report,_=outputs_end(&o,false,context.temp_allocator) for w in report.assets { if w.kind!=asset.KIND_TEXTURE {continue} types:asset.Types;asset.types_init(&types,context.temp_allocator) @@ -216,21 +247,160 @@ an_image_only_its_format_decodes_is_taken_from_the_cache_without_decoding :: pro } return } - first:=written(t,root,assets,"first",texels_of,cache) + first,report:=written(t,s,root,assets,"first",texels_of,cache) testing.expect(t,len(first)==4*4*4,"its format makes the texels") - testing.expect(t,written(t,root,assets,"second",refused,cache)==first,"the cache gives the texture of the same data") - testing.expect(t,written(t,root,assets,"third",refused,"")=="","without the cache the texels are made, and here they cannot be") + second,_:=written(t,s,root,assets,"second",refused,cache) + testing.expect(t,second==first,"the cache gives the texture of the same data") + third,_:=written(t,s,root,assets,"third",refused,"") + testing.expect(t,third=="","without the cache the texels are made, and here they cannot be") // Imported again unchanged, its outputs are current: a model read ahead for it cooks nothing. - current :: proc(root,assets:string) -> bool { + current :: proc(s:^store.Store,root,assets:string) -> bool { o:Outputs source:=fmt.tprintf("%s/first.map",assets) - if outputs_begin(&o,source,"first",5,base.hash64(transmute([]u8)string("first")),import_version(0),{root=root,textures=.Uncompressed}).code!=.None {return false} + if outputs_begin(&o,source,"first",5,base.hash64(transmute([]u8)string("first")),import_version(0),{store=s,root=root,textures=.Uncompressed}).code!=.None {return false} defer outputs_end(&o,true) return outputs_current(&o) } - testing.expect(t,current(root,assets),"nothing changed") - entries,_:=os.read_all_directory_by_path(fmt.tprintf("%s/.mirai/imported",root),context.temp_allocator) - for entry in entries {_=os.remove(entry.fullpath)} - testing.expect(t,!current(root,assets),"its files are gone") + testing.expect(t,current(s,root,assets),"nothing changed") + _,build,read:=test_package(t,s,report) + if !read {return} + for o in build.outputs { + if o.kind==asset.KIND_MESH {store.remove(s,o.blob)} + } + testing.expect(t,!current(s,root,assets),"a blob is gone") +} + +// A source a model was read from, in a folder of its own that no project +// holds, its bytes those of name. +@(private="file") +model_source :: proc(t:^testing.T,folder,name:string,version:u32) -> (model:Imported_Model) { + model=model_of("Crate","SM_Crate","T_Wood","M_Wood") + model.source=path_key(fmt.tprintf("%s/%s.model",folder,name)) + testing.expect(t,os.write_entire_file(model.source,name)==nil) + model.source_size,model.source_hash=u64(len(name)),base.hash64(transmute([]u8)name) + model.format,model.folder,model.version="test",name,version + return +} + +// The blobs of a build by asset, sorted by id. +@(private="file") +build_blobs :: proc(t:^testing.T,s:^store.Store,report:Import_Report) -> []asset.Build_Output { + _,b,_:=test_package(t,s,report) + slice.sort_by(b.outputs[:],proc(a,b:asset.Build_Output) -> bool {return a.id[0] (report:Import_Report,ok:bool) { + model:=model_source(t,outside,"crate",import_version(0)) + error:Error + report,error=import_write(&model,{store=s,textures=.Uncompressed},context.temp_allocator) + return report,testing.expectf(t,error.code==.None,"%s",error.message) + } + first,ok:=import_once(t,s,outside) + if !ok {return} + entries,_:=os.read_all_directory_by_path(outside,context.temp_allocator) + testing.expect_value(t,len(entries),1) + again,again_ok:=import_once(t,s,outside) + if !again_ok {return} + testing.expect(t,again.package_id==first.package_id && again.manifest==first.manifest && again.build==first.build,"the same package, manifest and build") + testing.expect(t,again.mesh_ids[0]==first.mesh_ids[0] && again.material_ids[0]==first.material_ids[0] && again.image_ids[0]==first.image_ids[0] && again.prefab==first.prefab,"the same ids") + kept:=0 + for w in again.assets {kept+=int(w.kept)} + testing.expect(t,kept==2,"the mesh and the texture are taken from the build, not built") + p,_,_:=test_package(t,s,first) + testing.expect(t,p.origin.path==path_key(fmt.tprintf("%s/crate.model",outside)) && p.origin.importer=="test","its origin is where the source was") + for &a in p.assets { + if a.kind==asset.KIND_MESH {testing.expect(t,a.owner==first.prefab && a.part && slice.contains(a.dependencies[:],first.material_ids[0]),"a mesh is a part of the prefab and depends on its material")} + } +} + +// The identity of a package is what its source is: the settings change it, a +// version of the engine does not, and changes the key of its build instead. +@(test) +a_changed_setting_changes_the_identity_and_a_changed_engine_does_not :: proc(t:^testing.T) { + folder:=base.test_folder(t,"outputs-identity") + if folder=="" {return} + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} + written :: proc(t:^testing.T,s:^store.Store,folder:string,version:u32,options:Import_Options) -> (p:asset.Package,b:asset.Build) { + model:=model_source(t,folder,"crate",version) + options:=options + options.store,options.textures,options.ignore_stored=s,.Uncompressed,true + report,error:=import_write(&model,options,context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + p,b,_=test_package(t,s,report) + return + } + first,first_build:=written(t,s,folder,import_version(0),{}) + upgraded,upgraded_build:=written(t,s,folder,import_version(9),{}) + testing.expect(t,first.identity!={} && upgraded.identity==first.identity && upgraded_build.key!=first_build.key,"another version of the reader keeps the identity and changes the key") + lods,_:=written(t,s,folder,import_version(0),{lods=.None}) + testing.expect(t,lods.identity!=first.identity && lods.id==first.id,"a setting changes the identity of the same package") + testing.expect(t,strings.contains(lods.settings,"lods"),"the package keeps the settings that differ from the defaults") +} + +// A build of Fast textures stays on this machine: never under the namespace +// peers share, while the machine chooses it for the package. +@(test) +a_fast_build_is_not_published_where_peers_find_builds :: proc(t:^testing.T) { + folder:=base.test_folder(t,"outputs-fast") + if folder=="" {return} + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} + model:=model_source(t,folder,"crate",import_version(0)) + report,error:=import_write(&model,{store=s,textures=.Uncompressed,fast_textures=true},context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + testing.expect(t,!report.final,"its texture is Fast") + digits:[64]u8 + _,shared:=store.ref_get(s,asset.REF_IMPORTS,store.hex(report.key,&digits)) + fast,local:=store.ref_get(s,asset.REF_IMPORTS_FAST,store.hex(report.key,&digits)) + testing.expect(t,shared==.Missing && local==.None && fast==report.build,"published under the local namespace alone") + p,_,_:=test_package(t,s,report) + chosen,_:=store.ref_get(s,asset.REF_BUILDS,store.hex(asset.build_select(report.manifest,&p),&digits)) + testing.expect(t,chosen==report.build,"the build this machine chose") + list:=asset.Packages{packages=make([dynamic]asset.Project_Package,context.temp_allocator)} + append(&list.packages,asset.Project_Package{id=p.id,manifest=report.manifest}) + resolved,resolve_error:=asset.packages_resolve(s,&list,context.temp_allocator) + testing.expect(t,resolve_error==.None && len(resolved.problems)==0 && len(resolved.listed)==2,"a project lists the mesh and the texture of the package") + for l in resolved.listed {testing.expectf(t,l.state==.Built && os.exists(l.file),"%s is built",l.path)} +} + +// Two machines that import one source as the same package write the same +// blobs: what an import writes depends on no machine. +@(test) +two_stores_that_import_one_source_hold_the_same_blobs :: proc(t:^testing.T) { + folder:=base.test_folder(t,"outputs-stores") + if folder=="" {return} + here:=test_store(t,fmt.tprintf("%s/here",folder)) + there:=test_store(t,fmt.tprintf("%s/there",folder)) + if here==nil || there==nil {return} + model:=model_source(t,folder,"crate",import_version(0)) + first,error:=import_write(&model,{store=here,textures=.Uncompressed},context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + // The other machine has the manifest, as a peer gives it, and the source. + p,_,_:=test_package(t,here,first) + manifest,put_error:=asset.package_put(there,&p) + testing.expect(t,put_error==.None && manifest==first.manifest,"the manifest is the same blob") + model=model_source(t,folder,"crate",import_version(0)) + second:Import_Report + second,error=import_write(&model,{store=there,manifest=manifest,textures=.Uncompressed},context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + testing.expect(t,second.manifest==first.manifest && second.build==first.build,"the same manifest and build") + a,b:=build_blobs(t,here,first),build_blobs(t,there,second) + testing.expect(t,len(a)==len(b) && len(a)>0) + for o,at in a { + if at string { + settings,defaults:=settings,Import_Settings{} + schema:serial.Schema + serial.schema_init(&schema,context.temp_allocator) + defer serial.schema_destroy(&schema) + type,_:=serial.schema_add(&schema,Import_Settings) + w:serial.Writer + serial.writer_init(&w,512,allocator) + serial.text_write_changed(&w,&schema,type,([^]u8)(&settings)[:size_of(Import_Settings)],([^]u8)(&defaults)[:size_of(Import_Settings)],.Pretty,depth=1) + return string(serial.writer_take(&w)) +} + +// False for a text that is not settings this build reads: a field or a value +// it does not know, or one out of its range. "" is the defaults. +settings_parse :: proc(text:string) -> (settings:Import_Settings,ok:bool) { + if text!="" { + schema:serial.Schema + serial.schema_init(&schema,context.temp_allocator) + defer serial.schema_destroy(&schema) + type,_:=serial.schema_add(&schema,Import_Settings) + r:serial.Text_Reader + serial.text_reader_init(&r,text,context.temp_allocator) + defer serial.text_reader_destroy(&r) + serial.text_read(&r,&schema,type,([^]u8)(&settings)[:size_of(Import_Settings)]) or_return + } + return settings,settings_valid(settings) +} + +@(private="file") +settings_valid :: proc(settings:Import_Settings) -> bool { + return settings.textures<=max(textures.Encoding) && settings.rdo_lambda<=textures.RDO_LAMBDA_MAX && + settings.image<=max(Image_Use) && settings.alpha_coverage>=0 && settings.alpha_coverage<=1 && settings.lods<=max(Lods) && settings.parts<=max(Parts) && settings.front<=max(Front) && settings.rig<=max(Rig) && + settings.rig_crotch>=0 && settings.rig_neck>=0 && (settings.rig==.Humanoid || settings.rig_crotch==0 && settings.rig_neck==0) +} + +// What the outputs of an import are made from, without any version of the +// engine (asset.Package.identity): the bytes of the source at path and of each +// input, named from the folder of the source, the text of its settings and the +// code of the plugin that imports it whole (Import_Options.importer_code). A +// file that cannot be read, such as a folder, counts as no bytes. +identity_of :: proc(path:string,inputs:[]string,settings:Import_Settings,code:u64) -> store.Hash { + TAG :: "mirai import identity" + bytes:=make([dynamic]u8,context.temp_allocator) + append(&bytes,TAG) + address :: proc(bytes:^[dynamic]u8,file:string) { + hash:=store.address_of_bytes(nil) + if !unsafe_path(file) { + if read,_,error:=store.address_of_file(file);error==.None {hash=read} + } + append(bytes,..hash[:]) + } + address(&bytes,path) + folder:=path[:max(strings.last_index_byte(path,'/'),0)] + for input in inputs { + append(&bytes,input) + append(&bytes,0) + address(&bytes,fmt.tprintf("%s/%s",folder,input)) + } + append(&bytes,settings_text(settings,context.temp_allocator)) + append(&bytes,0) + code:=code + append(&bytes,..([^]u8)(&code)[:size_of(code)]) + return store.address_of_bytes(bytes[:]) +} + +// The versions of what reads a source with a format of version +// (import_version) and writes its assets, as asset.import_key combines them: +// the writer and the format, every kind of asset and what builds textures. +import_versions :: proc(version:u32) -> [3]u64 { + kinds:=make([dynamic]u64,0,len(asset.KINDS),context.temp_allocator) + for info in asset.KINDS {append(&kinds,u64(info.kind)<<16|u64(info.version))} + return {u64(version),base.hash64(slice.to_bytes(kinds[:])),u64(textures.BUILD_VERSION)} +} + +// The manifest of the package the source at origin (an absolute path with +// forward slashes) was ingested as: the one manifest names, else the one the +// ref REF_PACKAGES of this machine names whose origin is origin. found is +// false without one. In allocator, an arena. +package_previous :: proc(s:^store.Store,manifest:store.Hash,origin:string,allocator:=context.allocator) -> (p:asset.Package,hash:store.Hash,found:bool,error:Error) { + digits:[64]u8 + if manifest!={} { + read_error:asset.Error + p,read_error=asset.package_get(s,manifest,allocator) + if read_error==.Not_Found {return {},{},false,error_make(.Not_Found,"the manifest %s of the package of %s is not in the store at %s: fetch it from a peer that has it, or import the source as a new package",store.hex(manifest,&digits),origin,s.root)} + if read_error!=.None {return {},{},false,error_make(.Malformed,"the manifest %s of the package of %s does not read (%v): import the source as a new package",store.hex(manifest,&digits),origin,read_error)} + return p,manifest,true,{} + } + refs,_:=store.ref_list(s,asset.REF_PACKAGES,context.temp_allocator) + for ref in refs { + candidate,read_error:=asset.package_get(s,ref.hash,context.temp_allocator) + if read_error!=.None || candidate.origin.path!=origin {continue} + p,read_error=asset.package_get(s,ref.hash,allocator) + if read_error!=.None {continue} + return p,ref.hash,true,{} + } + return +} + +// The settings of the package the source at path was ingested as +// (package_previous with options.manifest), the defaults (found false) when it +// was not or without options.store. +settings_load :: proc(options:Import_Options,path:string) -> (settings:Import_Settings,found:bool,error:Error) { + if options.store==nil {return} + p:asset.Package + p,_,found,error=package_previous(options.store,options.manifest,path_key(path),context.temp_allocator) + if !found || error.code!=.None {return} + valid:bool + if settings,valid=settings_parse(p.settings);!valid { + return {},false,error_make(.Malformed,"the settings the package of %s keeps (%s) are not those of this build: import it as a new package",path,p.settings) + } + return +} + +// The asset the package the source at path was ingested as on this machine +// gave what the source names name, of kind; false when it gave none. +output_id :: proc(s:^store.Store,path:string,kind:asset.Fourcc,name:string) -> (id:base.Asset_Id,found:bool) { + p,_,known,error:=package_previous(s,{},path_key(path),context.temp_allocator) + if !known || error.code!=.None {return} + for a in p.assets { + if a.kind==kind && a.name==name {return a.id,true} + } + return +} + +// Puts the manifest of the package and its build in the store and names +// them: the package with REF_PACKAGES, the build with REF_IMPORTS when its +// textures are Final, else REF_IMPORTS_FAST, and as the build this machine +// chose for the manifest (REF_BUILDS). +@(private) +writer_publish :: proc(w:^Writer,name,importer:string) -> bool { + s:=w.options.store + writer_identity(w) + p:=asset.Package{id=w.package_id,name=name,identity=w.identity,inputs=w.inputs,assets=w.outputs} + p.origin={path=w.origin,folder=os.is_dir(w.path),importer=importer} + if settings:=settings_text(w.report.settings,context.temp_allocator);settings!="{}" {p.settings=settings} + slice.sort(p.inputs[:]) + b:=asset.Build{package_id=w.package_id,key=w.key,final=!w.fast,encoders=textures.encoders_id()} + b.outputs=make([dynamic]asset.Build_Output,context.temp_allocator) + for output,at in w.outputs { + if w.blobs[at]=={} {continue} + info,_:=asset.kind_info(output.kind) + append(&b.outputs,asset.Build_Output{id=output.id,kind=output.kind,kind_version=info.version,blob=w.blobs[at]}) + } + manifest,manifest_error:=asset.package_put(s,&p) + if manifest_error!=.None {return writer_fail(w,.Write,"the manifest of the package of %s could not be put in the store at %s: %v",w.path,s.root,manifest_error)} + build,build_error:=asset.build_put(s,&b) + if build_error!=.None {return writer_fail(w,.Write,"the build of the package of %s could not be put in the store at %s: %v",w.path,s.root,build_error)} + id_text:[base.ASSET_ID_TEXT]u8 + key_text,select_text:[64]u8 + refs:=[3]struct{namespace,name:string}{ + {asset.REF_PACKAGES,base.asset_id_format(w.package_id,id_text[:])}, + {asset.REF_IMPORTS if b.final else asset.REF_IMPORTS_FAST,store.hex(w.key,&key_text)}, + {asset.REF_BUILDS,store.hex(asset.build_select(manifest,&p),&select_text)}, + } + for ref,at in refs { + if error:=store.ref_set(s,ref.namespace,ref.name,manifest if at==0 else build);error!=.None { + return writer_fail(w,.Write,"the ref %s/%s of the package of %s could not be set in the store at %s: %v",ref.namespace,ref.name,w.path,s.root,error) + } + } + w.report.package_id,w.report.manifest,w.report.build,w.report.key,w.report.final=w.package_id,manifest,build,w.key,b.final + return true +} + +// Fixes what the import is made from once its inputs are whole: its identity +// and the key of its build (asset.import_key). +@(private) +writer_identity :: proc(w:^Writer) { + if w.fixed {return} + w.fixed=true + slice.sort(w.inputs[:]) + w.identity=identity_of(w.origin,w.inputs[:],w.report.settings,w.options.importer_code) + versions:=import_versions(w.version) + w.key,_=asset.import_key(w.identity,versions[:]) +} diff --git a/cook/import/progress.odin b/cook/import/progress.odin index 359867d14..19b73d391 100644 --- a/cook/import/progress.odin +++ b/cook/import/progress.odin @@ -5,7 +5,7 @@ import "core:io" import "core:time" // What an import says of how far it is while it runs, for a sync to print -// under the line of its source: the assets it wrote, of those its sidecar +// under the line of its source: the assets it wrote, of those its package // listed (what the previous import gave), and how long it has run; a line at // most every PROGRESS_EVERY. The caller owns it; the thread that writes the // import updates it. diff --git a/cook/import/prompts.odin b/cook/import/prompts.odin index d1301f1e9..b65a09544 100644 --- a/cook/import/prompts.odin +++ b/cook/import/prompts.odin @@ -220,10 +220,12 @@ import_prompts :: proc(path:string,options:Import_Options={},allocator:=context. w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) w.options.textures=.Uncompressed - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { for file in files {writer_input(&w,file)} - ok=prompts_write(&w,path,bytes,entries,of_entry,images[:],places,pages) && writer_finish(&w) + name:string + ok,name=prompts_write(&w,path,bytes,entries,of_entry,images[:],places,pages) + ok=ok && writer_finish(&w,name,"prompts") } if ok && skipped>0 {append(&report.warnings,fmt.aprintf("%d images of %s say no input by their names and were left out; Project Settings > UI gives any input an image of its own",skipped,path,allocator=allocator))} if !ok { @@ -236,13 +238,14 @@ import_prompts :: proc(path:string,options:Import_Options={},allocator:=context. } @(private="file") -prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,of_entry:[]int,images:[]textures.Image,places:[]Prompts_Place,pages:[][2]int) -> bool { +prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,of_entry:[]int,images:[]textures.Image,places:[]Prompts_Place,pages:[][2]int) -> (ok:bool,name:string) { PAD :: PROMPTS_PADDING source:=w.source file:=filepath.base(path) - name:=file[:len(file)-len(PROMPTS_MANIFEST)] if strings.has_suffix(strings.to_lower(file,context.temp_allocator),PROMPTS_MANIFEST) else filepath.stem(file) + name=file[:len(file)-len(PROMPTS_MANIFEST)] if strings.has_suffix(strings.to_lower(file,context.temp_allocator),PROMPTS_MANIFEST) else filepath.stem(file) target:string - w.owner,target=root_take(w,asset.KIND_PROMPTS,name,".prompts") + root:int + w.owner,target,root=root_take(w,asset.KIND_PROMPTS,name,".prompts") // The texels of each page: the images, each with its edge repeated around it. jobs:=make([]Texture_Job,len(pages),context.temp_allocator) @@ -267,12 +270,10 @@ prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,o parts[page]=Part{kind=asset.KIND_TEXTURE,name=job.name,index=page,extension=".tex"} } parts_plan(w,parts) - for &part in parts {part.owner=w.owner} - parts_record(w,parts) for &part,page in parts { - if !write_texture(w,jobs[page],part.id,w.owner,part.path,source) { - if w.error.code==.None {writer_fail(w,.Write,"the atlas %s could not be built",part.path)} - return false + if !write_texture(w,jobs[page],part.id,w.owner,part.path,source,part.output) { + if w.error.code==.None {writer_fail(w,.Write,"the atlas %s of %s could not be built",part.name,path)} + return false,name } } @@ -288,8 +289,8 @@ prompts_write :: proc(w:^Writer,path:string,bytes:[]u8,entries:[]Prompts_Entry,o row.aspect=f32(image.width)/f32(image.height) } info:=import_info(source,name,0,u64(len(bytes)),base.hash64(bytes),0,import_version(0)) - existed:=os.exists(target) - if error:=asset.prompts_save(target,&w.types,{id=w.owner,images=rows,source=&info},context.temp_allocator);error!=.None {return save_failed(w,error,target)} - record(w,w.owner,asset.KIND_PROMPTS,name,target,source,existed,fmt.tprintf("%d prompts of %d images in %d atlas%s",len(rows),len(images),len(pages),len(pages)==1?"":"es")) - return true + if error:=asset.prompts_save(target,&w.types,{id=w.owner,images=rows,source=&info},context.temp_allocator);error!=.None {return save_failed(w,error,target),name} + if !writer_put(w,root,target) {return false,name} + record(w,root,name,fmt.tprintf("%d prompts of %d images in %d atlas%s",len(rows),len(images),len(pages),len(pages)==1?"":"es")) + return true,name } diff --git a/cook/import/reader_test.odin b/cook/import/reader_test.odin index 5df0119d8..5cdf81e13 100644 --- a/cook/import/reader_test.odin +++ b/cook/import/reader_test.odin @@ -3,7 +3,9 @@ package importer import "core:fmt" import "core:os" import "core:testing" +import "mirai:asset" import "mirai:std/base" +import "mirai:store" // What the reader of the test below saw in its options. @(private="file") @@ -22,23 +24,29 @@ SETTINGS_FORMAT :: Format{name="Settings test",extensions={0=".seen"},space=ENGI }} // A reader that acts on a setting (Import_Settings.skeleton: clips that keep no skeleton of their -// own) reads it from the options: those of .import when the source has one, as the writer -// applies them, else those the import was given. +// own) reads it from the options: those the package of the source keeps when it was ingested, as +// the writer applies them, else those the import was given. @(test) a_reader_sees_the_settings_kept_with_its_source :: proc(t:^testing.T) { folder:=base.test_folder(t,"reader-settings") if folder=="" {return} + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} path:=fmt.tprintf("%s/model.seen",folder) _=os.write_entire_file(path,"anything") formats:=[1]Format{SETTINGS_FORMAT} given:=base.Asset_Id{1,2} - model,error:=import_model(path,formats[:],{root=folder,settings={skeleton=given}}) + model,error:=import_model(path,formats[:],{store=s,settings={skeleton=given}}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model_destroy(&model) testing.expect_value(t,seen.skeleton,given) kept:=base.Asset_Id{3,4} - if !testing.expect(t,sidecar_save(path,{settings={skeleton=kept}})) {return} - model,error=import_model(path,formats[:],{root=folder,settings={skeleton=given}}) + ingested:=asset.Package{id={5,6},name="model",origin={path=path_key(path)},settings=settings_text({skeleton=kept},context.temp_allocator)} + manifest,put_error:=asset.package_put(s,&ingested) + if !testing.expect_value(t,put_error,asset.Error.None) {return} + id_text:[base.ASSET_ID_TEXT]u8 + testing.expect_value(t,store.ref_set(s,asset.REF_PACKAGES,base.asset_id_format(ingested.id,id_text[:]),manifest),store.Error.None) + model,error=import_model(path,formats[:],{store=s,settings={skeleton=given}}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model_destroy(&model) testing.expect_value(t,seen.skeleton,kept) diff --git a/cook/import/settings.odin b/cook/import/settings.odin index d43706d2b..1429012b8 100644 --- a/cook/import/settings.odin +++ b/cook/import/settings.odin @@ -6,10 +6,11 @@ import "core:strings" import "mirai:cook/textures" import "mirai:std/base" -// How a source becomes assets. The settings of a source are kept in its -// sidecar (sidecar.odin), so importing it again gives the same assets whoever -// does it: an import reads them when the sidecar exists and uses them instead -// of those of its options. The zero value is the default of every setting. +// How a source becomes assets. The settings of a source are kept in the +// manifest of its package (package.odin), so importing it again gives the same +// assets whoever does it: an import of a package ingested before uses them +// instead of those of its options. The zero value is the default of every +// setting. Lods :: enum u8 { Auto, // simplified levels are stored after level 0 (mesh_levels) diff --git a/cook/import/sidecar.odin b/cook/import/sidecar.odin deleted file mode 100644 index aafd9c5fa..000000000 --- a/cook/import/sidecar.odin +++ /dev/null @@ -1,209 +0,0 @@ -package importer - -import "core:fmt" -import "core:os" -import "core:path/filepath" -import "core:slice" -import "core:strconv" -import "core:strings" -import "core:time" -import "mirai:asset" -import "mirai:cook/textures" -import "mirai:std/base" -import "mirai:std/serial" - -// .import, the sidecar (asset/sidecar.odin): what is versioned of an -// import with its source, beside it in the assets of a project or a plugin. -// It says how the source becomes assets (settings), where its outputs go and -// which asset each thing of the source is (outputs): nothing else decides an -// id. The meshes, textures, sounds and fonts (asset.import_cached), which -// nobody edits, are files of the imported folder of the root named by their -// ids, never versioned: the registry lists them by where the sidecar is, and -// sources.sync builds them again. The other kinds are documents of the -// assets, found by their id. - -SIDECAR_EXTENSION :: asset.SIDECAR_EXTENSION -// Of the empty file of the imported folder that says an import of documents -// alone is current, named as a file of the cache is by the id of its first -// output and its key: what sync looks for when no output is of the cache. -STAMP_EXTENSION :: ".stamp" - -Output :: asset.Import_Output - -Sidecar :: struct { - settings:Import_Settings, - folder:string, // under the folder of the source, where its outputs go; "": beside it - inputs:[dynamic]string, // the files the import read besides the source, from the folder of the source: with it, what the outputs are built from - outputs:[dynamic]Output, -} - -@(private="file") -settings_valid :: proc(settings:Import_Settings) -> bool { - return settings.textures<=max(textures.Encoding) && settings.rdo_lambda<=textures.RDO_LAMBDA_MAX && - settings.image<=max(Image_Use) && settings.alpha_coverage>=0 && settings.alpha_coverage<=1 && settings.lods<=max(Lods) && settings.parts<=max(Parts) && settings.front<=max(Front) && settings.rig<=max(Rig) && - settings.rig_crotch>=0 && settings.rig_neck>=0 && (settings.rig==.Humanoid || settings.rig_crotch==0 && settings.rig_neck==0) -} - -// The settings as the text of their object, in the sidecar: only where they -// differ from their defaults (the zero value), so a sidecar shows what was -// chosen; reading fills the rest. -@(private="file") -settings_text :: proc(settings:Import_Settings,allocator:=context.allocator) -> string { - settings,defaults:=settings,Import_Settings{} - schema:serial.Schema - serial.schema_init(&schema,context.temp_allocator) - defer serial.schema_destroy(&schema) - type,_:=serial.schema_add(&schema,Import_Settings) - w:serial.Writer - serial.writer_init(&w,512,allocator) - serial.text_write_changed(&w,&schema,type,([^]u8)(&settings)[:size_of(Import_Settings)],([^]u8)(&defaults)[:size_of(Import_Settings)],.Pretty,depth=1) - return string(serial.writer_take(&w)) -} - -sidecar_text :: proc(sidecar:Sidecar,allocator:=context.allocator) -> string { - return asset.sidecar_text(settings_text(sidecar.settings,context.temp_allocator),{folder=sidecar.folder,inputs=sidecar.inputs,outputs=sidecar.outputs},allocator) -} - -// False for a text that is not the sidecar of a source (asset.sidecar_parse) -// or whose settings this build does not read. What it holds lives in -// allocator, an arena: nothing frees it piece by piece. -sidecar_parse :: proc(text:string,allocator:=context.allocator) -> (sidecar:Sidecar,ok:bool) { - parsed:=asset.sidecar_parse(text,allocator) or_return - sidecar.folder,sidecar.inputs,sidecar.outputs=parsed.folder,parsed.inputs,parsed.outputs - if parsed.settings!="" { - schema:serial.Schema - serial.schema_init(&schema,context.temp_allocator) - defer serial.schema_destroy(&schema) - type,_:=serial.schema_add(&schema,Import_Settings) - r:serial.Text_Reader - serial.text_reader_init(&r,parsed.settings,context.temp_allocator) - defer serial.text_reader_destroy(&r) - serial.text_read(&r,&schema,type,([^]u8)(&sidecar.settings)[:size_of(Import_Settings)]) or_return - } - return sidecar,settings_valid(sidecar.settings) -} - -// What the outputs of an import are built from, as one number: the bytes of -// the source at path and of each input its sidecar lists, its settings, the -// version of what reads and writes it, the code of the plugin that imports it -// whole (Import_Options.importer_code), the versions of the kinds of the -// cache, the encoders of the program and whether its textures are built Fast. -// An output of the cache is current when its file is named by this key. With -// hashes, a file whose size and modification time it knows is not read. -import_key :: proc(path:string,sidecar:Sidecar,version:u32,code:u64,fast:bool,hashes:^Source_Hashes=nil) -> u64 { - words:=make([dynamic]u64,0,7+len(sidecar.inputs),context.temp_allocator) - append(&words,u64(version)<<1|u64(fast),u64(asset.SOUND_VERSION)<<48|u64(asset.MESH_VERSION)<<32|u64(asset.TEXTURE_VERSION)<<16|u64(textures.BUILD_VERSION),u64(asset.FONT_VERSION),code) - append(&words,base.hash64(transmute([]u8)settings_text(sidecar.settings,context.temp_allocator)),textures.encoders_id()) - folder:=filepath.dir(path) - for at in -1.. (hashes:Source_Hashes) { - hashes.files=make(map[string]Source_Hash,allocator=allocator) - data,error:=serial.file_read(path,context.temp_allocator) - if error!=.None {return} - text:=string(data) - for line in strings.split_lines_iterator(&text) { - parts:=strings.split_n(line," ",4,context.temp_allocator) - if len(parts)!=4 {continue} - hash,hash_ok:=strconv.parse_u64(parts[0],16) - size,size_ok:=strconv.parse_i64(parts[1]) - modified,modified_ok:=strconv.parse_i64(parts[2]) - if hash_ok && size_ok && modified_ok {hashes.files[strings.clone(parts[3],allocator)]={size,modified,hash}} - } - return -} - -// Writes the hashes to path when a build changed them. -source_hashes_save :: proc(hashes:^Source_Hashes,path:string) { - if !hashes.changed {return} - b:=strings.builder_make(context.temp_allocator) - for file,known in hashes.files {fmt.sbprintf(&b,"%016x %d %d %s\n",known.hash,known.size,known.modified,file)} - if folder:=filepath.dir(path);!os.is_dir(folder) {_=os.make_directory_all(folder)} - _=serial.file_write_atomic(path,transmute([]u8)strings.to_string(b)) -} - -// The hash of the bytes of a file, from hashes when its size and modification -// time are those it knows; 0 for a file that cannot be read. -@(private="file") -file_hash :: proc(hashes:^Source_Hashes,file:string) -> u64 { - info,stat_error:=os.stat(file,context.temp_allocator) - size,modified:=info.size,time.time_to_unix_nano(info.modification_time) - if hashes!=nil && stat_error==nil { - if known,found:=hashes.files[file];found && known.size==size && known.modified==modified {return known.hash} - } - bytes,_:=serial.file_read(file,context.temp_allocator,limit=MAX_SOURCE_BYTES) - hash:=base.hash64(bytes) - delete(bytes,context.temp_allocator) - if hashes!=nil && stat_error==nil && time.time_to_unix_nano(time.now())-modified>SETTLED_NANOSECONDS { - allocator:=hashes.files.allocator - if _,found:=hashes.files[file];!found {hashes.files[strings.clone(file,allocator)]={}} - hashes.files[file]={size,modified,hash} - hashes.changed=true - } - return hash -} - -// The sidecar of the source at source, in allocator. found is false, with an -// empty one, when the file is not there; a file that cannot be read as a -// sidecar is an error, not the defaults, so a source is never imported in a -// way nobody chose nor given new ids over the ones it had. -sidecar_load :: proc(source:string,allocator:=context.allocator) -> (sidecar:Sidecar,found:bool,error:Error) { - sidecar.outputs=make([dynamic]Output,allocator) - path:=strings.concatenate({source,SIDECAR_EXTENSION},context.temp_allocator) - // A source that names a device has no file next to it. - if unsafe_path(path) {return} - data,file_error:=serial.file_read(path,context.temp_allocator) - if file_error==.Not_Found {return} - if file_error!=.None {return sidecar,false,error_make(.Not_Found,"%s could not be read",path)} - parsed,valid:=sidecar_parse(string(data),allocator) - if !valid {return sidecar,false,error_make(.Malformed,"%s is not the settings and outputs of an import (version %d): fix it, or delete it and import its source again, which gives its assets new ids",path,asset.SIDECAR_VERSION)} - delete(sidecar.outputs) - return parsed,true,{} -} - -// Writes the sidecar of the source at source, unless the file already says the same. -sidecar_save :: proc(source:string,sidecar:Sidecar) -> bool { - path:=strings.concatenate({source,SIDECAR_EXTENSION},context.temp_allocator) - text:=sidecar_text(sidecar,context.temp_allocator) - if data,read_error:=os.read_entire_file(path,context.temp_allocator);read_error==nil && string(data)==text {return true} - return serial.file_write_atomic(path,transmute([]u8)text)==.None -} - -// The settings kept with a source, the defaults (found false) when it has no sidecar. -settings_load :: proc(source:string) -> (settings:Import_Settings,found:bool,error:Error) { - sidecar:Sidecar - sidecar,found,error=sidecar_load(source,context.temp_allocator) - return sidecar.settings,found,error -} - -// The asset the import of a source gave what the source names name, of kind; -// false when its sidecar lists none. -output_id :: proc(source:string,kind:asset.Fourcc,name:string) -> (id:base.Asset_Id,found:bool) { - sidecar,_,error:=sidecar_load(source,context.temp_allocator) - if error.code!=.None {return} - for output in sidecar.outputs { - if output.kind==kind && output.name==name {return output.id,true} - } - return -} diff --git a/cook/import/sound.odin b/cook/import/sound.odin index 0fa4d7102..cccd7f036 100644 --- a/cook/import/sound.odin +++ b/cook/import/sound.odin @@ -2,7 +2,6 @@ package importer import "core:fmt" import "core:mem" -import "core:os" import "core:path/filepath" import "core:strings" import "mirai:asset" @@ -75,7 +74,7 @@ i16_of :: proc(sample:f32) -> i16 { } // Imports a sound file as .sound. The sound is the root of the import: -// importing the file again writes the asset its sidecar lists (identity.odin). +// importing the file again gives the asset its package lists (identity.odin). import_sound :: proc(path:string,sound:Sound_Options={},options:Import_Options={},allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator scratch:mem.Dynamic_Arena @@ -101,20 +100,20 @@ import_sound :: proc(path:string,sound:Sound_Options={},options:Import_Options={ report_init(&report,allocator) w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { source:=w.source name:=filepath.stem(path) info:=import_info(source,name,0,u64(len(bytes)),base.hash64(bytes),u64(.Stream in desc.flags),import_version(0)) target:string - desc.id,target=root_take(&w,asset.KIND_SOUND,name,".sound") + output:int + desc.id,target,output=root_take(&w,asset.KIND_SOUND,name,".sound") desc.source=&info - existed:=os.exists(target) if save_error:=asset.sound_save(target,&w.types,desc,context.temp_allocator);save_error!=.None { ok=save_failed(&w,save_error,target) - } else { - record(&w,desc.id,asset.KIND_SOUND,name,target,source,existed) - ok=writer_finish(&w) + } else if ok=writer_put(&w,output,target);ok { + record(&w,output,name) + ok=writer_finish(&w,name,"sound") } } if !ok { diff --git a/cook/import/sound_test.odin b/cook/import/sound_test.odin index 2bcc4a06a..6ba0fc330 100644 --- a/cook/import/sound_test.odin +++ b/cook/import/sound_test.odin @@ -11,6 +11,7 @@ import "mirai:asset" import audio "mirai:audio" import "mirai:std/base" import "mirai:std/serial" +import "mirai:store" @(private="file") SINE_OGG :: #load("../../audio/testdata/sine_mono.ogg") // 0.5 s of 440 Hz at 44.1 kHz, amplitude 0.5 @(private="file") THREE_OGG :: #load("../../audio/testdata/three_channels.ogg") @@ -56,7 +57,7 @@ sound_open :: proc(t:^testing.T,path:string,types:^asset.Types,a:^asset.Asset) - @(test) a_wav_and_an_ogg_become_sounds_decoded_or_streamed_by_their_length :: proc(t:^testing.T) { defer free_all(context.temp_allocator) - root,folder:=test_assets(t,"sound") + _,folder,s:=test_assets(t,"sound") if folder=="" {return} types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) @@ -67,12 +68,12 @@ a_wav_and_an_ogg_become_sounds_decoded_or_streamed_by_their_length :: proc(t:^te path:=fmt.tprintf("%s/Step Left.wav",folder) _=os.write_entire_file(path,wav) - report,error:=import_sound(path,options={root=root}) + report,error:=import_sound(path,options={store=s}) testing.expect_value(t,error.code,Error_Code.None) testing.expect_value(t,len(report.assets),1) if len(report.assets)==1 { written:=report.assets[0] - testing.expect(t,written.kind==asset.KIND_SOUND && written.id!={} && strings.has_suffix(written.path,".sound") && strings.contains(written.path,"/.mirai/imported/") && written.bytes in if changed&{.Lods,.Parts,.Front,.Rig,.Rig_Crotch,.Rig_Neck,.Skeleton}!={} && kind!=.Model {fmt.wprintln(errors,"mirai import: --lods, --parts, --front, --skeleton and the --rig options apply only to models");return 2} if kind==.Model || kind==.Image { - stored,found,settings_error:=importer.settings_load(file) + stored,found,settings_error:=importer.settings_load(options,file) if settings_error.code!=.None { fmt.wprintfln(errors,"mirai import: %s: %s",file,settings_error.message) return 1 @@ -231,7 +231,6 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in } defer importer.report_destroy(&report) written_print(output,report.assets[:]) - if report.saved!="" {fmt.wprintfln(output,"sidecar %s",report.saved)} for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} return 0 } diff --git a/cook/import/sources/sync.odin b/cook/import/sources/sync.odin index 78edde95e..432e3899b 100644 --- a/cook/import/sources/sync.odin +++ b/cook/import/sources/sync.odin @@ -105,10 +105,6 @@ sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) - // The library, read once a source needs a format of its plugins or a model imports. library:Library_Read defer if library.release!=nil {library.release(library.state)} - hashes_path:=project.cache_path(top,importer.SOURCE_HASHES,allocator=context.temp_allocator) - hashes:=importer.source_hashes_load(hashes_path,context.temp_allocator) - defer importer.source_hashes_save(&hashes,hashes_path) - listed:=make(map[string]struct{},context.temp_allocator) // the files of the imported folder the sidecars list versions:=kind_versions(top,context.temp_allocator) complete:=true // The sources to import, found first so that each line says how many there are. @@ -121,11 +117,12 @@ sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) - for entry in entries { file:=fmt.tprintf("%s/%s",todo[next],entry.name) if entry.type==.Directory && !strings.has_prefix(entry.name,".") {append(&todo,file)} - source:=strings.trim_suffix(file,importer.SIDECAR_EXTENSION) + source:=strings.trim_suffix(file,asset.SIDECAR_EXTENSION) if entry.type!=.Regular || source==file || !os.exists(source) {continue} - sidecar,_,error:=importer.sidecar_load(source,context.temp_allocator) - if error.code!=.None { - append(&report.failures,strings.clone(error.message,allocator)) + text,read_error:=os.read_entire_file(file,context.temp_allocator) + sidecar,valid:=asset.sidecar_parse(string(text),context.temp_allocator) + if read_error!=nil || !valid { + append(&report.failures,fmt.aprintf("%s is not the settings and outputs of an import (version %d)",file,asset.SIDECAR_VERSION,allocator=allocator)) complete=false continue } @@ -157,22 +154,12 @@ sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) - head,_,_:=serial.file_read_head(source,buffer[:]) if format,found:=importer.format_find(formats,head,filepath.ext(source));found {version=importer.import_version(format.version)} } - // Final is current in either mode; Fast only for a sync of fast textures. - key:=importer.import_key(source,sidecar,version,code,false,&hashes) - current:=outputs_held(sidecar,imported,key) - if !current && options.fast_textures { - fast:=importer.import_key(source,sidecar,version,code,true,&hashes) - if outputs_held(sidecar,imported,fast) {key,current=fast,true} - } report.sources+=1 why:string switch { case options.rebuild:why="everything is built again" - case !current:why="what it is built from changed" case !documents_read(sidecar,®istry,versions):why="a document of it is of a version this build does not read" - case: - for file in outputs_files(sidecar,imported,key) {listed[filepath.base(file)]={}} - continue + case:why="what it builds is kept in the store" } append(&pending,Pending{source,formats,wholes,kind,version,code,why}) } @@ -201,49 +188,15 @@ sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) - if options.progress.procedure!=nil {fmt.wprintfln(options.progress," %s: %d assets (%d kept), %d warnings, %.1f s",named,len(written.assets),kept,len(written.warnings),time.duration_seconds(time.tick_since(started)))} for warning in written.warnings {append(&report.warnings,fmt.aprintf("%s: %s",p.source,warning,allocator=allocator))} importer.report_destroy(&written) - // What the import gives now: a part the source gained, or lost. - sidecar,_,error:=importer.sidecar_load(p.source,context.temp_allocator) - if error.code!=.None {complete=false;continue} - key:=importer.import_key(p.source,sidecar,p.version,p.code,options.fast_textures,&hashes) - for file in outputs_files(sidecar,imported,key) {listed[filepath.base(file)]={}} - } - if !complete {return} - entries,_:=os.read_all_directory_by_path(imported,context.temp_allocator) - for entry in entries { - if _,kept:=listed[entry.name];entry.type!=.Regular || kept || strings.has_suffix(entry.name,".tmp") {continue} - if os.remove(fmt.tprintf("%s/%s",imported,entry.name))==nil {report.removed+=1} } return } -// The files of the imported folder an import built from key keeps: one per -// output of the cache, else its stamp (importer.STAMP_EXTENSION). In the -// temp allocator. -@(private="file") -outputs_files :: proc(sidecar:importer.Sidecar,imported:string,key:u64) -> []string { - files:=make([dynamic]string,context.temp_allocator) - for output in sidecar.outputs { - known,_:=asset.kind_info(output.kind) - if asset.import_cached(output.kind) {append(&files,asset.imported_file(imported,output.id,key,known.extension,context.temp_allocator))} - } - if len(files)==0 && len(sidecar.outputs)>0 {append(&files,asset.imported_file(imported,sidecar.outputs[0].id,key,importer.STAMP_EXTENSION,context.temp_allocator))} - return files[:] -} - -// Whether the imported folder has every file of an import built from key. -@(private="file") -outputs_held :: proc(sidecar:importer.Sidecar,imported:string,key:u64) -> bool { - for file in outputs_files(sidecar,imported,key) { - if !os.exists(file) {return false} - } - return true -} - // Whether this build reads every document the sidecar lists that the assets // have: one of another version of its kind is made again by importing its // source. versions: of each kind this build knows (kind_versions). @(private="file") -documents_read :: proc(sidecar:importer.Sidecar,registry:^asset.Registry,versions:map[asset.Fourcc]u16) -> bool { +documents_read :: proc(sidecar:asset.Sidecar,registry:^asset.Registry,versions:map[asset.Fourcc]u16) -> bool { for output in sidecar.outputs { if asset.import_cached(output.kind) {continue} file,found:=asset.registry_find(registry,output.id) diff --git a/cook/import/space_test.odin b/cook/import/space_test.odin index 41eed906e..ef8458e6a 100644 --- a/cook/import/space_test.odin +++ b/cook/import/space_test.odin @@ -222,15 +222,15 @@ a_model_is_turned_to_minus_z_in_place_by_its_rig_or_by_the_front_its_settings_st the_front_of_a_source_is_a_setting_kept_with_it_that_makes_another_mesh :: proc(t:^testing.T) { for front in Front { stated:=Import_Settings{front=front} - parsed,ok:=sidecar_parse(sidecar_text({settings=stated},context.temp_allocator),context.temp_allocator) - testing.expectf(t,ok && parsed.settings==stated,"%v is read as written",front) + parsed,ok:=settings_parse(settings_text(stated,context.temp_allocator)) + testing.expectf(t,ok && parsed==stated,"%v is read as written",front) named,known:=front_from_name(strings.to_lower(fmt.tprint(front),context.temp_allocator)) testing.expectf(t,known && named==front,"%v is named in lowercase",front) for other in Front { if other!=front {testing.expectf(t,mesh_settings_hash(stated)!=mesh_settings_hash({front=other}),"%v and %v make the same mesh",front,other)} } } - _,ok:=sidecar_parse(`{"version": 3, "settings": {"front": "Up"}, "outputs": []}`,context.temp_allocator) + _,ok:=settings_parse(`{"front": "Up"}`) testing.expect(t,!ok,"a front the import does not know is not the settings of a source") } diff --git a/cook/import/write.odin b/cook/import/write.odin index f426ec418..bf57627af 100644 --- a/cook/import/write.odin +++ b/cook/import/write.odin @@ -17,16 +17,18 @@ import "mirai:std/base" import "mirai:std/jobs" import mm "mirai:std/math" import "mirai:std/serial" +import "mirai:store" Written :: struct { id:base.Asset_Id, kind:asset.Fourcc, name:string, - path:string, + path:string, // its blob in the store, or for a document its file in the folder of the documents + blob:store.Hash, source:string, // the source as imports name it bytes:int, detail:string, // what it holds, for a report: levels and colliders of a mesh, format and size of a texture - kept:bool, // its file already was the one this import gives: nothing was built + kept:bool, // a blob the store had was taken: nothing was built } Import_Report :: struct { @@ -36,15 +38,20 @@ Import_Report :: struct { // The asset of each mesh, material and image of the model, by row; zero for an image that was not written. mesh_ids,material_ids,image_ids,skeleton_ids:[]base.Asset_Id, animation_id:base.Asset_Id, - // The prefab of a model, the owner its parts name: the id the sidecar of - // the source lists and the file that has it, else new ones. Zero for a - // model without nodes that had no prefab. import_source writes it, with - // prefab_info. + // The prefab of a model, the owner its parts name: the id the previous + // manifest lists, else a new one, and the file it is written to: in the + // folder of the documents (none without one), or for Outputs the file + // output_written takes. Zero for a model without nodes that had no + // prefab. import_source writes it, with prefab_info. prefab:base.Asset_Id, prefab_path:string, prefab_info:asset.Import_Info, - settings:Import_Settings, // what the source was imported with: what its sidecar keeps - saved:string, // the sidecar of the source, which the import wrote or found saying the same + settings:Import_Settings, // what the source was imported with: what its package keeps + // The package the import wrote: its id, the hash of its manifest, its + // build, the key of the build and whether its textures are Final. + package_id:base.Asset_Id, + manifest,build,key:store.Hash, + final:bool, allocator:mem.Allocator, } @@ -64,7 +71,6 @@ report_destroy :: proc(report:^Import_Report) { delete(report.material_ids,report.allocator) delete(report.image_ids,report.allocator) delete(report.skeleton_ids,report.allocator) - delete(report.saved,report.allocator) delete(report.prefab_path,report.allocator) report^={} } @@ -87,22 +93,10 @@ report_take :: proc(dst,src:^Import_Report) { report_destroy(src) } -// A source path as imports name it: relative to options.root when it is inside, with forward slashes. -source_path :: proc(options:Import_Options,path:string,allocator:=context.temp_allocator) -> string { - absolute,error:=filepath.abs(path,context.temp_allocator) - if error!=nil {absolute=path} - cleaned,_:=filepath.clean(absolute,context.temp_allocator) - slashed,_:=strings.replace_all(cleaned,"\\","/",context.temp_allocator) - if options.root!="" { - root_absolute,_:=filepath.abs(options.root,context.temp_allocator) - root_clean,_:=filepath.clean(root_absolute,context.temp_allocator) - root,_:=strings.replace_all(root_clean,"\\","/",context.temp_allocator) - prefix:=strings.concatenate({strings.trim_right(root,"/"),"/"},context.temp_allocator) - if len(slashed)>len(prefix) && strings.equal_fold(slashed[:len(prefix)],prefix) { - return strings.clone(slashed[len(prefix):],allocator) - } - } - return strings.clone(slashed,allocator) +// A source path as imports name it, relative to its package: the name of its +// file, the same on every machine. +source_path :: proc(path:string,allocator:=context.temp_allocator) -> string { + return strings.clone(filepath.base(path_key(path)),allocator) } @(private) @@ -114,26 +108,34 @@ read_whole :: proc(path:string) -> ([]u8,serial.File_Error) { @(private) Writer :: struct { options:Import_Options, - folder:string, // where new documents go: the folder of the source, or the one of it its sidecar names - registry:^asset.Registry, // the assets of the root, where documents are found by id + folder:string, // where new documents go: under the folder of the documents; "": none are written there + registry:^asset.Registry, // the assets of the root, where documents are found by id; nil without a folder of the documents types:asset.Types, report:^Import_Report, - created:[dynamic]string, + created:[dynamic]string, // the documents the import wrote that were not there names:map[string]bool, // files of the folder new parts took, in lower case - source:string, // the source as imports name it (source_path) - sidecar:Sidecar, // of the source, as the import found it - taken:[]bool, // the outputs of sidecar the import gives again - outputs:[dynamic]Output, // what the import gives: the outputs of the sidecar it writes - paths:[dynamic]string, // the file of each of outputs - imported:string, // the imported folder of options.root path:string, // the source + origin:string, // the source as an absolute path with forward slashes + source:string, // the source as imports name it (source_path) + previous:asset.Package, // the manifest the source was ingested as, when stored + previous_manifest:store.Hash, + stored:bool, + package_id:base.Asset_Id, + taken:[]bool, // the assets of previous the import gives again + outputs:[dynamic]asset.Package_Asset, // what the import gives: the assets of the manifest it writes + paths:[dynamic]string, // the scratch file of each of outputs + targets:[dynamic]string, // the file of each document of outputs in the folder of the documents; "": none + blobs:[dynamic]store.Hash, // the blob of each of outputs once the store took it; zero: none + followed:[dynamic]Followed_Skeleton, // the baselines of the skeletons taken, which clips follow onto + scratch:string, // the folder parts are written in before the store takes them version:u32, // import_version of what reads the source - inputs:[dynamic]string, // the files read besides the source, as the sidecar lists them (writer_input) - key:u64, // import_key of the import, once a file of the cache is named; inputs are whole by then - previous:map[string]string, // the files of the imported folder under another key, by their name without it (writer_reuse) - previous_read:bool, + inputs:[dynamic]string, // the files read besides the source, as the manifest lists them (writer_input) + fixed:bool, // identity and key are fixed (writer_identity): inputs are whole + identity,key:store.Hash, + candidates:[dynamic]Candidate, // the builds outputs may be taken from (writer_candidates) + candidates_read:bool, + fast:bool, // a texture taken or built is Fast owner:base.Asset_Id, // of the parts: the prefab of a model - stored:bool, // the source has a sidecar whole:bool, // writes a model for Outputs: its prefab and clips are parts by their names given:int, // of outputs, those a part of the same kind and name is (parts_plan) rng:base.Rng, @@ -142,6 +144,13 @@ Writer :: struct { error:Error, } +// A build of the package this machine has, whose blobs an import takes. +@(private) +Candidate :: struct { + build:asset.Build, + current:bool, // of what the import is made from now (its key) +} + // The message goes to the allocator of the writer: the temp one where the // failure happens may be the one that ran out of memory. @(private) @@ -164,17 +173,20 @@ writer_warn :: proc(w:^Writer,format:string,args:..any) { append(&w.report.warnings,fmt.aprintf(format,..args,allocator=w.allocator)) } +// Lists an output the store took in the report. @(private) -record :: proc(w:^Writer,id:base.Asset_Id,kind:asset.Fourcc,name,path,source:string,existed:bool,detail:="",kept:=false) { +record :: proc(w:^Writer,output:int,name:string,detail:="",kept:=false) { + o:=w.outputs[output] + path:=w.targets[output] + if path=="" {path,_=store.blob_path(w.options.store,w.blobs[output],context.temp_allocator)} size:int if info,error:=os.stat(path,context.temp_allocator);error==nil {size=int(info.size)} clone,_:=strings.clone(path,w.allocator) named,_:=strings.clone(name,w.allocator) - from,_:=strings.clone(source,w.allocator) + from,_:=strings.clone(w.source,w.allocator) described,_:=strings.clone(detail,w.allocator) - append(&w.report.assets,Written{id=id,kind=kind,name=named,path=clone,source=from,bytes=size,detail=described,kept=kept}) - if !existed {append(&w.created,clone)} - progress_written(w.options.progress,len(w.sidecar.outputs)) + append(&w.report.assets,Written{id=o.id,kind=o.kind,name=named,path=clone,blob=w.blobs[output],source=from,bytes=size,detail=described,kept=kept}) + progress_written(w.options.progress,len(w.previous.assets)) } @(private) @@ -191,74 +203,64 @@ import_info :: proc(source:string,name:string,index:int,size:u64,hash:u64,settin return info } -// The file of the cache an asset would be written to, open for its report, -// when it already is there: its name says it was built from what the import -// builds from now. Its owner, which the sidecar decides, is written again in -// its header when it changed. The caller closes it. -@(private="file") -writer_kept :: proc(w:^Writer,path:string,owner:base.Asset_Id,file:^asset.Asset) -> bool { - if w.options.rebuild || !os.exists(path) || asset.asset_load(file,path,context.temp_allocator)!=.None {return false} - if file.blob.header.owner==owner {return true} - if asset.header_owner_set(file.blob.data,owner)==.None && serial.file_write_atomic(path,file.blob.data)==.None { - file.blob.header.owner=owner - return true +// Reads the builds outputs may be taken from: that of what the import is made +// from now (which fixes it), Final or, with fast textures, Fast; and the one +// this machine chose for the previous manifest. Before any writer_reuse, on +// the writer's thread. +@(private) +writer_candidates :: proc(w:^Writer) { + if w.candidates_read {return} + w.candidates_read=true + w.candidates.allocator=w.keep + writer_identity(w) + s:=w.options.store + digits:[64]u8 + take :: proc(w:^Writer,s:^store.Store,namespace,name:string,current:bool) { + hash,error:=store.ref_get(s,namespace,name) + if error!=.None {return} + b,read_error:=asset.build_get(s,hash,w.keep) + if read_error==.None && b.package_id==w.package_id {append(&w.candidates,Candidate{b,current})} } - asset.asset_close(file) - return false + take(w,s,asset.REF_IMPORTS,store.hex(w.key,&digits),true) + if w.options.fast_textures {take(w,s,asset.REF_IMPORTS_FAST,store.hex(w.key,&digits),true)} + if w.stored {take(w,s,asset.REF_BUILDS,store.hex(asset.build_select(w.previous_manifest,&w.previous),&digits),false)} } -// The file of the imported folder that has the id and the extension of path -// under another key than that of the import, and its name without the key. -// The folder is read once, on the writer's thread. -@(private="file") -writer_previous :: proc(w:^Writer,path:string) -> (previous,unkeyed:string,found:bool) { - if !w.previous_read { - w.previous_read=true - w.previous=make(map[string]string,w.keep) - entries,_:=os.read_all_directory_by_path(w.imported,context.temp_allocator) - for entry in entries { - if _,key,named:=asset.imported_name(entry.name);named && key!=w.key && entry.type==.Regular { - w.previous[strings.concatenate({entry.name[:32],entry.name[49:]},w.keep)]=strings.clone(entry.fullpath,w.keep) - } +// The blob of a build of the package (writer_candidates) for the output id, +// to take without building it: of kind in the version it is written in now, +// with owner, and made from one of expected, which records what decides its +// bytes beyond the key (the image of a texture and its encoders, the mesh as +// read and what builds it). Never with rebuild. Reads what it is given only: +// any thread, with types of its own. +@(private) +writer_reuse :: proc(w:^Writer,types:^asset.Types,id:base.Asset_Id,kind:asset.Fourcc,owner:base.Asset_Id,expected:..asset.Import_Info) -> (hash:store.Hash,info:asset.Import_Info,found:bool) { + if w.options.rebuild {return} + known,_:=asset.kind_info(kind) + for &c in w.candidates { + for o in c.build.outputs { + if o.id!=id || o.kind!=kind || o.kind_version!=known.version {continue} + blob,path_error:=store.blob_path(w.options.store,o.blob,context.temp_allocator) + if path_error!=.None {continue} + header,recorded,imported,error:=asset.import_peek(blob,types) + if error!=.None || !imported || header.kind!=kind || header.owner!=owner || !slice.contains(expected,recorded) {continue} + return o.blob,recorded,true } } - name:=filepath.base(path) - if len(name)<49 {return} - unkeyed=strings.concatenate({name[:32],name[49:]},context.temp_allocator) - previous,found=w.previous[unkeyed] return } -// Takes the file an output of the cache had under another key as its file -// under the key of now (path), without building or writing it, when what it -// records it was built from is one of expected and it is of the version its -// kind is written in now: what is not in the key of the file but decides its -// bytes (the image of a texture and its encoders, the mesh as read and what -// builds it) is in what it records. False with rebuild or a file at path. +// The blob a texture had, built from the same image with the same settings and +// encoders (Import_Info.settings_hash): Final, or Fast for an import of fast +// textures while the texture cache holds no Final of it. fast: it is Fast. @(private="file") -writer_reuse :: proc(w:^Writer,kind:asset.Fourcc,path:string,expected:..asset.Import_Info) -> bool { - if w.options.rebuild || os.exists(path) {return false} - previous,unkeyed,found:=writer_previous(w,path) - if !found {return false} - header,info,imported,error:=asset.import_peek(previous,&w.types) - known,_:=asset.kind_info(kind) - if error!=.None || !imported || header.kind!=kind || header.kind_version!=known.version || !slice.contains(expected,info) {return false} - if os.rename(previous,path)!=nil {return false} - delete_key(&w.previous,unkeyed) - return true -} - -// Takes the file a texture had under another key when it was built from the -// same image with the same settings and encoders (Import_Info.settings_hash): -// Final, or Fast for an import of fast textures while the texture cache holds -// no Final of it. -@(private="file") -texture_reuse :: proc(w:^Writer,job:Texture_Job,path,source:string) -> bool { +texture_reuse :: proc(w:^Writer,types:^asset.Types,job:Texture_Job,id,owner:base.Asset_Id,source:string) -> (hash:store.Hash,fast,found:bool) { options:=texture_options(w,job) final:=import_info(source,job.name,job.index,job.size,job.hash,textures.settings_hash(options),job.version) - fast:=final - if w.options.fast_textures && !textures.cache_has(w.options.texture_cache,textures.cache_key_build(job.bytes,job.made,options)) {fast.settings_hash=fast_settings(options)} - return writer_reuse(w,asset.KIND_TEXTURE,path,final,fast) + quick:=final + if w.options.fast_textures && !textures.cache_has(w.options.texture_cache,textures.cache_key_build(job.bytes,job.made,options)) {quick.settings_hash=fast_settings(options)} + info:asset.Import_Info + hash,info,found=writer_reuse(w,types,id,asset.KIND_TEXTURE,owner,final,quick) + return hash,found && info.settings_hash!=final.settings_hash,found } // The slots of a mesh as its file stores them, naming the materials by row; in @@ -288,23 +290,6 @@ mesh_info :: proc(source:string,mesh:^Imported_Mesh,index:int,content:u64,slots: return import_info(source,mesh.name,index,size,hash,mesh_build_hash(settings),version) } -// Writes a material the source gives as desc. Over the file of the same -// material it follows what the source gives now and keeps what was changed -// after the last import, with the parameters, layers and hidden names it has. -@(private="file") -material_write :: proc(w:^Writer,path:string,existed:bool,desc:asset.Material_Desc) -> asset.Error { - desc:=desc - file:asset.Asset - if !existed || asset.asset_load(&file,path,context.temp_allocator)!=.None {return asset.material_save(path,&w.types,desc)} - defer asset.asset_close(&file) - if view,error:=asset.material_view(&file,&w.types);error==.None && file.blob.header.id==desc.id { - desc.info=asset.material_follow(&w.types,view,desc.imported^) - desc.parameters=asset.material_parameters_follow(view,desc.imported_parameters,context.temp_allocator) - desc.overlays,desc.hidden=view.overlays,view.hidden - } - return asset.material_save(path,&w.types,desc) -} - // What the source gives a material, its maps naming the textures of the // images by row. @(private) @@ -351,27 +336,31 @@ texture_options :: proc(w:^Writer,job:Texture_Job) -> textures.Build_Options { return image_build_options({srgb=job.srgb,normal_map=job.normal_map,green_down=job.green_down,alpha_cutoff=job.alpha_cutoff,keep_precision=job.keep_precision,channels=job.channels},w.options.settings) } -// Builds and writes one texture of source as the asset id at path; false when -// the image cannot be turned into one (a warning says why). +// Builds one texture of source as the asset id at path, or takes the blob it +// had, and puts it in the store as output; false when the image cannot be +// turned into one (a warning says why) or the store does not take it (the +// writer failed). @(private) -write_texture :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,source:string) -> bool { - return texture_written(w,job,texture_write(w,job,id,owner,path,source,w.options.worker),id,path,source) +write_texture :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,source:string,output:int) -> bool { + writer_candidates(w) + return texture_written(w,job,texture_write(w,job,id,owner,path,source,w.options.worker),output) } // What building and saving a texture gave, for its writer to record. @(private) Texture_Write :: struct { - kept:bool, // its file already was the one the import gives - existed:bool, // a file was at its path before + kept:bool, // a blob of a build was taken (hash) + hash:store.Hash, + fast:bool, // its blocks are Fast format:asset.Texture_Format, width,height,mips:int, built:textures.Error, saved:asset.Error, } -// Builds and saves a texture as write_texture does, or keeps its file, -// without changing w: on worker, so that the textures of a model are written -// on the job system at once. +// Builds and saves a texture as write_texture does, or finds the blob to take +// (writer_reuse), without changing w: on worker, so that the textures of a +// model are written on the job system at once. @(private) texture_write :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,source:string,worker:jobs.Worker_Ref) -> (out:Texture_Write) { import_options:=w.options @@ -382,10 +371,14 @@ texture_write :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,sour types.codecs=w.types.codecs options:=texture_options(w,job) info:=import_info(source,job.name,job.index,job.size,job.hash,textures.settings_hash(options),job.version) - if file:asset.Asset;writer_kept(w,path,owner,&file) { - defer asset.asset_close(&file) - if view,error:=asset.texture_view(&file,&types);error==.None { - return {kept=true,existed=true,format=view.info.format,width=int(view.info.width),height=int(view.info.height),mips=len(view.mips)} + if hash,fast,found:=texture_reuse(w,&types,job,id,owner,source);found { + file:asset.Asset + blob,_:=store.blob_path(w.options.store,hash,context.temp_allocator) + if asset.asset_load(&file,blob,context.temp_allocator)==.None { + defer asset.asset_close(&file) + if view,error:=asset.texture_view(&file,&types);error==.None { + return {kept=true,hash=hash,fast=fast,format=view.info.format,width=int(view.info.width),height=int(view.info.height),mips=len(view.mips)} + } } } cooked:=job.cooked @@ -398,20 +391,21 @@ texture_write :: proc(w:^Writer,job:Texture_Job,id,owner:base.Asset_Id,path,sour out.built=cooked.error if cooked.error!=.None {return} built:=&cooked.built + out.fast=cooked.settings!=info.settings_hash info.settings_hash=cooked.settings desc:=textures.built_desc(built,id,&info,types.codecs==.Best) desc.owner=owner - out.existed=os.exists(path) out.format,out.width,out.height,out.mips=built.format,built.width,built.height,len(built.mips) zone(import_options,"texture.save") out.saved=asset.texture_save(path,&types,desc) return } -// Records what texture_write gave; false, with a warning, for an image that -// does not become a texture. +// Puts what texture_write gave in the store as output and records it; false, +// with a warning, for an image that does not become a texture, or with the +// writer failed. @(private) -texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,id:base.Asset_Id,path,source:string) -> bool { +texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,output:int) -> bool { switch { case out.built==.Too_Large: writer_warn(w,"image %s is not used: a side is above the %d texels a texture holds; scale it down in the source",job.name,asset.TEXTURE_MAX_SIZE) @@ -420,9 +414,11 @@ texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,id:base.Asse writer_warn(w,"image %s is not used: %v",job.name,out.built) return false case out.saved!=.None: - return save_failed(w,out.saved,path) + return save_failed(w,out.saved,w.paths[output]) } - record(w,id,asset.KIND_TEXTURE,job.name,path,source,out.existed,texture_detail(out.format,out.width,out.height,out.mips),kept=out.kept) + if !(writer_take(w,output,out.hash) if out.kept else writer_put(w,output,w.paths[output])) {return false} + w.fast||=out.fast + record(w,output,job.name,texture_detail(out.format,out.width,out.height,out.mips),kept=out.kept) return true } @@ -430,6 +426,7 @@ texture_written :: proc(w:^Writer,job:Texture_Job,out:Texture_Write,id:base.Asse // them, each a job of the job system of the import: into out by row. @(private) textures_write :: proc(w:^Writer,texture_jobs:[]Texture_Job,parts:[]Part,source:string,out:[]Texture_Write) { + writer_candidates(w) Work :: struct {w:^Writer,jobs:[]Texture_Job,parts:[]Part,source:string,out:[]Texture_Write} work:=Work{w,texture_jobs,parts,source,out} run :: proc(work:^Work,first,last:int,worker:int) { @@ -456,15 +453,15 @@ texture_detail :: proc(format:asset.Texture_Format,width,height,mips:int) -> str return fmt.tprintf("%v %dx%d, %d mips",format,width,height,mips) } -// Starts the writes of the import of the source at path: reads its sidecar -// and, without options.registry, the assets of the root its documents are -// found among. folder: under the folder of the source, where the documents of -// a first import go ("" beside it). +// Starts the writes of the import of the source at path (writer_source). +// folder: under the folder of the documents, where the documents of a first +// import go ("" in it). @(private) writer_init :: proc(w:^Writer,path,folder:string,options:Import_Options,report:^Import_Report,allocator:mem.Allocator) -> bool { w^={options=options,report=report,allocator=allocator,version=import_version(0)} w.created.allocator=context.temp_allocator w.names.allocator=context.temp_allocator + w.followed.allocator=context.temp_allocator w.rng=base.rng_make(u64(time.time_to_unix_nano(time.now())),u64(uintptr(w))) asset.types_init(&w.types) w.types.codecs=asset.project_codecs(options.root,project.engine_root_find("",context.temp_allocator)) @@ -472,37 +469,36 @@ writer_init :: proc(w:^Writer,path,folder:string,options:Import_Options,report:^ return writer_source(w,path,folder) } -// Whether every output of the cache the sidecar lists has its file under the -// key of what the import builds from now, or one under another key that the -// writer may take (writer_reuse): then nothing is built ahead. +// Whether the build of what the import is made from now has a blob in the +// store for every output of the cache the previous manifest lists: then +// nothing is built ahead. It fixes what the import is made from. @(private) writer_current :: proc(w:^Writer) -> bool { - if w.options.rebuild || len(w.sidecar.outputs)==0 {return false} - for output in w.sidecar.outputs { - info,known:=asset.kind_info(output.kind) - if !known || !asset.import_cached(output.kind) {continue} - path:=imported_file(w,output.id,info.extension) - if _,_,previous:=writer_previous(w,path);!previous && !os.exists(path) {return false} + if w.options.rebuild || !w.stored {return false} + writer_candidates(w) + for a in w.previous.assets { + if !asset.import_cached(a.kind) {continue} + held:=false + for &c in w.candidates { + if !c.current {continue} + for o in c.build.outputs {held||=o.id==a.id && store.has(w.options.store,o.blob)} + } + if !held {return false} } return true } -@(private) -writer_rollback :: proc(w:^Writer) { - for path in w.created {os.remove(path)} -} - // Writes the meshes, materials, textures, skeletons and clips of a model as -// assets. Importing a source again writes each part again with the id its -// sidecar lists (identity.odin); a new document gets a random id and -// .mat, .skeleton or .anim in the folder of the import -// (model.folder beside the source the first time), and what nobody edits is a -// file of the imported folder of the root named by its id (sidecar.odin). -// The parts name the prefab of the model as their owner -// (Import_Report.prefab), which import_source writes. A mesh or a texture -// whose file already is the one it would write is kept as it is -// (Written.kept), unless options.rebuild. A failure removes the files this -// call created. +// the blobs of a build of its package (package.odin). Importing a source again +// gives each part the asset the previous manifest lists (identity.odin); a new +// one gets a random id. A document is also written or followed in the folder +// of the documents (documents.odin): .mat, .skeleton or +// .anim there (under model.folder the first time). The parts name the +// prefab of the model as their owner (Import_Report.prefab), which +// import_source writes. A mesh or a texture a build of the package already +// has, made from what it would be made from now, is taken as it is +// (Written.kept), unless options.rebuild. A failure removes the documents this +// call created and records nothing. import_write :: proc(model:^Imported_Model,options:Import_Options={},allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator scratch:mem.Dynamic_Arena @@ -513,11 +509,11 @@ import_write :: proc(model:^Imported_Model,options:Import_Options={},allocator:= w:Writer ok:=writer_init(&w,model.source,model.folder,options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) defer model_cooked_destroy(model) if ok { model_begin(&w,model) - ok=write_model(&w,model) && writer_finish(&w) + ok=write_model(&w,model) && writer_finish(&w,model.name if model.name!="" else filepath.stem(model.source),model.format) } if !ok { writer_rollback(&w) @@ -530,18 +526,19 @@ import_write :: proc(model:^Imported_Model,options:Import_Options={},allocator:= // Writes a model as parts of the import of o, with what import_write writes // one with: its meshes, materials, textures, skeletons and clips, each the -// output of the sidecar of its kind and name (then index), named as the model -// names them, a name with '/' in that folder under the one of the import. A -// part of the kind and name of one o already gave is that asset, which is not -// written again. When the model has nodes, its prefab is the scene of its -// name, the owner of its meshes and skeletons: the caller writes it -// (cook/prefab.model_prefab) and records it (output_written), unless -// prefab_path is "" (o gave it already). The report gives, in allocator, the -// ids by row as that of import_write and the prefab; what is written, the -// warnings and the programs of the model go to the report of o. The files the -// model was read from are inputs of the import (output_input). Its assets name -// the source of o, with its size, hash and version, and are made with its -// settings. A failure leaves in o what was written, for outputs_end. +// asset of the previous manifest of its kind and name (then index), named as +// the model names them, a name with '/' in that folder under the one of the +// documents. A part of the kind and name of one o already gave is that asset, +// which is not written again. When the model has nodes, its prefab is the +// scene of its name, the owner of its meshes and skeletons: the caller writes +// it at prefab_path (cook/prefab.model_prefab) and records it +// (output_written), unless prefab_path is "" (o gave it already). The report +// gives, in allocator, the ids by row as that of import_write and the prefab; +// what is written, the warnings and the programs of the model go to the +// report of o. The files the model was read from are inputs of the import +// (output_input). Its assets name the source of o, with its size, hash and +// version, and are made with its settings. A failure leaves in o what was +// written, for outputs_end. output_model :: proc(o:^Outputs,model:^Imported_Model,allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator w:=&o.w @@ -589,17 +586,17 @@ output_model :: proc(o:^Outputs,model:^Imported_Model,allocator:=context.allocat return report,{} } -// The error of an input given once the key of the import was fixed. +// The error of an input given once what the import is made from was fixed. @(private) input_late :: proc(file:string,allocator:mem.Allocator) -> Error { - return error_make(.Malformed,"%s is an input given after the first output of the cache was planned, when what the import is built from was fixed: give every input (output_input) before",file,allocator=allocator) + return error_make(.Malformed,"%s is an input given after the first output of the cache was planned, when what the import is made from was fixed: give every input (output_input) before",file,allocator=allocator) } // What a model is written with: its settings, its warnings and programs, and // the files it was read from. @(private="file") model_begin :: proc(w:^Writer,model:^Imported_Model) { - // The settings kept with the source are those of every import of it. + // The settings its package keeps are those of every import of it. if model.stored && !w.options.ignore_stored {w.options.settings=model.settings} w.report.settings=w.options.settings for warning in model.warnings {writer_warn(w,"%s",warning)} @@ -619,16 +616,16 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { name:=model.name if model.name!="" else filepath.stem(source) if chosen.skeleton!={} {return write_clips_onto(w,model,source,name,chosen.skeleton)} - // The prefab of the model: the one its sidecar lists, which a model that lost its nodes keeps. - // For Outputs, a scene of its name. + // The prefab of the model: the one its package lists, which a model that lost its nodes keeps. + // For Outputs, a scene of its name, written to the file output_written takes. listed:=false - for output in w.sidecar.outputs {listed||=output.kind==asset.KIND_SCENE && (!w.whole || output.name==name)} + for a in w.previous.assets {listed||=a.kind==asset.KIND_SCENE && (!w.whole || a.name==name)} w.owner={} if listed || len(model.nodes)>0 { prefab:=[1]Part{{kind=asset.KIND_SCENE,name=name,extension=".scene",only=!w.whole}} parts_plan(w,prefab[:]) w.owner=prefab[0].id - w.report.prefab,w.report.prefab_path=w.owner,strings.clone(prefab[0].path,w.allocator) + w.report.prefab,w.report.prefab_path=w.owner,strings.clone(prefab[0].path if w.whole else prefab[0].target,w.allocator) w.report.prefab_info=import_info(source,name,0,model.source_size,model.source_hash,0,model.version) } @@ -663,43 +660,48 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { parts_plan(w,parts[:]) reasons:=make([]string,len(model.materials),context.temp_allocator) parts_own(w,model,jobs[:],parts[:],first_material,reasons) - parts_record(w,parts[:]) - // What the writer keeps (its file under the key of now, or taken from - // another key) is not built; the rest is built ahead on a job system. - // Without one the writer builds each when it gets to it, holding one at a time. + // What a build of the package has, made from what it is made from now, + // is taken, not built; the rest is built ahead on a job system. Without + // one the writer builds each when it gets to it, holding one at a time. + writer_candidates(w) kept_images:=make([]bool,len(model.images),context.temp_allocator) - kept_meshes:=make([]bool,len(model.meshes),context.temp_allocator) + kept_meshes:=make([]store.Hash,len(model.meshes),context.temp_allocator) + given_meshes:=make([]bool,len(model.meshes),context.temp_allocator) if !w.options.rebuild { reuse_zone:=zone_begin(w.options,"import.reuse") defer base.profile_end(reuse_zone) for job,i in jobs { part:=parts[i] - kept_images[job.index]=part.given || os.exists(part.path) || texture_reuse(w,job,part.path,source) + _,_,found:=texture_reuse(w,&w.types,job,part.id,part.owner,source) + kept_images[job.index]=part.given || found } material_part_ids:=make([]base.Asset_Id,len(model.materials),context.temp_allocator) for i in 0..0 && mesh.skeleton>=0 && mesh.skeleton bool { if part.given {continue} info:=material_given(&material,image_ids) import_data:=import_info(source,material.name,i,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) parameters:=material_given_parameters(&material,image_ids) material_zone:=zone_begin(w.options,"material.save") - if error:=material_write(w,part.path,existed,{id=part.id,owner=part.owner,info=info,imported=&info,parameters=parameters,imported_parameters=parameters,source=&import_data});error!=.None {return save_failed(w,error,part.path)} - record(w,part.id,asset.KIND_MATERIAL,material.name,part.path,source,existed,fmt.tprintf("%s: %s",part.owner!={}?fmt.tprintf("part of %s",name):"of its own",reasons[i])) + if error:=asset.material_save(part.path,&w.types,{id=part.id,owner=part.owner,info=info,imported=&info,parameters=parameters,imported_parameters=parameters,source=&import_data});error!=.None {return save_failed(w,error,part.path)} + writer_put(w,part.output,part.path) or_return + record(w,part.output,material.name,fmt.tprintf("%s: %s",part.owner!={}?fmt.tprintf("part of %s",name):"of its own",reasons[i])) base.profile_end(material_zone) } @@ -722,36 +724,17 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { for skeleton,i in model.skeletons { part:=&parts[first_skeleton+i] w.report.skeleton_ids[i]=part.id - if part.given { - body:asset.Skeleton_Body - body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part - count,map_:=skeleton_body(nil,skeleton.bones,&body,&body_parts) - root_bones[i]=skeleton_root_bone(skeleton.bones,map_) if count>0 else -1 - continue - } - import_data:=import_info(source,skeleton.name,i,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) - // The file it replaces: what a person set on it stays (body parts, sockets of their own). - previous_file:asset.Asset - previous:^asset.Skeleton_View - previous_view:asset.Skeleton_View - loaded:bool - if existed && asset.asset_load(&previous_file,part.path,context.temp_allocator)==.None { - loaded=true - view,error:=asset.skeleton_view(&previous_file,&w.types) - if error==.None {previous_view=view;previous=&previous_view} - if error==.Version {writer_warn(w,"%s is of a version this build does not read: the body parts, axes and sockets set by hand on it were not kept; set them again",part.path)} - } - defer if loaded {asset.asset_close(&previous_file)} body:asset.Skeleton_Body body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part - count,map_:=skeleton_body(previous,skeleton.bones,&body,&body_parts) + count,map_:=skeleton_body(nil,skeleton.bones,&body,&body_parts) root_bones[i]=skeleton_root_bone(skeleton.bones,map_) if count>0 else -1 + if part.given {continue} + import_data:=import_info(source,skeleton.name,i,model.source_size,model.source_hash,0,model.version) sockets:=skeleton_sockets(model,skeleton) if count>0 {skeleton_grips(w,model,i,map_,&sockets)} - skeleton_sockets_kept(previous,skeleton.bones,&sockets) if error:=asset.skeleton_save(part.path,&w.types,{id=part.id,owner=part.owner,bones=skeleton.bones,sockets=sockets[:],body=&body if count>0 else nil,body_parts=body_parts[:count],source=&import_data});error!=.None {return save_failed(w,error,part.path)} - record(w,part.id,asset.KIND_SKELETON,skeleton.name,part.path,source,existed,fmt.tprintf("%d bones",len(skeleton.bones))) + writer_put(w,part.output,part.path) or_return + record(w,part.output,skeleton.name,fmt.tprintf("%d bones",len(skeleton.bones))) } w.report.mesh_ids=make([]base.Asset_Id,len(model.meshes),w.allocator) for &mesh,i in model.meshes { @@ -764,17 +747,23 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { if mesh.skeleton<0 || mesh.skeleton>=len(w.report.skeleton_ids) {return writer_fail(w,.Malformed,"mesh %s has no skeleton",mesh.name)} skeleton=w.report.skeleton_ids[mesh.skeleton] } - file:asset.Asset - kept_zone:=zone_begin(w.options,"mesh.kept") - if writer_kept(w,path,owner,&file) { - view,view_error:=asset.mesh_view(&file,&w.types) - if view_error==.None { - record(w,id,asset.KIND_MESH,mesh.name,path,source,true,mesh_detail(view.levels,view.colliders,len(view.collision_points),len(view.collision_indices)),kept=true) + if kept:=kept_meshes[i];kept!={} { + kept_zone:=zone_begin(w.options,"mesh.kept") + file:asset.Asset + blob,_:=store.blob_path(w.options.store,kept,context.temp_allocator) + if asset.asset_load(&file,blob,context.temp_allocator)==.None { + view,view_error:=asset.mesh_view(&file,&w.types) + detail:=mesh_detail(view.levels,view.colliders,len(view.collision_points),len(view.collision_indices)) if view_error==.None else "" + asset.asset_close(&file) + if view_error==.None { + writer_take(w,part.output,kept) or_return + record(w,part.output,mesh.name,detail,kept=true) + base.profile_end(kept_zone) + continue + } } - asset.asset_close(&file) - if view_error==.None {base.profile_end(kept_zone);continue} + base.profile_end(kept_zone) } - base.profile_end(kept_zone) slots:=mesh_slots(&mesh,material_ids) cooked:=mesh.cooked own:Cooked_Mesh @@ -793,49 +782,40 @@ write_model :: proc(w:^Writer,model:^Imported_Model) -> bool { id=id,owner=owner,geometry=geometry,slots=slots,joints=mesh.joints,skeleton=skeleton, colliders=collision.colliders,collision_points=collision.points,collision_indices=collision.indices,source=&import_data, } - existed:=os.exists(path) save_zone:=zone_begin(w.options,"mesh.save") saved:=asset.mesh_save(path,&w.types,desc) base.profile_end(save_zone) - if saved==.None {record(w,id,asset.KIND_MESH,mesh.name,path,source,existed,mesh_detail(geometry.levels,collision.colliders,len(collision.points),len(collision.indices)))} + detail:=mesh_detail(geometry.levels,collision.colliders,len(collision.points),len(collision.indices)) if cooked==&own {cooked_mesh_destroy(&own)} if saved!=.None {return save_failed(w,saved,path)} + writer_put(w,part.output,path) or_return + record(w,part.output,mesh.name,detail) } if len(model.animations)>0 && parts[animation].given {w.report.animation_id=parts[animation].id} if len(model.animations)>0 && !parts[animation].given { part:=&parts[animation] import_data:=import_info(source,name,0,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) clips:=make([]asset.Animation_Clip_Desc,len(model.animations),context.temp_allocator) copy(clips,model.animations) - // The file it replaces: the root a person set on a clip stays. - previous_file:asset.Asset - previous:asset.Animation_View - loaded:=existed && asset.asset_load(&previous_file,part.path,context.temp_allocator)==.None - defer if loaded {asset.asset_close(&previous_file)} - if loaded { - view_error:asset.Error - previous,view_error=asset.animation_view(&previous_file,&w.types) - if view_error==.Version {writer_warn(w,"%s is of a version this build does not read: the root bones set by hand on its clips were not kept; set them again",part.path)} - } for &clip in clips { index:int clip.skeleton,index=clip_skeleton(model,clip,w.report.skeleton_ids) - if index>=0 {clip.root=clip_root(model.skeletons[index].bones,root_bones[index],clip,previous)} + if index>=0 {clip.root=clip_root(model.skeletons[index].bones,root_bones[index],clip)} } desc:=asset.Animation_Desc{id=part.id,owner=part.owner,clips=clips,source=&import_data} if error:=asset.animation_save(part.path,&w.types,desc);error!=.None {return save_failed(w,error,part.path)} + writer_put(w,part.output,part.path) or_return w.report.animation_id=part.id - record(w,part.id,asset.KIND_ANIMATION,name,part.path,source,existed,fmt.tprintf("%d clips",len(model.animations))) + record(w,part.output,name,fmt.tprintf("%d clips",len(model.animations))) } return true } // Only the clips of a model, as an .anim of its own bound to a skeleton the project has // (Import_Settings.skeleton): the motion of another file onto a character imported before. Each clip -// keeps the root the model's own skeleton gives it, and the root a person set on the .anim it -// replaces. A clip that moves no bone at all (its markers alone, or curves) goes on that skeleton as -// well: only one whose bones the skeleton lacks is refused. +// keeps the root the model's own skeleton gives it. A clip that moves no bone at all (its markers +// alone, or curves) goes on that skeleton as well: only one whose bones the skeleton lacks is +// refused. @(private="file") write_clips_onto :: proc(w:^Writer,model:^Imported_Model,source,name:string,skeleton:base.Asset_Id) -> bool { if len(model.animations)==0 {return writer_fail(w,.Malformed,"%s has no clips to put on the skeleton of Import_Settings.skeleton",source)} @@ -845,14 +825,8 @@ write_clips_onto :: proc(w:^Writer,model:^Imported_Model,source,name:string,skel w.report.animation_id=part.id if part.given {return true} import_data:=import_info(source,name,0,model.source_size,model.source_hash,0,model.version) - existed:=os.exists(part.path) clips:=make([]asset.Animation_Clip_Desc,len(model.animations),context.temp_allocator) copy(clips,model.animations) - previous_file:asset.Asset - previous:asset.Animation_View - loaded:=existed && asset.asset_load(&previous_file,part.path,context.temp_allocator)==.None - defer if loaded {asset.asset_close(&previous_file)} - if loaded {previous,_=asset.animation_view(&previous_file,&w.types)} ids:=make([]base.Asset_Id,len(model.skeletons),context.temp_allocator) for &id in ids {id=skeleton} for &clip in clips { @@ -865,13 +839,14 @@ write_clips_onto :: proc(w:^Writer,model:^Imported_Model,source,name:string,skel body_parts:[len(anim.Body_Part)]asset.Skeleton_Body_Part bones:=model.skeletons[index].bones count,map_:=skeleton_body(nil,bones,&body,&body_parts) - clip.root=clip_root(bones,skeleton_root_bone(bones,map_) if count>0 else -1,clip,previous) + clip.root=clip_root(bones,skeleton_root_bone(bones,map_) if count>0 else -1,clip) } desc:=asset.Animation_Desc{id=part.id,owner=part.owner,clips=clips,source=&import_data} if error:=asset.animation_save(part.path,&w.types,desc);error!=.None {return save_failed(w,error,part.path)} + writer_put(w,part.output,part.path) or_return w.report.animation_id=part.id text:[base.ASSET_ID_TEXT]u8 - record(w,part.id,asset.KIND_ANIMATION,name,part.path,source,existed,fmt.tprintf("%d clips on skeleton %s",len(model.animations),base.asset_id_format(skeleton,text[:]))) + record(w,part.output,name,fmt.tprintf("%d clips on skeleton %s",len(model.animations),base.asset_id_format(skeleton,text[:]))) return true } @@ -913,16 +888,12 @@ skeleton_root_bone :: proc(bones:[]asset.Skeleton_Bone,map_:anim.Body_Map) -> in return parent if parent>=0 && parent (value:asset.Animation_Root) { - for &kept in previous.clips { - if base.text_get(kept.name[:])!=clip.name || !kept.root.translation && !kept.root.rotation {continue} - for &bone in bones {if bone.path==kept.root.bone {return kept.root}} - } +clip_root :: proc(bones:[]asset.Skeleton_Bone,root:int,clip:asset.Animation_Clip_Desc) -> (value:asset.Animation_Root) { if root<0 {return} for track in clip.tracks { if track.property!=.Position && track.property!=.Rotation || base.name_hash(track.target)!=bones[root].target || len(track.keys)<2 {continue} @@ -961,13 +932,13 @@ mesh_detail :: proc(levels:[]asset.Mesh_Level,colliders:[]asset.Collider,points, // Imports an image file as a texture of its own, read as options.image says // and keeping the coverage of options.alpha_coverage, unless the settings -// its sidecar keeps say otherwise. The texture is the root of the import: -// importing the file again writes the asset its sidecar lists. +// its package keeps say otherwise. The texture is the root of the import: +// importing the file again gives the asset its package lists. import_texture :: proc(path:string,options:Import_Options={},allocator:=context.allocator) -> (report:Import_Report,error:Error) { caller:=context.temp_allocator options:=options if !options.ignore_stored { - stored,found,settings_error:=settings_load(path) + stored,found,settings_error:=settings_load(options,path) if settings_error.code!=.None {return {},settings_error} if found {options.settings=stored} } @@ -980,7 +951,7 @@ import_texture :: proc(path:string,options:Import_Options={},allocator:=context. w:Writer ok:=writer_init(&w,path,"",options,&report,allocator) - defer asset.types_destroy(&w.types) + defer writer_close(&w) if ok { data,read_error:=read_whole(path) if read_error!=.None { @@ -992,10 +963,10 @@ import_texture :: proc(path:string,options:Import_Options={},allocator:=context. name=filepath.stem(path),bytes=data,size=u64(len(data)),hash=base.hash64(data), srgb=use==.Color,normal_map=use==.Normal || use==.Normal_Green_Down,green_down=use==.Normal_Green_Down,alpha_cutoff=options.alpha_coverage,version=import_version(0), } - id,target:=root_take(&w,asset.KIND_TEXTURE,job.name,".tex") - built:=write_texture(&w,job,id,{},target,source) + id,target,output:=root_take(&w,asset.KIND_TEXTURE,job.name,".tex") + built:=write_texture(&w,job,id,{},target,source,output) if w.error.code!=.None {ok=false} else if !built {ok=writer_fail(&w,.Unsupported,"%s is not an image that can be decoded",path)} - ok=ok && writer_finish(&w) + ok=ok && writer_finish(&w,job.name,"image") } } if !ok { @@ -1055,37 +1026,3 @@ skeleton_grips :: proc(w:^Writer,model:^Imported_Model,skeleton:int,map_:anim.Bo if found {append(sockets,socket)} } } - -// The sockets of the file an import replaces that it does not make again (one a person or an agent -// added, such as where a weapon hangs), on a bone it still has. Those it makes (named by the -// source, bone.*, grips) are made again from the source. -@(private="file") -skeleton_sockets_kept :: proc(previous:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,sockets:^[dynamic]asset.Skeleton_Socket) { - if previous==nil {return} - made:=len(sockets) - for &socket in previous.sockets { - name:=base.text_get(socket.name[:]) - if strings.has_prefix(name,"bone.") {continue} - again:=false - for &other in sockets[:made] {if other.name==socket.name {again=true;break}} - if again {continue} - for &bone in bones {if bone.path==socket.bone {append(sockets,socket);break}} - } -} - -// The body of an imported skeleton: found on its bones, keeping the parts a -// person set in the file it replaces when their bones are still there. -// Returns how many parts it wrote, 0 for a skeleton without a body, and the map they come from. -@(private="file") -skeleton_body :: proc(previous_file:^asset.Skeleton_View,bones:[]asset.Skeleton_Bone,body:^asset.Skeleton_Body,parts:^[len(anim.Body_Part)]asset.Skeleton_Body_Part) -> (count:int,map_:anim.Body_Map) { - previous:=anim.body_map_empty() - if previous_file!=nil { - previous=asset.skeleton_body_map(bones,previous_file.body,previous_file.body_parts) - for part in anim.Body_Part {if part not_in previous.manual {previous.bones[part],previous.confidence[part]=-1,0}} - } - found:bool - map_,found=asset.skeleton_body_detect(bones,previous) - if !found {return 0,map_} - count,_=asset.skeleton_body_rows(bones,map_,body,parts) - return count,map_ -} diff --git a/cook/prefab/prefab.odin b/cook/prefab/prefab.odin index 12e6381b9..9b7507539 100644 --- a/cook/prefab/prefab.odin +++ b/cook/prefab/prefab.odin @@ -282,9 +282,9 @@ model_import :: proc(path:string,formats:[]importer.Format,options:importer.Impo // prefab of its hierarchy; only its clips with Import_Settings.skeleton. model_write :: proc(model:^importer.Imported_Model,path:string,options:importer.Import_Options,allocator:=context.allocator) -> (report:importer.Import_Report,error:importer.Error) { if report,error=importer.import_write(model,options,allocator);error.code!=.None {return} - if len(model.nodes)==0 || report.settings.skeleton!={} {return report,{}} + if len(model.nodes)==0 || report.settings.skeleton!={} || report.prefab_path=="" {return report,{}} name:=filepath.stem(path) - source:=importer.source_path(options,path) + source:=importer.source_path(path) id,target:=report.prefab,report.prefab_path if options.world==nil { importer.report_destroy(&report) diff --git a/editor/asset_move.odin b/editor/asset_move.odin index 726e1a4da..c0b568cfb 100644 --- a/editor/asset_move.odin +++ b/editor/asset_move.odin @@ -7,7 +7,6 @@ import "core:os" import "core:path/filepath" import "core:strings" import "mirai:asset" -import importer "mirai:cook/import" import "mirai:editor/api" import "mirai:editor/history" import "mirai:host" @@ -107,10 +106,10 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess into,refused:=asset_folder_clean(r,folder) if refused!="" {return "",fmt.tprintf("folder: %s",refused)} if busy:=import_busy(s);busy!="" {return "",busy} - source:=strings.trim_suffix(sidecar,importer.SIDECAR_EXTENSION) + source:=strings.trim_suffix(sidecar,asset.SIDECAR_EXTENSION) target:=fmt.tprintf("%s/%s",asset.registry_file_path(r,into,context.temp_allocator),filepath.base(source)) if dir:=filepath.dir(target);!os.is_dir(dir) {_=os.make_directory_all(dir)} - if os.exists(target) || os.rename(source,target)!=nil || os.rename(sidecar,fmt.tprintf("%s%s",target,importer.SIDECAR_EXTENSION))!=nil { + if os.exists(target) || os.rename(source,target)!=nil || os.rename(sidecar,fmt.tprintf("%s%s",target,asset.SIDECAR_EXTENSION))!=nil { return "",fmt.tprintf("%s could not be moved to %s: a file is there, or another program has it open",source,target) } host.assets_scan(s.app.assets) diff --git a/editor/import.odin b/editor/import.odin index 8990bfaa7..24ab608c9 100644 --- a/editor/import.odin +++ b/editor/import.odin @@ -208,7 +208,7 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb _,rigged:=rig.? _,faced:=front.? if rigged || faced || skeleton!={} { - stored,_,settings_error:=importer.settings_load(source) + stored,_,settings_error:=importer.settings_load(options,source) if settings_error.code!=.None { import_say(s,"%s: %s",name,settings_error.message) return false @@ -221,7 +221,7 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb case .Image: options.image=sources.image_use(source) if use,given:=image.?;given { - stored,_,settings_error:=importer.settings_load(source) + stored,_,settings_error:=importer.settings_load(options,source) if settings_error.code!=.None { import_say(s,"%s: %s",name,settings_error.message) return false @@ -361,7 +361,6 @@ import_poll :: proc(s:^Session) { if !inside {written=false;break} append(&paths,path) } - if path,inside:=asset.registry_relative(&a.registry,report.saved);inside {append(&paths,path)} if written {host.assets_refresh(a,paths[:])} else {host.assets_scan(a)} base.profile_end(registry_zone) materials_zone:=base.profile_begin(profile,jobs.OWNER,"editor.materials") @@ -426,13 +425,15 @@ import_root :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (sidecar:s if file.imported!=0 && strings.has_prefix(file.path,mount.prefix) {root=filepath.dir(mount.folder)} } source=project.path(root,source,allocator=context.temp_allocator) - listed,has,load_error:=importer.sidecar_load(source,context.temp_allocator) - if load_error.code!=.None || !has || len(listed.outputs)==0 || listed.outputs[0].id!=id {return} + text,read_error:=os.read_entire_file(strings.concatenate({source,asset.SIDECAR_EXTENSION},context.temp_allocator),context.temp_allocator) + if read_error!=nil {return} + listed,valid:=asset.sidecar_parse(string(text),context.temp_allocator) + if !valid || len(listed.outputs)==0 || listed.outputs[0].id!=id {return} files:=make([dynamic]u32,context.temp_allocator) for output in listed.outputs { if at,known:=r.by_id[output.id];known && r.files[at].imported!=0 {append(&files,at)} } - return strings.concatenate({source,importer.SIDECAR_EXTENSION},context.temp_allocator),files[:] + return strings.concatenate({source,asset.SIDECAR_EXTENSION},context.temp_allocator),files[:] } // Waits for an import that runs and lets go of what the last one wrote: diff --git a/tools/demo_content/imported.odin b/tools/demo_content/imported.odin index 345773cdc..dbeadd25d 100644 --- a/tools/demo_content/imported.odin +++ b/tools/demo_content/imported.odin @@ -4,16 +4,20 @@ import "core:fmt" import "mirai:asset" import importer "mirai:cook/import" import "mirai:std/base" +import "mirai:store" // The ids of what the imports of the demo wrote. An import gives a part a -// random id the first time and keeps it every time after, in the sidecar of -// its source: the demo reads them from there. +// random id the first time and keeps it every time after, in the manifest of +// the package of its source in the store of the machine: the demo reads them +// from there. // The asset of kind that the import of source (relative to demo/) wrote for // what the source names name. @(private) imported_id :: proc(o:^Out,source:string,kind:asset.Fourcc,name:string) -> base.Asset_Id { - if id,found:=importer.output_id(path(o,source),kind,name);found {return id} + if s,error:=store.open(store.default_root(context.temp_allocator),context.temp_allocator);error==.None { + if id,found:=importer.output_id(&s,path(o,source),kind,name);found {return id} + } known,_:=asset.kind_info(kind) fmt.eprintfln("demo_content: the import of %s wrote no %s named %s",source,known.name,name) o.content.failed=true From 344bcdcb9b5029cb2e35b5f49a2ce45b1e132781 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:52:58 -0300 Subject: [PATCH 011/123] cook/cache: results are refs into the machine store Each result of cook/cache is a blob of mirai:store named by the ref (cache-, -). The store reaches the cache as a value (Import_Options.store, Sync_Options.store, Cook_Desc.store) and a miss asks the caller's cache.Source before building. A missing or damaged blob is a miss; a blob that no longer has its bytes is removed. Trimming by age and its stamp, the cache folders and --texture-cache/--mesh-cache go: the store owns space. mirai import takes --store. Co-Authored-By: Claude Opus 5.5 --- .../skills/mirai-assets/references/formats.md | 2 +- .agents/skills/mirai-imports/SKILL.md | 12 +- .../mirai-imports/references/imports.md | 45 +++--- .../skills/mirai-projects/references/build.md | 3 +- .../mirai-projects/references/projects.md | 1 - cook/build/build.odin | 10 +- cook/build/cook.odin | 14 +- cook/build/cook_bench_test.odin | 5 +- cook/build/cook_cache_test.odin | 15 +- cook/build/cook_test.odin | 5 +- cook/cache/cache.odin | 144 +++++++++++------- cook/cache/cache_test.odin | 133 ++++++++++++++++ cook/import/cook.odin | 43 ++++-- cook/import/model.odin | 8 +- cook/import/outputs_test.odin | 11 +- cook/import/sources/cache_test.odin | 57 ++++--- cook/import/sources/command.odin | 26 +++- cook/import/sources/cook_test.odin | 2 +- cook/import/sources/reimport_test.odin | 22 +-- cook/import/sources/support_test.odin | 39 +++++ cook/import/sources/sync.odin | 8 +- cook/import/write.odin | 2 +- cook/textures/cache.odin | 38 +++-- cook/textures/cache_test.odin | 52 ++----- editor/import.odin | 13 +- project/checkout.odin | 2 +- project/project.odin | 2 - 27 files changed, 487 insertions(+), 227 deletions(-) create mode 100644 cook/cache/cache_test.odin diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index cc2f9a366..2e777c5a6 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -291,7 +291,7 @@ sidecars; the importers, the registry and the build do not use them yet. imports; any machine with the manifest computes it, without the source, and a peer that built that key gives it. `build_select` combines the manifest with the version of each kind of the cache it holds, so a program without importers (the dev player) finds the build the refresh - chose. The folders of `cook/cache` name theirs in `cache-`. + chose. The caches of `cook/cache` name theirs in `cache-`. - `packages_resolve` lists, for each package of the list, its assets of the cache the project uses with what they depend on, at `/` as the sidecar lists them, with the path of their blob. A missing build lists them not built; a manifest or a build the store diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index 145a20858..36745512a 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -106,6 +106,10 @@ the assets. - **An import never touches another.** Two models that embed the same image get a texture each, encoded once in the content cache; no import edits the files or the sidecar of another source. Decided by the owner. +- **What imports and cooks build is kept as refs into the machine store** (`cook/cache`): a blob + named by `(cache-, -)`, found by any program or machine that shares blobs; + a miss asks the caller's `cache.Source` before building, and `cook/cache` removes nothing: the + store owns space ([texture cache](references/imports.md#texture-cache)). - **What nobody edits is a cache of the source, never versioned, whatever reads it.** The meshes, textures, sounds and fonts of every import, of an engine format or of a plugin's, are files of the imported folder of its root (`.mirai/imported/`), named by their ids and listed by where @@ -192,9 +196,11 @@ the assets. inverted green, round trip through `.tex`, cache by content; HDR, every BC6H mode, 16-bit and layers in `layers_test.odin`). The fixtures of `assets/fixtures/` and the models of `demo/assets` import (without warnings) and what is written reads back with `asset`. -- Caches: `cook/textures/cache_test.odin` and `cook/import/sources/cache_test.odin` (a texture or - a mesh that another import built is taken from the cache; one whose file under another key - records what it is built from now is taken without writing it, and one that records other +- Caches: `cook/cache/cache_test.odin` (a result kept through one store is found through another + of its root or through a source on a miss; a missing or damaged blob is a miss; hostile names), + `cook/textures/cache_test.odin` and `cook/import/sources/cache_test.odin` (a texture or a mesh + that another import built is taken from the cache or from a source; one whose file under another + key records what it is built from now is taken without writing it, and one that records other encoders or another builder is built again); `cook/build/cook_cache_test.odin` (a cook takes the blocks of a texture from the cache of its content) and `cook/import/sources/cook_test.odin` (the sync of a cook builds a fast texture diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index 2415c7556..e78a7064f 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -360,13 +360,17 @@ single thread and in the usual order: with or without a job system the same byte ### Texture cache What matters about a built texture does not depend on its id or its path: `cook/textures/cache.odin` -stores it by content in `.mirai/cache/textures` of the main checkout of the repository the project -is in (`project.content_cache_path(root, project.TEXTURE_CACHE)`; of the project itself outside a -repository), one `.tex` per key whose id is the key. Every project and worktree of the repository -shares it. - -- `Import_Options.texture_cache` is that folder (`""`, none); the editor, cooking and - `mirai import` of a map pass it, taking it from its project. +keeps it by content in the cache `textures` of the machine store (`cook/cache`), one texture per +key whose id is the key. Every project, worktree and program of the machine shares it, and a +machine that shares blobs with another finds what that one built. + +- `cook/cache` keeps each result as a blob of `mirai:store` named by the ref + (`cache-`, `-`) (`asset.cache_namespace`). The store reaches it as a value + (`Import_Options.store`, `Sync_Options.store`, `Cook_Desc.store`; nil, none): the editor's + imports, `mirai import` (the machine's store, or `--store`) and the cook of a build open the + machine's (`store.default_root`). Tests make one in a temporary folder. +- A miss asks `Import_Options.cache_source` (a `cache.Source`) once before building: it makes the + blob present in the store and sets the ref, or says it has none. - A texture that is not kept looks for its `textures.cache_key_build` (the bytes of the image, or the texels the format made with their size, layers and format, and the `settings_hash`): if it is there, its levels are written with the id and the `Import_Info` of this import; if not, it is @@ -379,15 +383,13 @@ shares it. holds. - [Cooking](../../mirai-projects/references/build.md#asset-cooking) stores there the blocks of an RGBA8 texture by `textures.cache_key_blocks`. -- A key names what made the blocks (`textures.encoders_id`), so checkouts with other encoders - share the folder without sharing entries. -- `cache_read` checks hash and key: a damaged file or one of another `TEXTURE_VERSION` is a miss and - is built again. Reading an entry marks it used. -- An entry belongs to no asset (an old one serves again when switching branch or setting): an - entry no build used for `cache.UNUSED_AGE` is deleted (`cache.trim`, at most once per - `cache.TRIM_EVERY` after a write), never one by size, so a project larger than any budget keeps - what its next import reads. Deleting the folder only costs time. The mechanism (file per key, - key id and version, read that marks use, age) is `cook/cache`, the same as the mesh cache. +- A key names what made the blocks (`textures.encoders_id`), so machines with other encoders + share the store without sharing results. +- A read checks the id (key and version) and the hash of the asset: a ref whose blob is missing, + damaged or of another version is a miss and is built again; a blob that no longer has the bytes + of its address is removed so that the build replaces it. +- A result belongs to no asset and `cook/cache` removes none: the store owns space, and what goes + is decided by the live set given to `store.sweep`. - An image only its format decodes (`Imported_Image.make`, added with `importer.image_made`) is keyed by its `data`, what the format makes the texels of: a texture the cache keeps for it is written without decoding it, and the format raises a version inside its data when the same data @@ -395,10 +397,10 @@ shares it. ### Mesh cache -A built mesh is worth by its geometry as read and its settings, not by its id: `mesh_cook` stores -it in `.mirai/cache/meshes` (`project.content_cache_path(root, project.MESH_CACHE)`, shared like -the texture cache; `Import_Options.mesh_cache`, which the editor, sync and `mirai import` of a map -pass), one `.mesh` per key with its levels, clusters and colliders, without materials. +A built mesh is worth by its geometry as read and its settings, not by its id: `mesh_cook` keeps +it in the cache `meshes` of the machine store (`importer.MESH_CACHE`, the same `cook/cache` and +`Import_Options.store` as the texture cache), one mesh per key with its levels, clusters and +colliders, without materials. - The key (`mesh_cache_key`) is the content of the mesh as the reader gives it (vertices, indices, skin, submeshes, morph targets and flags: `mesh_content_hash`) with `mesh_build_hash`; the @@ -408,8 +410,7 @@ pass), one `.mesh` per key with its levels, clusters and colliders, without mate - A mesh that is not kept looks for its key; if it is there, it is written with the id, the slots, the skeleton and the `Import_Info` of this import, the same bytes as building it. `rebuild` does not read the cache and rewrites it. -- It saves simplifying, the cluster DAG and the pages; reading the source, no. Its entries go by - age, as those of the texture cache. +- It saves simplifying, the cluster DAG and the pages; reading the source, no. ## Settings of a source diff --git a/.agents/skills/mirai-projects/references/build.md b/.agents/skills/mirai-projects/references/build.md index f7453a505..f19494941 100644 --- a/.agents/skills/mirai-projects/references/build.md +++ b/.agents/skills/mirai-projects/references/build.md @@ -249,7 +249,8 @@ sources and their sidecars ([sync](../../mirai-imports/references/imports.md#syn `cook_assets` (`cook/build/cook.odin`) leaves in a folder the assets the game can reach. The `Game` job calls it with the project assets, `cook.mirai`, the asset folders of the active plugins and the -texture cache of the project; a failure is `Cook_Failed` with `Result.cook.error` and `detail`. +machine store, whose texture cache it reads and fills; a failure is `Cook_Failed` with +`Result.cook.error` and `detail`. - **Roots.** Every `.scene` (code reaches a prefab only by instantiating it) and `.records` of the folder and what `cook.mirai` (`COOK_FILE`) names: the other assets only code refers to (a diff --git a/.agents/skills/mirai-projects/references/projects.md b/.agents/skills/mirai-projects/references/projects.md index 1fec2528a..c449748bd 100644 --- a/.agents/skills/mirai-projects/references/projects.md +++ b/.agents/skills/mirai-projects/references/projects.md @@ -192,7 +192,6 @@ the `Project` owns everything it points to. and, apart from that, in the project's order. A cycle between inactive plugins is not an error. - The caches of `.mirai/cache` (`cache_path`) and `.mirai/thumbs` cost only time if deleted: the [registry](../../mirai-assets/references/formats.md#registry), the - [textures](../../mirai-imports/references/imports.md#texture-cache), the [compressed animations](../../mirai-core/references/host.md#content) and the [previews](../../mirai-editor/references/panels.md#previews) are made again. - **A cache never makes a valid asset fail.** An entry is keyed on everything it depends on (the diff --git a/cook/build/build.odin b/cook/build/build.odin index a794c5a22..22b85b9ca 100644 --- a/cook/build/build.odin +++ b/cook/build/build.odin @@ -12,6 +12,7 @@ import "mirai:asset" import "mirai:project" import "mirai:std/base" import "mirai:std/subprocess" +import "mirai:store" // Builds of a project. A job generates the world and the main package of // its target and runs the compiler without blocking; whoever started it polls @@ -124,6 +125,7 @@ Job :: struct { copies:[]Copy, // Game: what goes next to the executable settings:string, // Game: its settings file: those of the engine and of the project merged, since it ships without the engine cook:Cook_Desc, // Game: the assets it ships; without assets, a project that has none + store:store.Store, // Game: the machine store, whose caches cook.store names; unopened: none plugin_assets:[dynamic]project.Asset_Folder, // Game: the folders of assets of its active plugins, in cook.plugins development:[dynamic]string, // Game: the folders of the active development plugins, which it must not reach plugins:[]Plugin_Library, // the plugin libraries the target links (plugins.odin) @@ -260,8 +262,14 @@ job_start :: proc(desc:Desc) -> ^Job { if p.plugins[index].manifest.development {append(&job.development,strings.clone(p.plugins[index].folder,a))} } if assets:=project.folder_path(p.root,.Assets,a);os.is_dir(assets) { - job.cook={assets=assets,output=project.folder_path(folder,.Assets,a),manifest=project.path(p.root,COOK_FILE,allocator=a),texture_cache=project.content_cache_path(p.root,project.TEXTURE_CACHE,allocator=a),mesh_cache=project.content_cache_path(p.root,project.MESH_CACHE,allocator=a),collision_cache=project.content_cache_path(p.root,project.COLLISION_CACHE,allocator=a),root=p.root,server=desc.server,quick=desc.dev,keys=project.path(job.work,fmt.tprintf("%s.cook",name),allocator=a),allocator=a} + job.cook={assets=assets,output=project.folder_path(folder,.Assets,a),manifest=project.path(p.root,COOK_FILE,allocator=a),collision_cache=project.content_cache_path(p.root,project.COLLISION_CACHE,allocator=a),root=p.root,server=desc.server,quick=desc.dev,keys=project.path(job.work,fmt.tprintf("%s.cook",name),allocator=a),allocator=a} job.cook.codecs=asset.project_codecs(p.root,p.engine_root) + if store_root:=store.default_root(context.temp_allocator);store_root!="" { + if opened,error:=store.open(store_root,a);error==.None { + job.store=opened + job.cook.store=&job.store + } + } job.plugin_assets=make([dynamic]project.Asset_Folder,a) for found in project.asset_folders(p,a) { if found.prefix!="" {append(&job.plugin_assets,found)} diff --git a/cook/build/cook.odin b/cook/build/cook.odin index 2e8caac9b..93f6d2f60 100644 --- a/cook/build/cook.odin +++ b/cook/build/cook.odin @@ -13,6 +13,7 @@ import "core:time" import "compressor:data" import "compressor:blocks" import "mirai:asset" +import "mirai:cook/cache" import "mirai:cook/far" import anim "mirai:std/anim" import "mirai:cook/textures" @@ -21,6 +22,7 @@ import "mirai:std/base" import "mirai:std/jobs" import "mirai:std/serial" import "mirai:std/subprocess" +import "mirai:store" // The assets a game ships: of the assets folder of a project, the ones the // game can reach, as the runtime loads them fastest. @@ -58,8 +60,8 @@ Cook_Desc :: struct { output:string, // the assets folder of the shipped game: it ends up holding what was cooked and nothing else manifest:string, // the cook.mirai of the project. "", or a file that does not exist: the scenes are the only roots plugins:[]project.Asset_Folder, // the assets of active plugins: every file a root, cooked under its prefix - texture_cache:string, // folder of compressed textures kept by content (textures.cache_read); "": none - mesh_cache:string, // folder of the meshes an import keeps (cook/import Import_Options); "": none + store:^store.Store, // whose texture cache keeps compressed textures by content (textures.cache_read); nil: none + cache_source:cache.Source, // asked on a miss of that cache before compressing collision_cache:string, // folder of the convex pieces of meshes kept by content (asset.mesh_collision_pieces); "": none codecs:asset.Codecs, // what the project ships with (build.compression) root:string, // of the project: the sources of imported assets are named relative to it. "": the folder of assets @@ -698,20 +700,20 @@ texture_cook :: proc(work:^Cook_Work,source,target:string,types:^asset.Types,wor view,error:=asset.texture_view(&file,types) if error!=.None {return} if view.info.format!=.RGBA8 {return false,true} - cache:=work.desc.texture_cache + kept:=cache.Cache{store=work.desc.store,source=work.desc.cache_source} // Blocks that nothing compresses keep the least error. rdo:=work.desc.codecs!=.None key:=textures.cache_key_blocks(file.blob.header.content_hash,rdo) - built,cached:=textures.cache_read(cache,key,types,context.temp_allocator) + built,cached:=textures.cache_read(kept,key,types,context.temp_allocator) if !cached { built_ok:bool if built,built_ok=textures.blocks_of(view,context.temp_allocator,worker,rdo);!built_ok {return} // Textures of one content build the same blocks: the first to get here keeps them. - if cache!="" { + if kept.store!=nil { sync.mutex_lock(&work.cache) if key not_in work.cached { work.cached[key]={} - textures.cache_write(cache,key,types,&built) + textures.cache_write(kept,key,types,&built) } sync.mutex_unlock(&work.cache) } diff --git a/cook/build/cook_bench_test.odin b/cook/build/cook_bench_test.odin index fc66bf781..cb9f654de 100644 --- a/cook/build/cook_bench_test.odin +++ b/cook/build/cook_bench_test.odin @@ -5,6 +5,7 @@ import "core:os" import "core:testing" import "mirai:project" import "mirai:std/base" +import "mirai:store" // The cook of a project as the job Game does it: filling an empty texture // cache and from it. Run with --release -define:COOK_BENCH=true; @@ -32,7 +33,9 @@ bench_cook_of_a_project_filling_its_texture_cache_and_from_it :: proc(t:^testing if found.prefix!="" {append(&plugins,found)} } desc.plugins=plugins[:] - desc.texture_cache=fmt.tprintf("%s/cache",folder) + kept,opened:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + desc.store=&kept filling:=cook_assets(desc) testing.expect_value(t,filling.error,Cook_Error.None) cached:=cook_assets(desc) diff --git a/cook/build/cook_cache_test.odin b/cook/build/cook_cache_test.odin index 7a286c90f..a20f36204 100644 --- a/cook/build/cook_cache_test.odin +++ b/cook/build/cook_cache_test.odin @@ -5,15 +5,20 @@ import "core:os" import "core:slice" import "core:testing" import "mirai:asset" +import "mirai:cook/cache" import "mirai:cook/textures" import "mirai:std/base" +import "mirai:store" @(test) a_cook_takes_the_blocks_of_a_texture_from_the_cache_of_its_content :: proc(t:^testing.T) { types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) folder:=base.test_folder(t,"cook-texture-cache") if folder=="" {return} - assets,cache:=fmt.tprintf("%s/assets",folder),fmt.tprintf("%s/cache",folder) + assets:=fmt.tprintf("%s/assets",folder) + kept_store,opened:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + texture_cache:=cache.Cache{store=&kept_store} os.make_directory_all(fmt.tprintf("%s/textures",assets)) source:=fmt.tprintf("%s/textures/wood.tex",assets) pixels:[8*8*4]u8 @@ -30,7 +35,7 @@ a_cook_takes_the_blocks_of_a_texture_from_the_cache_of_its_content :: proc(t:^te } // The first cook compresses the texture and keeps its blocks. - report:=cook_assets({assets=assets,output=fmt.tprintf("%s/first",folder),manifest=manifest,texture_cache=cache,allocator=context.temp_allocator}) + report:=cook_assets({assets=assets,output=fmt.tprintf("%s/first",folder),manifest=manifest,store=&kept_store,allocator=context.temp_allocator}) testing.expect(t,report.error==.None && report.textures_compressed==1) testing.expect(t,report.seconds>0,"the report says how long the cook took") format,blocks:=level(t,&types,fmt.tprintf("%s/first/textures/wood.tex",folder)) @@ -38,13 +43,13 @@ a_cook_takes_the_blocks_of_a_texture_from_the_cache_of_its_content :: proc(t:^te original:asset.Asset testing.expect_value(t,asset.asset_load(&original,source,context.temp_allocator),asset.Error.None) key:=textures.cache_key_blocks(original.blob.header.content_hash) - kept,found:=textures.cache_read(cache,key,&types,context.temp_allocator) + kept,found:=textures.cache_read(texture_cache,key,&types,context.temp_allocator) testing.expect(t,found && kept.format==.BC1 && slice.equal(kept.mips[0],blocks)) // The next cook reads them: blocks the cache was given are the ones it ships. kept.mips[0][0]~=0xff - testing.expect(t,textures.cache_write(cache,key,&types,&kept)) - report=cook_assets({assets=assets,output=fmt.tprintf("%s/second",folder),manifest=manifest,texture_cache=cache,allocator=context.temp_allocator}) + testing.expect(t,textures.cache_write(texture_cache,key,&types,&kept)) + report=cook_assets({assets=assets,output=fmt.tprintf("%s/second",folder),manifest=manifest,store=&kept_store,allocator=context.temp_allocator}) testing.expect(t,report.error==.None && report.textures_compressed==1) _,cached:=level(t,&types,fmt.tprintf("%s/second/textures/wood.tex",folder)) testing.expect(t,len(cached)==32 && cached[0]==blocks[0]~0xff && slice.equal(cached[1:],blocks[1:])) diff --git a/cook/build/cook_test.odin b/cook/build/cook_test.odin index fea892fda..ce7e4e8be 100644 --- a/cook/build/cook_test.odin +++ b/cook/build/cook_test.odin @@ -12,6 +12,7 @@ import anim "mirai:std/anim" import mm "mirai:std/math" import "mirai:std/base" import "mirai:std/jobs" +import "mirai:store" @(private="file") WOOD_COLOR :: base.Asset_Id{1,1} @(private="file") WOOD_NORMAL :: base.Asset_Id{1,2} @@ -393,7 +394,9 @@ a_cook_on_many_workers_writes_what_a_cook_on_one_does :: proc(t:^testing.T) { js:jobs.Job_System if !jobs.system_init(&js,{threads=threads,allocator=runtime.heap_allocator()}) {return {error=.Write_Failed}} defer jobs.system_destroy(&js) - return cook_assets({assets=assets,output=fmt.tprintf("%s/%s",folder,name),manifest=manifest,texture_cache=fmt.tprintf("%s/%s_cache",folder,name),worker={system=&js},allocator=context.temp_allocator}) + kept,opened:=store.open(fmt.tprintf("%s/%s_store",folder,name),context.temp_allocator) + if opened!=.None {return {error=.Write_Failed}} + return cook_assets({assets=assets,output=fmt.tprintf("%s/%s",folder,name),manifest=manifest,store=&kept,worker={system=&js},allocator=context.temp_allocator}) } one:=cook(folder,assets,manifest,"one",0) many:=cook(folder,assets,manifest,"many",7) diff --git a/cook/cache/cache.odin b/cook/cache/cache.odin index df6188f63..132d961d1 100644 --- a/cook/cache/cache.odin +++ b/cook/cache/cache.odin @@ -1,78 +1,106 @@ package cache import "core:fmt" -import "core:os" -import "core:time" import "mirai:asset" import "mirai:std/base" +import "mirai:store" -// What an import or a cook built, kept by what it was made from in a folder -// of the caches of a project (project.cache_path), so that building it again -// reads it instead. A file of the cache is an asset whose id is its key and -// the version of what built it: a builder that gives other bytes misses every -// earlier file. A file that does not read back whole is a miss too. A read -// marks the file used, and a file no build used for UNUSED_AGE is removed: -// whatever its size, the cache keeps what the builds of the last weeks read, -// so a project larger than any fixed budget does not evict what its next -// build reads. Deleting the folder costs only the time of building again. +// What an import or a cook built, kept by what it was made from in the +// machine store, so that building it again reads it instead, here or on +// another machine that shares blobs with this one. A result is the file of an +// asset, kept as a blob, whose id is its key and the version of what built +// it; the ref (cache-, -) names it (asset.cache_namespace), +// so a builder that gives other bytes misses every earlier result. A ref +// whose blob is missing or does not read back whole is a miss, and a blob that +// no longer has the bytes of its address is removed so that building again +// replaces it. The store owns space: the cache removes nothing it can read. -// A file of the cache no build used for this long is removed. -UNUSED_AGE :: 30*24*time.Hour -// A folder is looked over for unused files at most this often, after a -// write: one stat of its stamp otherwise. -TRIM_EVERY :: 24*time.Hour -// The file of a folder whose modification time is when it was last looked over. -TRIM_STAMP :: ".trimmed" +// Asked on a miss before building: makes the blob of the ref (namespace, +// name) complete in into and sets the ref there, or says it has none. +Fetch :: #type proc(user:rawptr,into:^store.Store,namespace,name:string) -> bool + +Source :: struct { + fetch:Fetch, // nil: none + user:rawptr, +} + +Cache :: struct { + store:^store.Store, // nil: nothing is kept + source:Source, +} // The id of the file of key, built by version. id :: proc "contextless" (key,version:u64) -> base.Asset_Id {return {key,version}} -// Where the file of key goes, in the temp allocator. -file_path :: proc(folder:string,key:u64,extension:string) -> string { - return fmt.tprintf("%s/%016x%s",folder,key,extension) +// The file of key built by version in the cache name, open in file for the +// caller to close. On a miss the source is asked once. False without one, or +// with one that does not read back whole. +load :: proc(c:Cache,name:string,key,version:u64,file:^asset.Asset,allocator:=context.allocator) -> bool { + namespace_buffer:[store.NAME_MAX]u8 + ref_buffer:[REF_BYTES]u8 + namespace,ref,ok:=names(c,name,key,version,&namespace_buffer,&ref_buffer) + if !ok {return false} + if load_here(c.store,namespace,ref,key,version,file,allocator) {return true} + return fetched(c,namespace,ref) && load_here(c.store,namespace,ref,key,version,file,allocator) } -// The file of key built by version, open in file for the caller to close and -// marked used. False without one, or with one that does not read back whole. -load :: proc(folder:string,key,version:u64,extension:string,file:^asset.Asset,allocator:=context.allocator) -> bool { - if folder=="" {return false} - path:=file_path(folder,key,extension) - if asset.asset_load(file,path,allocator)!=.None {return false} - if file.blob.header.id!=id(key,version) || !asset.blob_verify_hash(&file.blob) { - asset.asset_close(file) - return false - } - now:=time.now() - _=os.change_times(path,now,now) - return true +// Whether the cache name keeps a complete blob for key built by version, +// without reading it; the source is asked on a miss. +has :: proc(c:Cache,name:string,key,version:u64) -> bool { + namespace_buffer:[store.NAME_MAX]u8 + ref_buffer:[REF_BYTES]u8 + namespace,ref,ok:=names(c,name,key,version,&namespace_buffer,&ref_buffer) + if !ok {return false} + if has_here(c.store,namespace,ref) {return true} + return fetched(c,namespace,ref) && has_here(c.store,namespace,ref) } -// Whether a file can be written into folder, which is made when missing. -ready :: proc(folder:string) -> bool { - if folder=="" {return false} - return os.is_dir(folder) || os.make_directory_all(folder)==nil +// Keeps bytes, the file of an asset whose id is id(key,version), as the +// result of key in the cache name. False when it cannot be kept, which costs +// only the time of building it the next time. +put :: proc(c:Cache,name:string,key,version:u64,bytes:[]u8) -> bool { + namespace_buffer:[store.NAME_MAX]u8 + ref_buffer:[REF_BYTES]u8 + namespace,ref,ok:=names(c,name,key,version,&namespace_buffer,&ref_buffer) + if !ok {return false} + hash,error:=store.put_bytes(c.store,bytes) + return error==.None && store.ref_set(c.store,namespace,ref,hash)==.None } -// Removes the files of folder no build used for UNUSED_AGE, and marks the -// folder looked over. How many went. -trim :: proc(folder:string) -> (removed:int) { - now:=time.now() - entries,error:=os.read_all_directory_by_path(folder,context.temp_allocator) - if error!=nil {return} - for entry in entries { - if entry.type!=.Regular || entry.name==TRIM_STAMP || time.diff(entry.modification_time,now)<=UNUSED_AGE {continue} - if os.remove(entry.fullpath)==nil {removed+=1} - } - stamp:=fmt.tprintf("%s/%s",folder,TRIM_STAMP) - if !os.exists(stamp) {_=os.write_entire_file(stamp,nil)} - _=os.change_times(stamp,now,now) - return +@(private) +REF_BYTES :: 33 // -, each 16 hex digits + +// The namespace and the name of the ref of key built by version; not ok +// without a store or for a name the store refuses. +@(private) +names :: proc(c:Cache,name:string,key,version:u64,namespace_buffer:^[store.NAME_MAX]u8,ref_buffer:^[REF_BYTES]u8) -> (namespace,ref:string,ok:bool) { + if c.store==nil || name=="" {return} + namespace=asset.cache_namespace(name,namespace_buffer) + if !store.name_valid(namespace) {return} + return namespace,fmt.bprintf(ref_buffer[:],"%016x-%016x",key,version),true +} + +@(private) +fetched :: proc(c:Cache,namespace,ref:string) -> bool { + return c.source.fetch!=nil && c.source.fetch(c.source.user,c.store,namespace,ref) } -// Called after a file was written into folder: trims it when it was not -// looked over for TRIM_EVERY. -written :: proc(folder:string) { - stamp,error:=os.stat(fmt.tprintf("%s/%s",folder,TRIM_STAMP),context.temp_allocator) - if error==nil && time.diff(stamp.modification_time,time.now()) bool { + hash,error:=store.ref_get(s,namespace,ref) + return error==.None && store.has(s,hash) +} + +@(private) +load_here :: proc(s:^store.Store,namespace,ref:string,key,version:u64,file:^asset.Asset,allocator:=context.allocator) -> bool { + hash,error:=store.ref_get(s,namespace,ref) + if error!=.None || !store.has(s,hash) {return false} + path,path_error:=store.blob_path(s,hash,context.temp_allocator) + if path_error!=.None {return false} + if asset.asset_load(file,path,allocator)==.None { + if file.blob.header.id==id(key,version) && asset.blob_verify_hash(&file.blob) {return true} + asset.asset_close(file) + } + if store.verify(s,hash)==.Corrupt {_=store.remove(s,hash)} + return false } diff --git a/cook/cache/cache_test.odin b/cook/cache/cache_test.odin new file mode 100644 index 000000000..69b6a2951 --- /dev/null +++ b/cook/cache/cache_test.odin @@ -0,0 +1,133 @@ +package cache + +import "core:fmt" +import "core:os" +import "core:slice" +import "core:strings" +import "core:testing" +import "mirai:asset" +import "mirai:std/base" +import "mirai:store" + +@(private="file") +NAME :: "textures" + +// The file of a result: a small texture whose id is that of key built by version. +@(private="file") +result_bytes :: proc(types:^asset.Types,key,version:u64,fill:u8) -> []u8 { + pixels:=make([]u8,4*4*4,context.temp_allocator) + for &value in pixels {value=fill} + mips:=[1][]u8{pixels} + doc:asset.Document + if asset.texture_document(&doc,types,{id=id(key,version),width=4,height=4,format=.RGBA8,mips=mips[:]},context.temp_allocator)!=.None {return nil} + defer asset.document_destroy(&doc) + bytes,_:=asset.document_bytes(&doc,context.temp_allocator) + return bytes +} + +@(private="file") +store_at :: proc(t:^testing.T,root:string,loc:=#caller_location) -> (s:store.Store,ok:bool) { + error:store.Error + s,error=store.open(root,context.temp_allocator) + return s,testing.expect_value(t,error,store.Error.None,loc=loc) +} + +// Copies what the store at user keeps, as a peer would give it. +@(private="file") +copy_fetch :: proc(user:rawptr,into:^store.Store,namespace,name:string) -> bool { + from:=(^store.Store)(user) + hash,error:=store.ref_get(from,namespace,name) + if error!=.None {return false} + path,path_error:=store.blob_path(from,hash,context.temp_allocator) + if path_error!=.None {return false} + if put,put_error:=store.put_file(into,path);put_error!=.None || put!=hash {return false} + return store.ref_set(into,namespace,name,hash)==.None +} + +@(private="file") +blob_of :: proc(s:^store.Store,key,version:u64) -> store.Hash { + namespace:[store.NAME_MAX]u8 + hash,_:=store.ref_get(s,asset.cache_namespace(NAME,&namespace),fmt.tprintf("%016x-%016x",key,version)) + return hash +} + +@(test) +a_result_is_found_by_its_ref_through_any_store_of_its_root_and_a_source_fills_a_miss :: proc(t:^testing.T) { + defer free_all(context.temp_allocator) + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + folder:=base.test_folder(t,"cache-refs") + if folder=="" {return} + root:=fmt.tprintf("%s/a",folder) + writer,writer_ok:=store_at(t,root) + reader,reader_ok:=store_at(t,root) + peer,peer_ok:=store_at(t,fmt.tprintf("%s/b",folder)) + if !writer_ok || !reader_ok || !peer_ok {return} + KEY,VERSION :: 0x1234,7 + file:asset.Asset + + // Kept through one Store value, found through another on the same root. + testing.expect(t,!load({store=&reader},NAME,KEY,VERSION,&file),"an empty cache has nothing") + testing.expect(t,put({store=&writer},NAME,KEY,VERSION,result_bytes(&types,KEY,VERSION,1))) + if testing.expect(t,load({store=&reader},NAME,KEY,VERSION,&file,context.temp_allocator)) { + testing.expect(t,file.blob.header.id==id(KEY,VERSION)) + asset.asset_close(&file) + } + testing.expect(t,has({store=&reader},NAME,KEY,VERSION) && !has({store=&reader},NAME,KEY,VERSION+1),"another version is another result") + + // A miss asks the source, which makes the blob present and sets the ref; one that has none is a miss. + from_writer:=Cache{store=&peer,source={fetch=copy_fetch,user=&writer}} + testing.expect(t,!load({store=&peer},NAME,KEY,VERSION,&file),"without a source the peer's result is not here") + if testing.expect(t,load(from_writer,NAME,KEY,VERSION,&file,context.temp_allocator)) {asset.asset_close(&file)} + testing.expect(t,has({store=&peer},NAME,KEY,VERSION),"what the source gave stays") + testing.expect(t,!load(from_writer,NAME,KEY+1,VERSION,&file) && !has(from_writer,NAME,KEY+1,VERSION)) + + // A ref whose blob is missing is a miss, and keeping the result again mends it. + hash:=blob_of(&reader,KEY,VERSION) + testing.expect_value(t,store.remove(&reader,hash),store.Error.None) + testing.expect(t,!load({store=&reader},NAME,KEY,VERSION,&file) && !has({store=&reader},NAME,KEY,VERSION)) + testing.expect(t,put({store=&writer},NAME,KEY,VERSION,result_bytes(&types,KEY,VERSION,1))) + if testing.expect(t,load({store=&reader},NAME,KEY,VERSION,&file,context.temp_allocator)) {asset.asset_close(&file)} + + // A blob that no longer has its bytes is a miss and goes, so that building again replaces it. + path,_:=store.blob_path(&reader,hash,context.temp_allocator) + whole,_:=os.read_entire_file(path,context.temp_allocator) + for damaged in ([][]u8{whole[:len(whole)/2],slice.concatenate([][]u8{whole[:len(whole)-1],{whole[len(whole)-1]~0xff}},context.temp_allocator)}) { + testing.expect(t,os.write_entire_file(path,damaged)==nil) + testing.expect(t,!load({store=&reader},NAME,KEY,VERSION,&file),"a damaged blob is a miss") + testing.expect(t,!store.has(&reader,hash),"and is removed") + testing.expect(t,put({store=&writer},NAME,KEY,VERSION,result_bytes(&types,KEY,VERSION,1))) + if testing.expect(t,load({store=&reader},NAME,KEY,VERSION,&file,context.temp_allocator)) {asset.asset_close(&file)} + } + + // A whole blob of another result under the ref is a miss, and stays: it is whole. + other:=result_bytes(&types,KEY+1,VERSION,2) + testing.expect(t,put({store=&writer},NAME,KEY,VERSION,other)) + testing.expect(t,!load({store=&reader},NAME,KEY,VERSION,&file)) + testing.expect(t,store.has(&reader,blob_of(&reader,KEY,VERSION))) +} + +@(test) +hostile_names_are_refused_and_any_key_is_kept_without_a_crash :: proc(t:^testing.T) { + defer free_all(context.temp_allocator) + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + folder:=base.test_folder(t,"cache-hostile") + if folder=="" {return} + s,ok:=store_at(t,folder) + if !ok {return} + c:=Cache{store=&s,source={fetch=copy_fetch,user=&s}} + bytes:=result_bytes(&types,1,1,3) + file:asset.Asset + for name in ([]string{"","Textures","../up","a/b","a\\b","nul\x00","..",strings.repeat("x",store.NAME_MAX,context.temp_allocator)}) { + testing.expectf(t,!put(c,name,1,1,bytes) && !load(c,name,1,1,&file) && !has(c,name,1,1),"%q is refused",name) + } + for key in ([]u64{0,1,max(u64)}) { + for version in ([]u64{0,max(u64)}) { + kept:=result_bytes(&types,key,version,4) + testing.expect(t,put(c,NAME,key,version,kept)) + if testing.expectf(t,load(c,NAME,key,version,&file,context.temp_allocator),"%x-%x",key,version) {asset.asset_close(&file)} + } + } + testing.expect(t,!put({},NAME,1,1,bytes) && !load({},NAME,1,1,&file) && !has({},NAME,1,1),"without a store nothing is kept") + _=put(c,NAME,9,9,nil) + testing.expect(t,!load(c,NAME,9,9,&file),"bytes that are no asset are no result") +} diff --git a/cook/import/cook.odin b/cook/import/cook.odin index 7f3f4d8c5..22ae99135 100644 --- a/cook/import/cook.odin +++ b/cook/import/cook.odin @@ -15,10 +15,12 @@ import "mirai:std/jobs" // builds when it gets to it. Either way they are the same bytes. An import // whose files of the cache are current builds nothing (writer_current). // -// The meshes built are kept by what they are made from in the mesh cache of -// the project (Import_Options.mesh_cache, cook/cache), as textures are in the -// texture cache: building the same geometry with the same settings again -// reads it. +// The meshes built are kept by what they are made from in the cache +// MESH_CACHE of the machine store (Import_Options.store, cook/cache), as +// textures are in theirs: building the same geometry with the same settings +// again reads it. + +MESH_CACHE :: "meshes" // The version of what builds a mesh: its file of the cache is built again // when the writer, the format of .mesh or the key of its settings change. @@ -105,12 +107,12 @@ mesh_cook :: proc(mesh:^Imported_Mesh,settings:Import_Settings,options:Import_Op types:asset.Types asset.types_init(&types,context.temp_allocator) defer asset.types_destroy(&types) - folder:=options.mesh_cache + c:=import_cache(options) key:u64 - if folder!="" { + if c.store!=nil { key=mesh_cache_key(cooked.content,settings) file:asset.Asset - if !options.rebuild && cache.load(folder,key,MESH_CACHE_VERSION,".mesh",&file,allocator) { + if !options.rebuild && cache.load(c,MESH_CACHE,key,MESH_CACHE_VERSION,&file,allocator) { if view,error:=asset.mesh_view(&file,&types);error==.None { cooked.geometry=view.geometry cooked.collision={view.colliders,view.collision_points,view.collision_indices} @@ -126,7 +128,7 @@ mesh_cook :: proc(mesh:^Imported_Mesh,settings:Import_Settings,options:Import_Op clustered:=levels_geometry(levels,mesh.flags,&cooked.geometry,allocator) base.profile_end(clusters_zone) if !clustered {cooked.failure=.Clusters;return} - if folder!="" && cache.ready(folder) { + if c.store!=nil { // The slots of the file of the cache name no material: an import names its own. slots:=make([]asset.Material_Slot,len(mesh.slots),context.temp_allocator) for slot,at in mesh.slots {slots[at].id=slot.id} @@ -134,10 +136,21 @@ mesh_cook :: proc(mesh:^Imported_Mesh,settings:Import_Settings,options:Import_Op id=cache.id(key,MESH_CACHE_VERSION),geometry=cooked.geometry,slots=slots,joints=mesh.joints, colliders=cooked.collision.colliders,collision_points=cooked.collision.points,collision_indices=cooked.collision.indices, } - if asset.mesh_save(cache.file_path(folder,key,".mesh"),&types,desc)==.None {cache.written(folder)} + doc:asset.Document + if asset.mesh_document(&doc,&types,desc,context.temp_allocator)!=.None {return} + defer asset.document_destroy(&doc) + if bytes,error:=asset.document_bytes(&doc);error==.None { + _=cache.put(c,MESH_CACHE,key,MESH_CACHE_VERSION,bytes) + mem.free_bytes(bytes) + } } } +// The caches of the store an import reads and fills. +import_cache :: proc(options:Import_Options) -> cache.Cache { + return {store=options.store,source=options.cache_source} +} + // Builds the levels of a texture, or reads them from the texture cache, into // the arena of cooked. bytes are the encoded image, the texels of made when // its width is not 0, or what make makes the texels of; decoded, when given, @@ -153,21 +166,21 @@ texture_cook :: proc(bytes:[]u8,made:textures.Source,decoded:textures.Image,buil asset.types_init(&types,context.temp_allocator) defer asset.types_destroy(&types) // The same image with the same settings, built before by any import, is read from the cache. - folder:=options.texture_cache + c:=import_cache(options) key:u64 - if folder!="" { + if c.store!=nil { key=textures.cache_key_build(bytes,made,build) if !options.rebuild { - if built,cached:=textures.cache_read(folder,key,&types,allocator);cached {cooked.built=built;return} + if built,cached:=textures.cache_read(c,key,&types,allocator);cached {cooked.built=built;return} } } if options.fast_textures { build.speed=.Fast cooked.settings=textures.settings_hash(build) - if folder!="" { + if c.store!=nil { key=textures.cache_key_build(bytes,made,build) if !options.rebuild { - if built,cached:=textures.cache_read(folder,key,&types,allocator);cached {cooked.built=built;return} + if built,cached:=textures.cache_read(c,key,&types,allocator);cached {cooked.built=built;return} } } } @@ -186,7 +199,7 @@ texture_cook :: proc(bytes:[]u8,made:textures.Source,decoded:textures.Image,buil case: cooked.built,cooked.error=textures.from_bytes(bytes,build,allocator,options.worker) } - if cooked.error==.None {textures.cache_write(folder,key,&types,&cooked.built)} + if cooked.error==.None {textures.cache_write(c,key,&types,&cooked.built)} } // The settings hash of a texture of build built Fast. diff --git a/cook/import/model.odin b/cook/import/model.odin index bc2baf0ab..f5a223fe6 100644 --- a/cook/import/model.odin +++ b/cook/import/model.odin @@ -4,10 +4,12 @@ import "core:fmt" import "core:mem" import "mirai:asset" import "mirai:core" +import "mirai:cook/cache" import "mirai:cook/textures" import "mirai:std/base" import "mirai:std/jobs" import mm "mirai:std/math" +import "mirai:store" // What an importer reads from a source file, as plain data that depends on no // engine component: the geometry as .mesh is built of it, the materials with @@ -59,12 +61,14 @@ Import_Options :: struct { registry:^asset.Registry, // the scanned assets of root, which the import does not change; nil: it scans them world:^core.World_Info, // the types of the game of root, which the prefab of a model is written in rebuild:bool, // writes every asset, also one whose file already is the one the import would write - texture_cache:string, // folder of textures built before, kept by image and settings (textures.cache_read); "": none + // The machine store, whose caches keep textures and meshes built before by + // what they are built from (textures.cache_read, mesh_cook); nil: none. + store:^store.Store, + cache_source:cache.Source, // asked on a miss of those caches before building // A texture is built Fast unless the texture cache keeps it as Final // (textures.Encode_Speed): the imports of the editor, which a cook // encodes again as Final. False: Final, what tools and the CLI build. fast_textures:bool, - mesh_cache:string, // folder of meshes built before, kept by geometry and settings (mesh_cook); "": none // The identity of the code of the plugin that imports the source whole // (sources.code_identity), 0 for the engine's: part of what its outputs // are built from (import_key). diff --git a/cook/import/outputs_test.odin b/cook/import/outputs_test.odin index dda99e784..be98448e0 100644 --- a/cook/import/outputs_test.odin +++ b/cook/import/outputs_test.odin @@ -8,6 +8,7 @@ import "core:testing" import "mirai:asset" import "mirai:cook/textures" import "mirai:std/base" +import "mirai:store" import mm "mirai:std/math" // What a plugin that imports a source whole writes with Outputs: the root and @@ -187,18 +188,20 @@ the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { an_image_only_its_format_decodes_is_taken_from_the_cache_without_decoding :: proc(t:^testing.T) { root,assets:=test_assets(t,"outputs-made") if root=="" {return} - cache:=fmt.tprintf("%s/cache",root) + kept,opened:=store.open(fmt.tprintf("%s/store",root),context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + cache:=&kept texels_of :: proc(data:[]u8,allocator:mem.Allocator) -> (made:textures.Source,ok:bool) { texels:=make([]u8,4*4*4,allocator) for &texel,at in texels {texel=u8(at*7)+data[0]} return {texels=texels,width=4,height=4,format=.RGBA8},true } refused :: proc(data:[]u8,allocator:mem.Allocator) -> (made:textures.Source,ok:bool) {return} - written :: proc(t:^testing.T,root,assets,name:string,make:Image_Make,cache:string) -> (texels:string) { + written :: proc(t:^testing.T,root,assets,name:string,make:Image_Make,cache:^store.Store) -> (texels:string) { source:=fmt.tprintf("%s/%s.map",assets,name) testing.expect(t,os.write_entire_file(source,name)==nil) o:Outputs - error:=outputs_begin(&o,source,name,u64(len(name)),base.hash64(transmute([]u8)name),import_version(0),{root=root,textures=.Uncompressed,texture_cache=cache}) + error:=outputs_begin(&o,source,name,u64(len(name)),base.hash64(transmute([]u8)name),import_version(0),{root=root,textures=.Uncompressed,store=cache}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model:=model_of("Crate","SM_Crate","T_Wood","M_Wood") model.images[0]={name="T_Wood",data=transmute([]u8)string("packed"),make=make,srgb=true} @@ -219,7 +222,7 @@ an_image_only_its_format_decodes_is_taken_from_the_cache_without_decoding :: pro first:=written(t,root,assets,"first",texels_of,cache) testing.expect(t,len(first)==4*4*4,"its format makes the texels") testing.expect(t,written(t,root,assets,"second",refused,cache)==first,"the cache gives the texture of the same data") - testing.expect(t,written(t,root,assets,"third",refused,"")=="","without the cache the texels are made, and here they cannot be") + testing.expect(t,written(t,root,assets,"third",refused,nil)=="","without the cache the texels are made, and here they cannot be") // Imported again unchanged, its outputs are current: a model read ahead for it cooks nothing. current :: proc(root,assets:string) -> bool { diff --git a/cook/import/sources/cache_test.odin b/cook/import/sources/cache_test.odin index 75f323f32..66a19438c 100644 --- a/cook/import/sources/cache_test.odin +++ b/cook/import/sources/cache_test.odin @@ -6,8 +6,10 @@ import "core:os" import "core:strings" import "core:testing" import "mirai:asset" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/prefab" +import "mirai:cook/textures" import "mirai:std/base" @(test) @@ -150,7 +152,8 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) root,folder:=test_project(t,"texture-cache") if folder=="" {return} - cache:=fmt.tprintf("%s/cache",root) + kept:=test_store(t,fmt.tprintf("%s/store",root)) + if kept==nil {return} bytes:=png_bytes(16,16,{200,100,50}) // Each import is another image of the project, with its own id: the levels are what the cache gives. imported :: proc(t:^testing.T,types:^asset.Types,folder,name:string,bytes:[]u8,options:importer.Import_Options) -> (format:asset.Texture_Format,mips:int,texels:string,flags:asset.Texture_Flags) { @@ -165,10 +168,10 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr if !testing.expect_value(t,view_error,asset.Error.None) {return} return view.info.format,len(view.mips),strings.clone(string(view.texels),context.temp_allocator),view.info.flags } - options:=importer.Import_Options{texture_cache=cache} + options:=importer.Import_Options{store=kept} format,mips,first,_:=imported(t,&types,folder,"a",bytes,options) testing.expect(t,format==.BC1 && mips==5) - entries:=cached_files(cache,".tex") + entries:=cached_refs(kept,textures.CACHE) if !testing.expect_value(t,len(entries),1) {return} // Another import of the same image has the same levels, made from the cache. @@ -176,13 +179,21 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr testing.expect(t,len(first)>0 && second==first) // What the cache holds is what the import takes: a texture put under its key is the one written. - kept:asset.Asset - testing.expect_value(t,asset.asset_load(&kept,entries[0].fullpath,context.temp_allocator),asset.Error.None) plain:[16*16*4]u8 - testing.expect_value(t,asset.texture_save(entries[0].fullpath,&types,{id=kept.blob.header.id,width=16,height=16,format=.RGBA8,mips={plain[:]}}),asset.Error.None) + planes:=[1][]u8{plain[:]} + flat:=textures.Built{width=16,height=16,format=.RGBA8,mips=planes[:]} + testing.expect(t,textures.cache_write({store=kept},cached_key(entries[0]),&types,&flat)) format,mips,_,_=imported(t,&types,folder,"c",bytes,options) testing.expect(t,format==.RGBA8 && mips==1) + // A store that misses asks its source before building: what a peer built is taken as it is. + peer:=options + peer.store=test_store(t,fmt.tprintf("%s/peer",root)) + peer.cache_source=copy_source(kept) + format,mips,_,_=imported(t,&types,folder,"p",bytes,peer) + testing.expect(t,format==.RGBA8 && mips==1,"the levels of the source, not built") + testing.expect_value(t,len(cached_refs(peer.store,textures.CACHE)),1) + // Rebuilding does not read the cache, and builds it again. rebuilt:=options rebuilt.rebuild=true @@ -196,7 +207,7 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr fast.fast_textures=true _,_,taken,_:=imported(t,&types,folder,"f",bytes,fast) testing.expect(t,taken==first,"the Final the cache holds") - fast.texture_cache=fmt.tprintf("%s/empty",root) + fast.store=test_store(t,fmt.tprintf("%s/empty",root)) flags:asset.Texture_Flags format,_,_,flags=imported(t,&types,folder,"g",bytes,fast) testing.expect(t,.Fast in flags && format==.BC1,"Fast, for the sync of a build to encode again") @@ -207,8 +218,10 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) root,_:=test_project(t,"mesh-cache") if root=="" {return} + kept:=test_store(t,fmt.tprintf("%s/store",root)) + if kept==nil {return} source:=copy_in(root,DEMO,"gallery/vase.obj","gallery/vase.mtl") - options:=importer.Import_Options{world=prefab.import_world(t),mesh_cache=fmt.tprintf("%s/cache",root)} + options:=importer.Import_Options{world=prefab.import_world(t),store=kept} colliders :: proc(t:^testing.T,types:^asset.Types,path:string) -> int { a:asset.Asset if !testing.expect_value(t,asset.asset_load(&a,path,context.temp_allocator),asset.Error.None) {return -1} @@ -225,7 +238,7 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ mesh_path,ok:=imported(t,source,options) if !ok {return} built,_:=os.read_entire_file(mesh_path,context.temp_allocator) - entries:=cached_files(options.mesh_cache,".mesh") + entries:=cached_refs(kept,importer.MESH_CACHE) if !testing.expect_value(t,len(entries),1) {return} // A mesh whose file is gone is built from the cache, to the same bytes. @@ -235,18 +248,29 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ testing.expect(t,len(built)>0 && string(again)==string(built),"what the cache gives is what building gives") // What the cache holds is what the import takes: a mesh put under its key, with a collider, is the one written. + key:=cached_key(entries[0]) cached:asset.Asset - if !testing.expect_value(t,asset.asset_load(&cached,entries[0].fullpath,context.temp_allocator),asset.Error.None) {return} + if !testing.expect(t,cache.load({store=kept},importer.MESH_CACHE,key,importer.MESH_CACHE_VERSION,&cached,context.temp_allocator)) {return} view,view_error:=asset.mesh_view(&cached,&types) if !testing.expect_value(t,view_error,asset.Error.None) {return} points:=[4][3]f32{{0,0,0},{1,0,0},{0,1,0},{0,0,1}} hull:=[1]asset.Collider{{kind=.Hull,rotation=1,point_count=4}} - desc:=asset.Mesh_Desc{id=cached.blob.header.id,geometry=view.geometry,slots=view.slots,colliders=hull[:],collision_points=points[:]} - testing.expect_value(t,asset.mesh_save(entries[0].fullpath,&types,desc),asset.Error.None) + doc:asset.Document + if !testing.expect_value(t,asset.mesh_document(&doc,&types,{id=cached.blob.header.id,geometry=view.geometry,slots=view.slots,colliders=hull[:],collision_points=points[:]},context.temp_allocator),asset.Error.None) {return} + hulled,bytes_error:=asset.document_bytes(&doc,context.temp_allocator) + testing.expect(t,bytes_error==.None && cache.put({store=kept},importer.MESH_CACHE,key,importer.MESH_CACHE_VERSION,hulled)) testing.expect(t,os.remove(mesh_path)==nil) if _,ok=imported(t,source,options);!ok {return} testing.expect_value(t,colliders(t,&types,mesh_path),1) + // A store that misses asks its source before building: what a peer built is taken as it is. + peer:=options + peer.store=test_store(t,fmt.tprintf("%s/peer",root)) + peer.cache_source=copy_source(kept) + testing.expect(t,os.remove(mesh_path)==nil) + if _,ok=imported(t,source,peer);!ok {return} + testing.expect_value(t,colliders(t,&types,mesh_path),1) + // Rebuilding does not read the cache, and builds it again. rebuilt:=options rebuilt.rebuild=true @@ -257,12 +281,3 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ last,_:=os.read_entire_file(mesh_path,context.temp_allocator) testing.expect(t,string(last)==string(built)) } - -// The files of a cache folder that hold what was built, of extension. -@(private="file") -cached_files :: proc(folder,extension:string) -> []os.File_Info { - entries,_:=os.read_all_directory_by_path(folder,context.temp_allocator) - files:=make([dynamic]os.File_Info,context.temp_allocator) - for entry in entries {if strings.has_suffix(entry.name,extension) {append(&files,entry)}} - return files[:] -} diff --git a/cook/import/sources/command.odin b/cook/import/sources/command.odin index 14fd4c57f..db08cfb8a 100644 --- a/cook/import/sources/command.odin +++ b/cook/import/sources/command.odin @@ -10,9 +10,9 @@ import "core:time" import "mirai:asset" import importer "mirai:cook/import" import "mirai:cook/textures" -import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" +import "mirai:store" // The command mirai import, which the import program of the engine runs: it // turns a model (OBJ, glTF, GLB, USD, FBX, or a format a plugin of the @@ -33,7 +33,9 @@ import "mirai:std/jobs" // under it whose outputs in its imported folder are missing or stale (sync): // what a checkout needs before it runs. --fast builds the textures of a sync // or of an import Fast, as the editor imports them, unless the texture cache -// keeps them Final: what a build that does not cook takes. +// keeps them Final: what a build that does not cook takes. What imports build +// is kept in the caches of the machine store (cook/cache), or of the store at +// --store. // The report goes to output, one asset per line: id, kind, bytes and path, and // under it, indented, what the asset holds; then the settings file and the // warnings. How far a long import is goes to errors while it runs, and a @@ -46,7 +48,7 @@ import "mirai:std/jobs" // Exit codes: 0 done, 1 failed, 2 misuse. @(private="file") -IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--rebuild] [--fast] [--texture-cache ] [--mesh-cache ] [--profile] [--release]" +IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--rebuild] [--fast] [--store ] [--profile] [--release]" // library reads the formats and the importers of whole sources the plugins of // a project add and the types of its game (in the import program of the @@ -56,6 +58,7 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in options:importer.Import_Options roots:=make([dynamic]string,context.temp_allocator) // of --sync profiled:bool + store_root:string // of the store whose caches keep what imports build; "": that of the machine // What the options change of the settings the source has. Change :: enum {Textures,Rdo_Lambda,Image,Alpha_Coverage,Lods,Parts,Front,Rig,Rig_Crotch,Rig_Neck,Skeleton} given:importer.Import_Settings @@ -97,8 +100,7 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in given.alpha_coverage=f32(cutoff) changed+={.Alpha_Coverage} } - case "--texture-cache": options.texture_cache,ok=value(args,&at) - case "--mesh-cache": options.mesh_cache,ok=value(args,&at) + case "--store": store_root,ok=value(args,&at) case "--profile": profiled=true case "--rebuild": options.rebuild=true case "--fast": options.fast_textures=true @@ -165,6 +167,16 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in } defer jobs.system_destroy(&js) options.worker={system=&js,worker=jobs.OWNER} + if store_root=="" {store_root=store.default_root(context.temp_allocator)} + kept:store.Store + if store_root!="" { + opened:store.Error + if kept,opened=store.open(store_root,context.temp_allocator);opened!=.None { + fmt.wprintfln(errors,"mirai import: the store at %s could not be opened (%v): pass another with --store",store_root,opened) + return 1 + } + options.store=&kept + } if len(roots)>0 { return sync_command(roots[:],options,library,output,errors) } @@ -172,8 +184,6 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in root,in_assets:=source_root(file,context.temp_allocator) if !in_assets {fmt.wprintfln(errors,"mirai import: %s is in no assets folder of a project or a plugin: move it beside what it makes, under assets/",file);return 1} options.root=root - if options.texture_cache=="" {options.texture_cache=project.content_cache_path(root,project.TEXTURE_CACHE,context.temp_allocator)} - if options.mesh_cache=="" {options.mesh_cache=project.content_cache_path(root,project.MESH_CACHE,context.temp_allocator)} formats:=FORMATS[:] kind:=source_kind(file) plugins:Library_Imports @@ -242,7 +252,7 @@ sync_command :: proc(roots:[]string,options:importer.Import_Options,library:Libr code:=0 for root in roots { started:=time.tick_now() - report:=sync(root,{texture_cache=options.texture_cache,mesh_cache=options.mesh_cache,rebuild=options.rebuild,fast_textures=options.fast_textures,worker=options.worker,library=library,progress=output}) + report:=sync(root,{store=options.store,rebuild=options.rebuild,fast_textures=options.fast_textures,worker=options.worker,library=library,progress=output}) defer sync_report_destroy(&report) for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} for failure in report.failures {fmt.wprintfln(errors,"mirai import: %s",failure)} diff --git a/cook/import/sources/cook_test.odin b/cook/import/sources/cook_test.odin index 95cdb4d38..f08c2ba1b 100644 --- a/cook/import/sources/cook_test.odin +++ b/cook/import/sources/cook_test.odin @@ -57,7 +57,7 @@ a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { report:=build.cook_assets({assets=assets,output=fmt.tprintf("%s/program",folder),manifest=manifest,root=folder,allocator=context.temp_allocator}) testing.expectf(t,report.error==.Source_Failed && strings.contains(report.detail,"assets/noise.tga") && strings.contains(report.detail,"mirai build game"),"%v %s",report.error,report.detail) - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",folder),mesh_cache=fmt.tprintf("%s/cache/meshes",folder)} + caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",folder))} fast:=caches fast.fast_textures=true kept:=sync(folder,fast,context.temp_allocator) diff --git a/cook/import/sources/reimport_test.odin b/cook/import/sources/reimport_test.odin index cde5d4d6d..86ca32c4b 100644 --- a/cook/import/sources/reimport_test.odin +++ b/cook/import/sources/reimport_test.odin @@ -17,6 +17,7 @@ import "mirai:core" import importer "mirai:cook/import" import "mirai:cook/prefab" import "mirai:cook/import/gltf" +import "mirai:cook/textures" import "mirai:project" import "mirai:std/base" @@ -300,9 +301,10 @@ two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other if root=="" {return} atlas_images(fmt.tprintf("%s/first",assets)) atlas_images(fmt.tprintf("%s/second",assets)) - cache:=fmt.tprintf("%s/cache/textures",root) + cache:=test_store(t,fmt.tprintf("%s/store",root)) + if cache==nil {return} first_source:=atlas_source(fmt.tprintf("%s/first",assets),"crate.gltf",{"atlas","waves","across","shared"}) - first,error:=import_source(first_source,{world=prefab.import_world(t),root=root,texture_cache=cache}) + first,error:=import_source(first_source,{world=prefab.import_world(t),root=root,store=cache}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return};defer importer.report_destroy(&first) atlas,_:=written_of(&first,asset.KIND_MATERIAL,"Atlas") texture,_:=written_of(&first,asset.KIND_TEXTURE,"atlas") @@ -316,10 +318,10 @@ two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other bytes,_:=os.read_entire_file(written.path,context.temp_allocator) append(&kept,Kept{written.path,bytes}) } - cached,_:=os.read_all_directory_by_path(cache,context.temp_allocator) + cached:=cached_refs(cache,textures.CACHE) second_source:=atlas_source(fmt.tprintf("%s/second",assets),"barrel.gltf",{"atlas","waves","across","shared"}) - second,second_error:=import_source(second_source,{world=prefab.import_world(t),root=root,texture_cache=cache}) + second,second_error:=import_source(second_source,{world=prefab.import_world(t),root=root,store=cache}) if !testing.expectf(t,second_error.code==.None,"%s",second_error.message) {return};defer importer.report_destroy(&second) second_atlas,wrote_material:=written_of(&second,asset.KIND_MATERIAL,"Atlas") second_texture,wrote_texture:=written_of(&second,asset.KIND_TEXTURE,"atlas") @@ -329,7 +331,7 @@ two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other bytes,_:=os.read_entire_file(file.path,context.temp_allocator) testing.expectf(t,slice.equal(bytes,file.bytes),"%s is as the first import left it",file.path) } - again,_:=os.read_all_directory_by_path(cache,context.temp_allocator) + again:=cached_refs(cache,textures.CACHE) testing.expectf(t,len(again)==len(cached),"the images are encoded once: %d entries of the cache, then %d",len(cached),len(again)) } @@ -348,10 +350,10 @@ a_checkout_without_its_imported_folder_gets_the_same_assets_from_its_sources :: sources:=[3]string{crate,barrel,fmt.tprintf("%s/sign.png",assets)} Built :: struct {path:string,id:base.Asset_Id,bytes:[]u8} built:=make([dynamic]Built,context.temp_allocator) - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",root),mesh_cache=fmt.tprintf("%s/cache/meshes",root),library=test_library} + caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",root)),library=test_library} context.user_ptr=prefab.import_world(t) for source in sources { - report,error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,texture_cache=caches.texture_cache,mesh_cache=caches.mesh_cache}) + report,error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,store=caches.store}) if !testing.expectf(t,error.code==.None,"%s: %s",source,error.message) {return} for written in report.assets {append(&built,Built{path=strings.clone(written.path,context.temp_allocator),id=written.id})} importer.report_destroy(&report) @@ -511,7 +513,7 @@ the_sources_of_a_plugin_format_build_in_the_cache_and_a_source_outside_the_asset mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") testing.expect(t,has_mesh && strings.contains(mesh.path,"/.mirai/imported/"),"what a plugin format gives that nobody edits is of the cache") // A clone: sync builds it again with the formats of the plugins, and says so without them. - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",root),mesh_cache=fmt.tprintf("%s/cache/meshes",root)} + caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",root))} testing.expect(t,base.remove_all(fmt.tprintf("%s/.mirai",project))==nil) lacking:=sync(project,caches,context.temp_allocator) testing.expectf(t,len(lacking.failures)==1 && strings.contains(lacking.failures[0],"no format"),"%v",lacking) @@ -574,8 +576,8 @@ sync_imports_again_a_whole_source_when_what_it_is_built_from_changes :: proc(t:^ level:=write_text(assets,"hall.level","LEVEL props/crate.gltf") state:=Level_Importer{kit=2} wholes:=[1]Whole_Source{{claims=level_claims,run=level_run,state=&state}} - caches:=Sync_Options{texture_cache=fmt.tprintf("%s/cache/textures",root),mesh_cache=fmt.tprintf("%s/cache/meshes",root)} - first,error:=import_source(level,{root=root,textures=.Uncompressed,texture_cache=caches.texture_cache,mesh_cache=caches.mesh_cache},wholes=wholes[:]) + caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",root))} + first,error:=import_source(level,{root=root,textures=.Uncompressed,store=caches.store},wholes=wholes[:]) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} defer importer.report_destroy(&first) mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") diff --git a/cook/import/sources/support_test.odin b/cook/import/sources/support_test.odin index 41de9397d..5e08d81c8 100644 --- a/cook/import/sources/support_test.odin +++ b/cook/import/sources/support_test.odin @@ -5,16 +5,19 @@ import "core:math/linalg" import "core:mem" import "core:os" import "core:path/filepath" +import "core:strconv" import "core:strings" import "core:testing" import "mirai:asset" import "mirai:core" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/import/gltf" import "mirai:cook/import/obj" import "mirai:cook/import/usd" import "mirai:std/base" import mm "mirai:std/math" +import "mirai:store" FIXTURES :: #directory+"/../../../assets/fixtures/" DEMO :: #directory+"/../../../demo/assets/" @@ -101,6 +104,42 @@ copy_in :: proc(root,from:string,files:..string) -> string { // A folder of a test as the root of a project, with its assets folder, where // the sources of its imports are. "" when there is none. +// A store at folder, in the temp allocator; nil when it cannot be opened. +test_store :: proc(t:^testing.T,folder:string,loc:=#caller_location) -> ^store.Store { + s,error:=store.open(folder,context.temp_allocator) + if !testing.expect_value(t,error,store.Error.None,loc=loc) {return nil} + kept:=new(store.Store,context.temp_allocator) + kept^=s + return kept +} + +// The results a cache of s keeps (cook/cache), in the temp allocator. +cached_refs :: proc(s:^store.Store,name:string) -> []store.Ref { + buffer:[store.NAME_MAX]u8 + refs,_:=store.ref_list(s,asset.cache_namespace(name,&buffer),context.temp_allocator) + return refs +} + +// The key of a result of a cache by the name of its ref, -. +cached_key :: proc(ref:store.Ref) -> u64 { + key,_:=strconv.parse_u64_of_base(ref.name[:min(16,len(ref.name))],16) + return key +} + +// A source that copies what the store at user keeps, as a peer would give it. +copy_source :: proc(from:^store.Store) -> cache.Source { + fetch :: proc(user:rawptr,into:^store.Store,namespace,name:string) -> bool { + from:=(^store.Store)(user) + hash,error:=store.ref_get(from,namespace,name) + if error!=.None {return false} + path,path_error:=store.blob_path(from,hash,context.temp_allocator) + if path_error!=.None {return false} + if put,put_error:=store.put_file(into,path);put_error!=.None || put!=hash {return false} + return store.ref_set(into,namespace,name,hash)==.None + } + return {fetch=fetch,user=from} +} + test_project :: proc(t:^testing.T,name:string) -> (root,assets:string) { root=base.test_folder(t,name) if root=="" {return} diff --git a/cook/import/sources/sync.odin b/cook/import/sources/sync.odin index 78edde95e..2a4b2c637 100644 --- a/cook/import/sources/sync.odin +++ b/cook/import/sources/sync.odin @@ -9,11 +9,13 @@ import "core:strings" import "core:time" import "mirai:asset" import "mirai:core" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" import "mirai:std/serial" +import "mirai:store" // Builds again what the imports of a project or a plugin keep in its imported // folder, which is not versioned, from the sources in its assets folder and @@ -49,7 +51,8 @@ Library_Imports :: struct { } Sync_Options :: struct { - texture_cache,mesh_cache:string, // "": those of the root (project.content_cache_path) + store:^store.Store, // whose caches keep what imports build (importer.Import_Options); nil: none + cache_source:cache.Source, rebuild:bool, // imports every source again // Textures built Fast, as the editor imports them, unless the texture // cache keeps them Final: what a build that does not cook needs. Outputs @@ -99,8 +102,7 @@ sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) - asset.registry_scan(®istry) import_options:=importer.Import_Options{ root=top,registry=®istry,rebuild=options.rebuild,worker=options.worker,fast_textures=options.fast_textures, - texture_cache=options.texture_cache if options.texture_cache!="" else project.content_cache_path(top,project.TEXTURE_CACHE,context.temp_allocator), - mesh_cache=options.mesh_cache if options.mesh_cache!="" else project.content_cache_path(top,project.MESH_CACHE,context.temp_allocator), + store=options.store,cache_source=options.cache_source, } // The library, read once a source needs a format of its plugins or a model imports. library:Library_Read diff --git a/cook/import/write.odin b/cook/import/write.odin index f426ec418..7a2a1c175 100644 --- a/cook/import/write.odin +++ b/cook/import/write.odin @@ -257,7 +257,7 @@ texture_reuse :: proc(w:^Writer,job:Texture_Job,path,source:string) -> bool { options:=texture_options(w,job) final:=import_info(source,job.name,job.index,job.size,job.hash,textures.settings_hash(options),job.version) fast:=final - if w.options.fast_textures && !textures.cache_has(w.options.texture_cache,textures.cache_key_build(job.bytes,job.made,options)) {fast.settings_hash=fast_settings(options)} + if w.options.fast_textures && !textures.cache_has(import_cache(w.options),textures.cache_key_build(job.bytes,job.made,options)) {fast.settings_hash=fast_settings(options)} return writer_reuse(w,asset.KIND_TEXTURE,path,final,fast) } diff --git a/cook/textures/cache.odin b/cook/textures/cache.odin index 18feb24bf..877163337 100644 --- a/cook/textures/cache.odin +++ b/cook/textures/cache.odin @@ -1,16 +1,18 @@ package textures import "core:hash/xxhash" -import "core:os" +import "core:mem" import "core:slice" import "mirai:asset" import "mirai:cook/cache" import "mirai:std/base" -// Built textures kept by what they are made from (cook/cache): an import that -// builds the same image with the same settings, and a cook that compresses -// the same RGBA8 texture, read the levels instead of making them again. Its -// files are textures built by BUILD_VERSION. +// Built textures kept by what they are made from in the cache CACHE +// (cook/cache): an import that builds the same image with the same settings, +// and a cook that compresses the same RGBA8 texture, read the levels instead +// of making them again. Its files are textures built by BUILD_VERSION. + +CACHE :: "textures" // An image built with options: its bytes, encoded, or the texels of made // (whose width is 0 for encoded bytes). @@ -26,15 +28,15 @@ cache_key_blocks :: proc(content_hash:u64,rdo:=true) -> u64 { } // Whether the cache keeps a texture under key, without reading it. -cache_has :: proc(folder:string,key:u64) -> bool { - return folder!="" && os.exists(cache.file_path(folder,key,".tex")) +cache_has :: proc(c:cache.Cache,key:u64) -> bool { + return cache.has(c,CACHE,key,BUILD_VERSION) } // The texture the cache keeps under key, its levels in allocator. False // without one, or with one that does not read back whole. -cache_read :: proc(folder:string,key:u64,types:^asset.Types,allocator:=context.allocator) -> (built:Built,ok:bool) { +cache_read :: proc(c:cache.Cache,key:u64,types:^asset.Types,allocator:=context.allocator) -> (built:Built,ok:bool) { file:asset.Asset - if !cache.load(folder,key,BUILD_VERSION,".tex",&file,context.temp_allocator) {return} + if !cache.load(c,CACHE,key,BUILD_VERSION,&file,context.temp_allocator) {return} defer asset.asset_close(&file) view,error:=asset.texture_view(&file,types) if error!=.None {return} @@ -45,11 +47,15 @@ cache_read :: proc(folder:string,key:u64,types:^asset.Types,allocator:=context.a return built,true } -// Keeps built under key; false when it cannot be written, which costs only -// the time of building it the next time. -cache_write :: proc(folder:string,key:u64,types:^asset.Types,built:^Built) -> bool { - if !cache.ready(folder) {return false} - if asset.texture_save(cache.file_path(folder,key,".tex"),types,built_desc(built,cache.id(key,BUILD_VERSION),nil))!=.None {return false} - cache.written(folder) - return true +// Keeps built under key; false when it cannot be kept, which costs only the +// time of building it the next time. +cache_write :: proc(c:cache.Cache,key:u64,types:^asset.Types,built:^Built) -> bool { + if c.store==nil {return false} + doc:asset.Document + if asset.texture_document(&doc,types,built_desc(built,cache.id(key,BUILD_VERSION),nil))!=.None {return false} + defer asset.document_destroy(&doc) + bytes,error:=asset.document_bytes(&doc) + if error!=.None {return false} + defer mem.free_bytes(bytes) + return cache.put(c,CACHE,key,BUILD_VERSION,bytes) } diff --git a/cook/textures/cache_test.odin b/cook/textures/cache_test.odin index e30b9fa08..1728bb242 100644 --- a/cook/textures/cache_test.odin +++ b/cook/textures/cache_test.odin @@ -1,14 +1,11 @@ package textures -import "core:fmt" -import "core:os" import "core:slice" import "core:testing" -import "core:time" import "mirai:asset" import "mirai:cook/cache" import "mirai:std/base" -import "mirai:std/serial" +import "mirai:store" @(test) built_textures_are_kept_by_image_and_settings_and_read_back_whole :: proc(t:^testing.T) { @@ -16,6 +13,9 @@ built_textures_are_kept_by_image_and_settings_and_read_back_whole :: proc(t:^tes types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) folder:=base.test_folder(t,"textures-cache") if folder=="" {return} + kept,opened:=store.open(folder,context.temp_allocator) + if !testing.expect_value(t,opened,store.Error.None) {return} + c:=cache.Cache{store=&kept} pixels:=make([]u8,16*16*4,context.temp_allocator) for &value,at in pixels {value=at%4==3?255:u8(at*7)} options:=Build_Options{srgb=true} @@ -27,49 +27,21 @@ built_textures_are_kept_by_image_and_settings_and_read_back_whole :: proc(t:^tes testing.expect(t,key!=cache_key_build(other,rgba8(other,16,16),options),"and other bytes") testing.expect(t,cache_key_blocks(1)!=cache_key_blocks(2)) - _,missing:=cache_read(folder,key,&types,context.temp_allocator) + _,missing:=cache_read(c,key,&types,context.temp_allocator) testing.expect(t,!missing,"an empty cache has nothing") built,error:=build(rgba8(pixels,16,16),options,context.temp_allocator) testing.expect_value(t,error,Error.None) - testing.expect(t,cache_write(folder,key,&types,&built)) - read,found:=cache_read(folder,key,&types,context.temp_allocator) + testing.expect(t,cache_write(c,key,&types,&built)) + read,found:=cache_read(c,key,&types,context.temp_allocator) testing.expect(t,found && read.width==built.width && read.height==built.height && read.format==built.format && read.flags==built.flags && len(read.mips)==len(built.mips)) for level,at in built.mips { side:=max(16>>uint(at),1) testing.expect(t,same_level(built.format,read.mips[at],level,side,side)) } - _,other_key:=cache_read(folder,key+2,&types,context.temp_allocator) + _,other_key:=cache_read(c,key+2,&types,context.temp_allocator) testing.expect(t,!other_key) - _,no_folder:=cache_read("",key,&types,context.temp_allocator) - testing.expect(t,!no_folder && !cache_write("",key,&types,&built)) - - // A file no build used for UNUSED_AGE goes, however small the folder; a read is a use. - trimmed:=base.test_folder(t,"textures-trim") - if trimmed=="" {return} - keys:[2]u64 - long_ago:=time.time_add(time.now(),-cache.UNUSED_AGE-time.Hour) - for &key_of,at in keys { - other[0]=u8(at) - key_of=cache_key_build(other,rgba8(other,16,16),options) - level_built,_:=build(rgba8(other,16,16),options,context.temp_allocator) - testing.expect(t,cache_write(trimmed,key_of,&types,&level_built)) - testing.expect(t,os.change_times(fmt.tprintf("%s/%016x.tex",trimmed,key_of),long_ago,long_ago)==nil) - } - _,first_read:=cache_read(trimmed,keys[0],&types,context.temp_allocator) - testing.expect(t,first_read) - testing.expect_value(t,cache.trim(trimmed),1) - _,kept_first:=cache_read(trimmed,keys[0],&types,context.temp_allocator) - _,kept_second:=cache_read(trimmed,keys[1],&types,context.temp_allocator) - testing.expect(t,kept_first && !kept_second,"the one read lately stays and the one unused for long goes") - - // A file of the cache that does not read back whole is no texture. - path:=fmt.tprintf("%s/%016x.tex",folder,key) - whole,_:=serial.file_read(path,context.temp_allocator) - damaged:=slice.clone(whole,context.temp_allocator) - damaged[len(damaged)-1]~=0xff - for bytes in ([][]u8{whole[:len(whole)/2],damaged}) { - testing.expect_value(t,serial.file_write_atomic(path,bytes),serial.File_Error.None) - _,read_back:=cache_read(folder,key,&types,context.temp_allocator) - testing.expect(t,!read_back) - } + _,no_store:=cache_read({},key,&types,context.temp_allocator) + testing.expect(t,!no_store && !cache_write({},key,&types,&built)) + testing.expect(t,cache_has(c,key) && !cache_has(c,key+2) && !cache_has({},key)) } + diff --git a/editor/import.odin b/editor/import.odin index 8990bfaa7..dc690881d 100644 --- a/editor/import.odin +++ b/editor/import.odin @@ -21,6 +21,7 @@ import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" import "mirai:std/serial" +import "mirai:store" import "mirai:ui" // Import from the content browser: a source file in the assets becomes @@ -77,6 +78,7 @@ Import_Job :: struct { whole:Source_Importer, // of a source a plugin imports whole; run is nil otherwise state:rawptr, // what whole.prepare gave options:importer.Import_Options, + store:store.Store, // the machine store, whose caches options.store names; unopened: none allocator:mem.Allocator, started:time.Tick, profiled:bool, @@ -196,8 +198,6 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb options:=importer.Import_Options{ root=strings.clone(root,allocator), world=s.info, - texture_cache=project.content_cache_path(s.project_root,project.TEXTURE_CACHE,allocator=allocator), - mesh_cache=project.content_cache_path(s.project_root,project.MESH_CACHE,allocator=allocator), fast_textures=true, } formats:=import_formats(s) @@ -233,6 +233,12 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb } job:=new(Import_Job,allocator) job^={kind=kind,options=options,allocator=allocator,started=time.tick_now(),profiled=profiled} + if store_root:=store.default_root(context.temp_allocator);store_root!="" { + if opened,error:=store.open(store_root,allocator);error==.None { + job.store=opened + job.options.store=&job.store + } + } job.source=strings.clone(source,allocator) if kind==.Model {import_formats_prepare(s,job)} if kind==.None && !claimed { @@ -313,7 +319,8 @@ import_job_free :: proc(job:^Import_Job) { if job.thread!=nil {thread.destroy(job.thread)} importer.report_destroy(&job.report) delete(job.source,allocator);delete(job.scene,allocator) - delete(job.options.root,allocator);delete(job.options.texture_cache,allocator);delete(job.options.mesh_cache,allocator) + delete(job.options.root,allocator) + if job.store.root!="" {store.close(&job.store)} for provider in job.plugin_formats { if provider.prepare!=nil && provider.release!=nil {provider.release(provider.format.data,allocator)} } diff --git a/project/checkout.odin b/project/checkout.odin index 087ac0217..a9c72b9f3 100644 --- a/project/checkout.odin +++ b/project/checkout.odin @@ -68,7 +68,7 @@ compressor_flags :: proc(engine_root:string,allocator:=context.allocator) -> (fl return } -// A cache of what imports build, kept by content (TEXTURE_CACHE, MESH_CACHE): +// A cache of what is built from content (COLLISION_CACHE): // what a file of it holds depends on what it was made from and on no project, // so every project of a checkout shares it, and so do the worktrees of the // repository:
/.mirai/cache/, where main is the main checkout of diff --git a/project/project.odin b/project/project.odin index 60a6e1767..c1a1fa662 100644 --- a/project/project.odin +++ b/project/project.odin @@ -33,8 +33,6 @@ PLUGIN_LIBRARIES :: "plugins/lib" // it and keeps so as not to make it again. Deleting them costs only time. CACHE_FOLDER :: "cache" REGISTRY_CACHE :: "registry" // in it: what the registry of the assets saw -TEXTURE_CACHE :: "textures" // in it: built textures, by image and settings -MESH_CACHE :: "meshes" // in it: built meshes, by geometry and settings ANIMATION_CACHE :: "animations" // in it: compressed .anim libraries, by asset and inputs COLLISION_CACHE :: "collision" // in it: the convex pieces of meshes, by their content FAR_CACHE :: "far" // in it: the far meshes of scenes the editor made, by what they are made of (asset.far_key) From bb1d750c4fad6e5dd92b5ca46f7e9c0b0976de4c Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 18:54:51 -0300 Subject: [PATCH 012/123] net: streams survive outages, long acks and gaps of small messages - Past RECORD_WINDOW a datagram still goes each keepalive, so a stream whose every datagram in flight was lost resumes; an interval that delivered next to nothing starts the rate again, and the rate grows only with acks. - The ack takes what the clock or inputs leave of the datagram (time_max_bits). - A reliable channel delivers only while the command buffer has room; held chunks pause reading datagrams until delivered. - Reliable windows grow by chunks acknowledged, bounded by the size those chunks had, so streams of small messages open their window. - The journal channel holds 256 KiB again. - The prefab replica test waits for the reads its jobs began instead of racing them with ticks that run faster than any frame. Co-Authored-By: Claude Opus 5.5 --- .../mirai-networking/references/networking.md | 18 +- net/channel.odin | 89 ++++++--- net/clock.odin | 9 + net/flow_test.odin | 184 +++++++++++++++++- net/host.odin | 18 +- net/peer.odin | 42 +++- plugins/multiplayer/runtime/journal.odin | 6 +- plugins/multiplayer/runtime/prefab_test.odin | 4 + 8 files changed, 314 insertions(+), 56 deletions(-) diff --git a/.agents/skills/mirai-networking/references/networking.md b/.agents/skills/mirai-networking/references/networking.md index f5d1d91ee..6f645fd5f 100644 --- a/.agents/skills/mirai-networking/references/networking.md +++ b/.agents/skills/mirai-networking/references/networking.md @@ -128,7 +128,9 @@ like `replication.hysteresis`) and the values it admits. before it arrived, from what the other end might not have resolved when reading the last ack this end received (`ack_known`), capped by `ACK_WINDOW`, as runs of arrived and missing datagrams in Elias gamma code: a stream with few losses acks thousands of datagrams in a few bytes. Everything - the sender has in flight arrives this way with its real fate, also after a receiver stall. + the sender has in flight arrives this way with its real fate, also after a receiver stall. The + ack takes only what the clock or the inputs leave of the datagram: runs that do not fit are left + out, and the sender takes those datagrams as lost. - A receiver acks within `ACK_DELAY`, or with what it sends anyway, and at once after `ACK_EVERY` datagrams that carried reliable chunks: a stream is acked as it arrives, and a connection without one sends no datagram more for it. @@ -145,7 +147,9 @@ like `replication.hysteresis`) and the values it admits. no ack. - **Send limit**: a token bucket with the connection's rate that stores at most one tick of rate (`send_burst`), so a network half that updates more often than the tick spreads the datagrams. - An update sends while the tokens last, up to `SENT_WINDOW` datagrams in flight. A reliable + An update sends while the tokens last, up to `RECORD_WINDOW` datagrams in flight; past it, one + datagram a keepalive still goes, so a connection whose every datagram in flight was lost to an + outage learns their fate from the ack of a newer one and resumes. A reliable channel with a datagram of chunks ready sends at every update the tokens allow; the rest goes once a tick. The object budget of each send is the lesser of `budget_bytes` and the tokens: replication follows the rate. @@ -156,7 +160,8 @@ like `replication.hysteresis`) and the values it admits. bytes per second for every connection (at most a quarter each interval), so connections on one path converge to equal shares and the queue stays short for the game traffic on it. Losses cut it only with a queue or past `CC_OVERRUN_SHARE`; a link that loses at random keeps its rate, which - never drops below `CC_MIN_RATE`. + never drops below `CC_MIN_RATE`. An interval that delivered next to nothing, as after an outage, + starts again from `CC_START_RATE`. - **Close**: `host_disconnect` tells the other end with its reason; silence closes with `Timeout`, and an authentic datagram that cannot be interpreted, with `Protocol_Error`. The reasons for rejection and closing are `Disconnect_Reason` (`host.odin`). @@ -181,13 +186,18 @@ others their share. receiver joins; the library's own reliable channel takes only whole messages. - **Window.** The chunks a reliable channel keeps in flight grow by what the path acknowledges (`channel_window`, each congestion interval), so they double each round trip while they hold the - channel back. Twice what the connection's rate sends in a round trip bounds them, so congestion + channel back. Twice what the connection's rate sends in a round trip, in chunks of the size the + channel's acknowledged ones had, bounds them, so congestion control decides what a stream takes, and the channel's `queue_bytes` sets the most it holds in flight and receives ahead (`channel_chunks`, a power of two up to `MAX_WINDOW`): a stream that fills a path holds twice what the path carries in a round trip. A loss does not shrink the window: its resend waits a round trip with the chunks after it acknowledged, ahead of new ones. - While objects wait for the update, the reliable game channels leave half of each datagram to them: a part that streams does not starve the state. +- **Receiving.** A reliable channel delivers what follows what it delivered only while the command + buffer has room besides what a datagram may still bring. A gap that closes over many small + messages leaves the rest held, and the network half reads no datagram until it is delivered: what + arrives is never acked without room to keep it. - The library channels count their queue the same way: `event_send` of a `Reliable` event and the authority requests return `Full` and send nothing if they do not fit in the queue of some connection they may go to, and the sender retries. The connection is not closed; diff --git a/net/channel.odin b/net/channel.odin index 819b235e1..b8041a1dc 100644 --- a/net/channel.odin +++ b/net/channel.odin @@ -101,7 +101,7 @@ Channel :: struct { // from fresh on were never sent, and resend holds the ones lost, in the order they were found. send_base,send_next,fresh:u16, window:int, - acked:int, // bytes of chunks acknowledged since channel_window last measured them + acked,acked_bytes:int, // chunks acknowledged since channel_window last measured them, and their bytes ready:int, // chunks in flight waiting to be sent or resent resend:[]u16, resend_head,resend_count:int, @@ -142,7 +142,7 @@ channel_release :: proc(h:^Host,c:^Channel) { channel_reset :: proc(c:^Channel) { c.queue.head,c.queue.used,c.applied=0,0,0 c.send_base,c.send_next,c.fresh,c.recv_base=0,0,0,0 - c.ready,c.front_left,c.front_open,c.assembly_used,c.spent,c.acked=0,0,false,0,0,0 + c.ready,c.front_left,c.front_open,c.assembly_used,c.spent,c.acked,c.acked_bytes=0,0,false,0,0,0,0 c.resend_head,c.resend_count=0,0 c.window=CHUNK_WINDOW for &chunk in c.chunks {chunk={}} @@ -218,17 +218,19 @@ channel_fill :: proc(c:^Channel) { } } -// The chunks in flight a reliable channel may keep. Each byte the path acknowledged lets one more be in +// The chunks in flight a reliable channel may keep. Each chunk the path acknowledged lets one more be in // flight, so the window doubles each round trip while it holds the channel back; a loss does not shrink // it, since a resend waits a round trip with the chunks after it acknowledged. What the path carries // bounds it through the connection's rate, which congestion control finds: never above twice what that -// rate sends in a round trip, nor below CHUNK_WINDOW, nor above what the channel holds. +// rate sends in a round trip, in chunks of the size the channel's acknowledged ones had, nor below +// CHUNK_WINDOW, nor above what the channel holds. @(private) channel_window :: proc "contextless" (c:^Channel,rtt,rate:f64) { - acked:=c.acked - c.acked=0 + acked,bytes:=c.acked,c.acked_bytes + c.acked,c.acked_bytes=0,0 if !(rtt>0) || !(rate>0) {return} - chunks:=min(f64(c.window+acked/CHUNK_BYTES),2*rate*rtt/CHUNK_BYTES,f64(len(c.chunks))) + wire:=acked>0?f64(bytes)/f64(acked)+4:CHUNK_BYTES+4 // bytes a chunk takes in a datagram + chunks:=min(f64(c.window)+f64(acked),2*rate*rtt/wire,f64(len(c.chunks))) c.window=max(int(chunks),min(CHUNK_WINDOW,len(c.chunks))) } @@ -330,7 +332,8 @@ channel_resolve :: proc(p:^Peer,record:Record,received:bool) { return } chunk.state=.Acked - c.acked+=int(chunk.length) + c.acked+=1 + c.acked_bytes+=int(chunk.length) for c.send_base!=c.send_next && c.chunks[int(c.send_base)%size].state==.Acked { last:=!c.chunks[int(c.send_base)%size].more c.chunks[int(c.send_base)%size]={} @@ -372,31 +375,59 @@ channels_read :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,late:=false) -> bool if c.recv_chunks[slot].present {continue} c.recv_chunks[slot]={length=u16(length),more=more,present=true} copy(c.recv_data[slot*CHUNK_BYTES:],scratch[:length]) - for c.recv_chunks[int(c.recv_base)%size].present { - head:=int(c.recv_base)%size - chunk:=c.recv_chunks[head] - if c.assembly_used+int(chunk.length)>min(len(c.assembly),c.max_message) { - // A message of the game goes on in pieces; one of the library is whole. - if index>=USER_CHANNELS || !channel_deliver(h,p,index,c.assembly[:c.assembly_used],true) {return false} - c.assembly_used=0 - if p.state!=.Connected {return true} - } - copy(c.assembly[c.assembly_used:],c.recv_data[head*CHUNK_BYTES:][:chunk.length]) - c.assembly_used+=int(chunk.length) - c.recv_chunks[head]={} - c.recv_base+=1 - if !chunk.more { - message:=c.assembly[:c.assembly_used] - c.assembly_used=0 - p.stats.messages_received+=1 - if !channel_deliver(h,p,index,message) {return false} - if p.state!=.Connected {return true} - } - } + if !channel_take(h,p,index) {return false} + if p.state!=.Connected {return true} } return !r.error } +// Delivers the chunks of a reliable channel that follow what it delivered, while the command buffer has +// room for them besides what a datagram may still bring: a gap that closes over many small messages +// leaves the rest held, and the network half reads no datagram until they are delivered (channels_held). +// False when the other end sent something that cannot be parsed. +@(private) +channel_take :: proc(h:^Host,p:^Peer,index:int) -> bool { + c:=&p.channels[index] + size:=len(c.recv_chunks) + for c.recv_chunks[int(c.recv_base)%size].present && command_room(h,2) { + head:=int(c.recv_base)%size + chunk:=c.recv_chunks[head] + if c.assembly_used+int(chunk.length)>min(len(c.assembly),c.max_message) { + // A message of the game goes on in pieces; one of the library is whole. + if index>=USER_CHANNELS || !channel_deliver(h,p,index,c.assembly[:c.assembly_used],true) {return false} + c.assembly_used=0 + if p.state!=.Connected {return true} + } + copy(c.assembly[c.assembly_used:],c.recv_data[head*CHUNK_BYTES:][:chunk.length]) + c.assembly_used+=int(chunk.length) + c.recv_chunks[head]={} + c.recv_base+=1 + if !chunk.more { + message:=c.assembly[:c.assembly_used] + c.assembly_used=0 + p.stats.messages_received+=1 + if !channel_deliver(h,p,index,message) {return false} + if p.state!=.Connected {return true} + } + } + h.back.held||=c.recv_chunks[int(c.recv_base)%size].present + return true +} + +// Delivers what the reliable channels of every connection hold, as far as there is room. +@(private) +channels_held :: proc(h:^Host) { + h.back.held=false + for &peer in h.back.peers { + if peer.state!=.Connected {continue} + for &c,index in peer.channels { + if !c.active || c.kind!=.Reliable || !c.recv_chunks[int(c.recv_base)%len(c.recv_chunks)].present {continue} + if !channel_take(h,&peer,index) {peer_drop(h,&peer,.Protocol_Error,true)} + if peer.state!=.Connected {break} + } + } +} + @(private) channel_deliver :: proc(h:^Host,p:^Peer,index:int,data:[]u8,more:=false) -> bool { if index>=USER_CHANNELS {return repl_message(h,p,index,data)} diff --git a/net/clock.odin b/net/clock.odin index faa98454b..1e1dbe0ce 100644 --- a/net/clock.odin +++ b/net/clock.odin @@ -166,6 +166,15 @@ clock_interpolation_tick :: proc(h:^Host,tick:u32) -> f64 { } // Start of the payload of a data datagram: the server's tick and input margin, or the client's inputs. +// The most bits time_write writes. +@(private) +time_max_bits :: proc(h:^Host) -> int { + protocol:=h.desc.protocol + if protocol==nil {return 0} + if h.desc.mode==.Server {return 32+1+1+32+1+8} + return 4+32+h.desc.replication.input_redundancy*protocol.input.max_bits +} + @(private) time_write :: proc(h:^Host,p:^Peer,w:^serial.Bit_Writer,seq:u16) -> (wrote:bool) { protocol:=h.desc.protocol diff --git a/net/flow_test.odin b/net/flow_test.odin index 43cf73ae4..e355866dd 100644 --- a/net/flow_test.odin +++ b/net/flow_test.odin @@ -108,13 +108,14 @@ hostile_acknowledgements_are_refused_and_never_push_a_window_past_its_bounds :: c.acked=0 channel_resolve(p,record,true) channel_resolve(p,record,true) - testing.expect_value(t,c.acked,CHUNK_BYTES) + testing.expect_value(t,c.acked,1) - Case :: struct {start,acked:int,rtt,rate:f64,low,high:int} + Case :: struct {start,acked,bytes:int,rtt,rate:f64,low,high:int} nan:=math.nan_f64() cases:=[]Case{ {start=64,acked=max(int)/2,rtt=1e9,rate=1e30,low=1024,high=1024}, - {start=64,acked=max(int)/2,rtt=0.05,rate=1<<20,low=1,high=409}, + {start=64,acked=1<<20,bytes=(1<<20)*CHUNK_BYTES,rtt=0.05,rate=1<<20,low=1,high=409}, + {start=64,acked=max(int)/2,bytes=max(int)/2,rtt=0.05,rate=1<<20,low=1,high=1024}, {start=512,acked=0,rtt=0.05,rate=16<<20,low=512,high=512}, {start=512,acked=1<<20,rtt=nan,rate=1<<20,low=512,high=512}, {start=512,acked=1<<20,rtt=0.05,rate=nan,low=512,high=512}, @@ -123,9 +124,9 @@ hostile_acknowledgements_are_refused_and_never_push_a_window_past_its_bounds :: {start=512,acked=1<<20,rtt=1e-9,rate=1,low=CHUNK_WINDOW,high=CHUNK_WINDOW}, } for k in cases { - c.window,c.acked=k.start,k.acked + c.window,c.acked,c.acked_bytes=k.start,k.acked,k.bytes channel_window(c,k.rtt,k.rate) - testing.expectf(t,c.window>=k.low && c.window<=k.high && c.acked==0,"%v gives a window of %d",k,c.window) + testing.expectf(t,c.window>=k.low && c.window<=k.high && c.acked==0 && c.acked_bytes==0,"%v gives a window of %d",k,c.window) } wr:serial.Bit_Writer @@ -163,13 +164,25 @@ acknowledgement_runs_read_back_and_refuse_what_does_not_add_up :: proc(t:^testin buffer:[PACKET_BYTES]u8 w:serial.Bit_Writer serial.bit_writer_init(&w,buffer[:]) - ack_bits_write(&w,&bits,count) + ack_bits_write(&w,&bits,count,max(int)) r:serial.Bit_Reader serial.bit_reader_init(&r,serial.bit_writer_flush(&w)) got,got_count,ok:=ack_bits_read(&r) same:=ok && int(got_count)==count for behind in 1..=ACK_WINDOW {same&&=ack_bit(&got,u64(behind))==(behind<=count && ack_bit(&bits,u64(behind)))} testing.expectf(t,same && !w.overflow,"%v over %d datagrams reads back",pattern,count) + // Within a budget it covers fewer datagrams, never more bits, and those read back. + for budget in ([]int{0,ACK_COUNT_BITS,ACK_COUNT_BITS+3,200,1000}) { + small:serial.Bit_Writer + small_buffer:[PACKET_BYTES]u8 + serial.bit_writer_init(&small,small_buffer[:]) + ack_bits_write(&small,&bits,count,budget) + serial.bit_reader_init(&r,serial.bit_writer_flush(&small)) + cut,cut_count,cut_ok:=ack_bits_read(&r) + fits:=cut_ok && int(cut_count)<=count && (small.bits<=budget || cut_count==0 && small.bits==ACK_COUNT_BITS) + for behind in 1..=int(cut_count) {fits&&=ack_bit(&cut,u64(behind))==ack_bit(&bits,u64(behind))} + testing.expectf(t,fits,"%v over %d datagrams within %d bits: %d bits for %d",pattern,count,budget,small.bits,cut_count) + } if pattern==.Sparse_Loss && count==ACK_WINDOW {testing.expectf(t,len(serial.bit_writer_flush(&w))0.6*BULK_LINK,"together %.0f B/s over a link of %.0f",total,BULK_LINK) testing.expectf(t,min(r.delivered[0],r.delivered[1])>0.3*total,"shares of %.0f and %.0f B/s",r.delivered[0],r.delivered[1]) } + +// Steps a server and its clients at the worker's cadence of 1 ms, streaming parts to each client on +// its first len(sent)/len(clients) channels, and adds the bytes each received; false if a connection +// closed. +@(private="file") +bulk_steps :: proc(t:^testing.T,w:^Test_World,server:^Host,clients:[]^Host,on_server:[]Connection,blob:[]u8,sent:[]int,seconds:f64,got:[]int) -> bool { + streams:=len(sent)/len(clients) + start:=w.now + for w.now-start=RECORD_WINDOW,"the outage fills what the connection keeps in flight") + } + } + testing.expectf(t,f64(got[0])/4>0.3*BULK_LINK,"after the outage %.0f B/s arrive over a link of %.0f",f64(got[0])/4,BULK_LINK) +} + +@(private="file") +Large_Input :: struct {bytes:[INPUT_MAX_BYTES-8]u8} + +// A client whose acknowledgement covers thousands of datagrams, half of them lost, while it repeats its +// largest inputs, still sends datagrams the server reads: the acknowledgement takes what is left. +@(test) +a_long_acknowledgement_never_overflows_a_datagram :: proc(t:^testing.T) { + schema:serial.Schema;serial.schema_init(&schema);defer serial.schema_destroy(&schema) + protocol:Protocol;protocol_init(&protocol);defer protocol_destroy(&protocol) + input,_:=serial.schema_add(&schema,Large_Input) + testing.expect(t,input_register(&protocol,&schema,input)) + protocol_seal(&protocol) + blob:=make([]u8,BULK_PART);defer delete(blob) + w:Test_World;test_world_init(&w,2,seed=43,packets=1<<16);defer test_world_destroy(&w) + replication:=Replication_Desc{input_redundancy=8} + server:=test_host(&w,{mode=.Server,protocol=&protocol,channels=BULK_TWO,replication=replication}) + client:=test_host(&w,{mode=.Client,protocol=&protocol,channels=BULK_TWO,replication=replication}) + on_server,_:=test_connect(&w,server,client) + if !testing.expect(t,on_server!=0) {return} + state:=u64(5) + tick:=u32(0) + sent:[2]int + got:[1]int + Phase :: struct {seconds:f64,forward,back:f32} + longest:=0 + for phase in ([]Phase{{3,0,0},{3,0.5,1},{3,0,0}}) { + sim_endpoint_conditions(&w.sim,0,{latency=0.06,bandwidth=BULK_LINK,loss=phase.forward}) + sim_endpoint_conditions(&w.sim,1,{latency=0.06,loss=phase.back}) + got[0]=0 + for step in 0..p.ack_known {longest=max(longest,int(p.recv_counter-p.ack_known))} + _=step + } + } + testing.expectf(t,longest>=3000,"the acknowledgement covers %d datagrams",longest) + testing.expectf(t,f64(got[0])/3>0.3*BULK_LINK,"after the lossy spell %.0f B/s arrive",f64(got[0])/3) +} + +// Drops the datagram of a connection numbered drop among those its host sends, once. +@(private="file") +Drop_Once :: struct {inner:Transport,count,drop:int} +@(private="file") +drop_once_transport :: proc(d:^Drop_Once) -> Transport { + return {d,proc(data:rawptr,to:Address,bytes:[]u8) -> bool { + d:=(^Drop_Once)(data) + if len(bytes)>0 && bytes[0]==u8(Packet_Type.Data) { + d.count+=1 + if d.count==d.drop {return true} + } + return d.inner.send(d.inner.data,to,bytes) + },proc(data:rawptr,buffer:[]u8) -> (int,Address,bool) { + d:=(^Drop_Once)(data) + return d.inner.receive(d.inner.data,buffer) + }} +} + +// Thousands of tiny reliable messages held behind one lost chunk all arrive, in order, when the gap +// closes: what the receiver cannot take in one poll waits for the next. +@(test) +tiny_messages_held_behind_a_gap_arrive_without_overflowing_the_receiver :: proc(t:^testing.T) { + COUNT :: 60000 + w:Test_World;test_world_init(&w,2,seed=47,packets=1<<16);defer test_world_destroy(&w) + drop:=Drop_Once{inner=sim_endpoint_transport(&w.sim,0),drop=120} + server:=new(Host) + if !testing.expect(t,host_init(server,{mode=.Server,transport=drop_once_transport(&drop),protocol_id=77,seed=3,channels=BULK_CHANNELS,frame_ops=1<<16})==.None) {free(server);return} + w.hosts[0]=server;w.count=1 + client:=test_host(&w,{mode=.Client,channels=BULK_CHANNELS}) + on_server,_:=test_connect(&w,server,client) + if !testing.expect(t,on_server!=0) {return} + sim_endpoint_conditions(&w.sim,0,{latency=0.05}) + sim_endpoint_conditions(&w.sim,1,{latency=0.05}) + queued,arrived:=0,0 + for step:=0;arrived Error { h.records=d.replication.records for i in 0.. bool { + b:=&h.back + out:=b.outbox + return len(out.bytes)-out.used>=b.reserve_bytes+extra*(PIECE_BYTES+8) && len(out.commands)-out.count>=b.reserve_count+extra +} + @(private) command_push :: proc(h:^Host,command:Command,data:[]u8=nil,words:=0) -> ^Command { out:=h.back.outbox diff --git a/net/peer.odin b/net/peer.odin index 2799129df..b1cc21d65 100644 --- a/net/peer.odin +++ b/net/peer.odin @@ -44,6 +44,7 @@ import "mirai:std/serial" MAX_CONNECTIONS :: 1024 @(private) SENT_WINDOW :: 4096 // datagrams in flight: twice what 100 Mbit/s holds over 120 ms, and more @(private) ACK_WINDOW :: SENT_WINDOW-1 // datagrams before the last received an acknowledgement covers at most +@(private) RECORD_WINDOW :: SENT_WINDOW-1-64 // datagrams in flight past which only one a keepalive goes @(private) ACK_COUNT_BITS :: 12 // of how many datagrams an acknowledgement covers: up to ACK_WINDOW @(private) LATE_WINDOW :: 32 // datagrams behind the last received that are still taken late @(private) HANDSHAKE_BYTES :: 256 @@ -170,9 +171,11 @@ ack_bits_advance :: proc "contextless" (bits:^Ack_Bits,shift:u64) { bits[(shift-1)/64]|=1<<((shift-1)%64) } // Which of the count datagrams before the acknowledged one arrived: whether the first did, then the -// lengths of the runs of arrived and missing ones in turn, each in Elias gamma code. +// lengths of the runs of arrived and missing ones in turn, each in Elias gamma code. Within budget bits: +// the runs that do not fit are left out, and the other end takes those datagrams as lost. @(private) -ack_bits_write :: proc(w:^serial.Bit_Writer,bits:^Ack_Bits,count:int) { +ack_bits_write :: proc(w:^serial.Bit_Writer,bits:^Ack_Bits,count,budget:int) { + count:=ack_fit(bits,count,budget) serial.bit_write(w,u64(count),ACK_COUNT_BITS) if count==0 {return} set:=ack_bit(bits,1) @@ -186,6 +189,19 @@ ack_bits_write :: proc(w:^serial.Bit_Writer,bits:^Ack_Bits,count:int) { at+=run } } +// The most of count datagrams whose runs fit in budget bits, at the end of a run. +@(private) +ack_fit :: proc "contextless" (bits:^Ack_Bits,count,budget:int) -> int { + used:=ACK_COUNT_BITS+1 + set:=ack_bit(bits,1) + for at:=0;atbudget {return at} + at+=run + } + return count +} // How many bits from bit at on, below count, equal set. @(private) ack_run :: proc "contextless" (bits:^Ack_Bits,at,count:int,set:bool) -> int { @@ -424,7 +440,8 @@ peer_service :: proc(h:^Host,p:^Peer) { // buffer that overflows, and from there grows by the same CC_GROWTH a second while it wants more and // finds no queue: connections that share a path converge to equal shares, and the queue stays short for // what else travels on it. A slow path grows by at most a quarter of its rate each interval. Losses -// without a queue do not cut the rate, so a link that drops datagrams at random keeps it. +// without a queue do not cut the rate, so a link that drops datagrams at random keeps it. An +// interval that delivered next to nothing, as after an outage, starts again. @(private) congestion_update :: proc(h:^Host,p:^Peer) { now:=h.back.now @@ -439,9 +456,11 @@ congestion_update :: proc(h:^Host,p:^Peer) { if queued || overrun { congested:=overrun || lost>0 && f64(lost)>CC_LOSS_SHARE*f64(acked+lost) p.rate=clamp(min(p.rate,delivered)*(congested?0.7:0.8),CC_MIN_RATE,CC_MAX_RATE) - p.starting=false + // Nothing delivered is an outage, not a path that narrowed: start again. + p.starting=delivered0 { p.rate=min(p.starting?p.rate*2:p.rate+min(CC_GROWTH*min(elapsed,1),p.rate/4),CC_MAX_RATE) } if now-p.window_time>CC_WINDOW { @@ -870,7 +889,7 @@ peer_wants_send :: proc(h:^Host,p:^Peer) -> bool { if b.send_pending || b.clock_due {return true} if idle>=h.desc.keepalive || (p.ack_pending && (idle>=ACK_DELAY || p.ack_chunks>=ACK_EVERY)) {return true} if idle>=1/f64(h.desc.tick_rate) && channels_pending(p) {return true} - if u16(p.send_seq-p.sent_tail)=PACKET_BYTES && channels_streaming(p) {return true} + if u16(p.send_seq-p.sent_tail)=PACKET_BYTES && channels_streaming(p) {return true} return p.sent_tail!=p.send_seq && idle>=clamp(p.rtt*2+p.rtt_variance*4,0.1,1) } @@ -886,7 +905,9 @@ peer_send :: proc(h:^Host,p:^Peer) { for &c in p.channels {c.spent=0} budget:=repl_budget(h,p) for packet:=0;;packet+=1 { - if u16(p.send_seq-p.sent_tail)>=SENT_WINDOW-1 {break} + // Past RECORD_WINDOW only a datagram a keepalive goes: if every one in flight was lost, the + // acknowledgement of a newer one settles them. + if u16(p.send_seq-p.sent_tail)>=RECORD_WINDOW && (packet>0 || b.now-p.last_send=SENT_WINDOW-1) {break} if packet>0 && p.tokens<=0 {break} first_record:=p.record_tail seq:=p.send_seq @@ -894,13 +915,16 @@ peer_send :: proc(h:^Host,p:^Peer) { data_begin(h,p,&w,.Data) serial.bit_write_bool(&w,p.recv_any) serial.bit_write(&w,u64(u16(p.recv_counter)),16) - ack_bits_write(&w,&p.recv_bits,p.recv_any && p.recv_counter>p.ack_known?int(min(p.recv_counter-p.ack_known,ACK_WINDOW)):0) + // What must go after it bounds the acknowledgement: its delay, the clock or the inputs, and the + // ends of the sections. + ack_budget:=serial.bit_writer_remaining(&w)-8-time_max_bits(h)-SECTION_RESERVE + ack_bits_write(&w,&p.recv_bits,p.recv_any && p.recv_counter>p.ack_known?int(min(p.recv_counter-p.ack_known,ACK_WINDOW)):0,ack_budget) serial.bit_write(&w,u64(clamp((b.now-p.recv_time)*1000,0,255)),8) wrote:=time_write(h,p,&w,seq) yield:=b.repl.key_cursor0?min(budget,PACKET_BYTES*4):0 if channels_write(h,p,&w,SECTION_RESERVE,yield) {wrote=true} more:=false - last:=u16(p.send_seq+1-p.sent_tail)>=SENT_WINDOW-1 || p.tokens<=PACKET_BYTES + last:=u16(p.send_seq+1-p.sent_tail)>=RECORD_WINDOW || p.tokens<=PACKET_BYTES if objects_write(h,p,&w,seq,&budget,&more,last) {wrote=true} if packet>0 && !wrote { p.record_tail=first_record diff --git a/plugins/multiplayer/runtime/journal.odin b/plugins/multiplayer/runtime/journal.odin index 84fc335d5..71515b95e 100644 --- a/plugins/multiplayer/runtime/journal.odin +++ b/plugins/multiplayer/runtime/journal.odin @@ -33,9 +33,9 @@ JOURNAL_BODY :: 168 JOURNAL_PART_BYTES :: 4<<20 // raw bytes of a part at most JOURNAL_CHANNEL :: 0 // of the host: the channel of parts JOURNAL_QUEUE :: 64<<10 // bytes of parts a connection has waiting for the channel before the journal hands it more -// The channels of a session's host. The window of parts holds 256 KiB in flight: what 2.5 MiB/s keep -// in flight over 100 ms. -JOURNAL_CHANNELS :: []wire.Channel_Desc{{kind=.Reliable,queue_bytes=JOURNAL_QUEUE}} +// The channels of a session's host. The channel of parts holds 256 KiB, which is also what it keeps in +// flight: what 2.5 MiB/s keep in flight over 100 ms. +JOURNAL_CHANNELS :: []wire.Channel_Desc{{kind=.Reliable,queue_bytes=256<<10}} JOURNAL_ASK_TICKS :: 30 // a client asks for parts again after this long without an answer, and for a base after a gap this long JOURNAL_AHEAD_TICKS :: 240 // server: an ask for a tick further ahead than this is let out at once (Journal_Held.ahead) JOURNAL_LOG :: 4096 // operations of a key a client holds ahead of where it stands diff --git a/plugins/multiplayer/runtime/prefab_test.odin b/plugins/multiplayer/runtime/prefab_test.odin index e6fad0950..260438064 100644 --- a/plugins/multiplayer/runtime/prefab_test.odin +++ b/plugins/multiplayer/runtime/prefab_test.odin @@ -89,6 +89,10 @@ a_replica_of_a_spawned_copy_is_filled_when_its_prefab_arrives :: proc(t:^testing apps[1].stats.frames+=1 scene.spawns_fill(client_ref,&apps[1],&warms[1],&problems[1]) core.settle(&info,client,nil) + // The ticks run without waiting, faster than any frame: each waits for the reads its jobs began. + for warm in warms[1].items { + if jobs.pending(&warm.job.counter) {jobs.wait(&apps[1].jobs,jobs.OWNER,&warm.job.counter)} + } if tick==10 { spawner:=scene.Spawner{core.commands_of(server_ref),&assets[0],&problems[0]} copy=scene.spawn(spawner,CRATE,{position={1,0,0}}) From 3ae24e8ad66eeab6cc52c7ecdeb5eee4768259d8 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 19:07:11 -0300 Subject: [PATCH 013/123] project: every registry lists the packages of its root from the store asset.root_packages resolves the packages.mirai of a project or plugin root against the machine store (or one the caller names) and words what it cannot list as problems naming the package. The application, cook/author, the world generator, mirai asset and the editor's import library go through it; the imported folder of the project layer is gone. Co-Authored-By: Claude Opus 5.5 --- .../skills/mirai-assets/references/formats.md | 10 +- .agents/skills/mirai-core/references/host.md | 6 +- .../mirai-projects/references/projects.md | 5 +- asset/packages_root.odin | 94 +++++++++++++++++++ asset/registry_test.odin | 20 +++- cook/author/command.odin | 19 +++- cook/build/build.odin | 3 +- cook/build/generate.odin | 4 +- cook/build/world.odin | 25 +++-- cook/import/identity.odin | 2 +- cook/import/sources/sync.odin | 2 +- .../game/animation_gallery_matching_test.odin | 5 +- demo/game/animation_motor_test.odin | 2 +- demo/game/mirai_matching_test.odin | 19 +++- editor/api_diagnostics.odin | 1 + editor/editor.odin | 6 +- editor/import.odin | 4 +- host/app.odin | 6 +- host/assets.odin | 26 ++++- host/assets_loading_test.odin | 75 +++++++++++++++ host/assets_watch.odin | 22 ++++- host/content.odin | 42 +++++++-- host/player.odin | 6 -- project/project.odin | 19 +--- tools/mirai/asset.odin | 32 ++++--- tools/mirai/session.odin | 7 +- tools/mirai/session_test.odin | 8 +- 27 files changed, 368 insertions(+), 102 deletions(-) create mode 100644 asset/packages_root.odin diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index 308f144f4..2a935803a 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -224,9 +224,13 @@ scan opens every file and writes it again. - **Listed outputs.** What the [imports](../../mirai-imports/references/imports.md#identity-and-reimport) build that nobody edits is not a file of the assets folder but a blob of the machine store. The root and each - mount take the entries `packages_resolve` gives for the packages their root pins - (`asset/packages.odin`), and a scan lists each like any asset (`Registry_File.listed`), with no - source, sidecar or imported folder anywhere; the registry reads no sidecar. An entry whose + mount take the entries `root_packages` gives for the packages their root pins: `packages_resolve` + of its `packages.mirai` against the store (`asset/packages_root.odin`). Every program that sets + up the registry of a root that is not shipped goes through it, with the store root as a value + (the machine's unless it names another); a root without the list opens no store, and what it + cannot list is a problem that names the package, reported where that program reports asset + problems, never a failure to start. A scan lists each like any asset (`Registry_File.listed`), + with no source, sidecar or imported folder anywhere; the registry reads no sidecar. An entry whose build is not on this machine, or whose blob is gone, stays listed as `Not_Built` (its id and kind, no file, not found by id), never removed and never an error of the scan; a blob that is not the asset its entry names is `Broken`. A path a file, a folder or another entry has takes diff --git a/.agents/skills/mirai-core/references/host.md b/.agents/skills/mirai-core/references/host.md index 640146bc7..299155845 100644 --- a/.agents/skills/mirai-core/references/host.md +++ b/.agents/skills/mirai-core/references/host.md @@ -39,7 +39,11 @@ without pacing (deterministic runs). settings only); `options`: folder of the player options, which the player provides (`app_user_folder`) and the editor does not; `assets_root`, `shaders_root`. `root`, `engine_root` and `options` are copied into the application. -- `asset_mounts`: plugin assets mounted under `plugins/`. +- `store_root`: the machine store the packages of `root` and of its plugins list from + (`project_assets`); empty is the machine's. The application mounts the assets of the active + plugins of `root` under `plugins/` with the outputs of their packages, logs and keeps + (`assets_package_problems`) what a package could not list, and opens no store for assets that + carry the index of a cook. `asset_mounts`: more folders mounted beside them. - `gpu_validation`. `App` stores the platform, `jobs`, `info`, `shared` (the `Shared` of the application), `world` (the diff --git a/.agents/skills/mirai-projects/references/projects.md b/.agents/skills/mirai-projects/references/projects.md index 1fec2528a..bdf450d87 100644 --- a/.agents/skills/mirai-projects/references/projects.md +++ b/.agents/skills/mirai-projects/references/projects.md @@ -10,6 +10,8 @@ Targets, jobs, fingerprint, project shaders, cooking and shipped game (`cook/bui project.mirai manifest: what makes the folder a project settings.mirai project settings, the bindings in Input_Settings: what differs (optional) cook.mirai cooking roots that no scene names (optional) + packages.mirai the packages of the machine store the project uses, by the hash of their + manifest, and the folder each is listed in (optional; a plugin may have one) assets/ assets, by theme: what is authored, the art sources beside what they make, each with its sidecar (.import: settings, folder and asset ids), and the documents their imports write @@ -20,9 +22,6 @@ Targets, jobs, fingerprint, project shaders, cooking and shipped game (`cook/bui panels/ editor panels built by the chat agent (engine-time) tests/scenarios/ gameplay tests the editor plays (`mirai play test`), optional .mirai/ generated; not edited or versioned - imported/ what imports build from the sources that nobody edits (meshes, textures, sounds, - fonts), each file named by its id and what it was built from; a build makes - it again gen/ generated packages: the worlds and the root of each target plugins/ libraries of the engine plugins, with their stages and declarations editor library of mirai-editor and of the players it opens; play, that of diff --git a/asset/packages_root.odin b/asset/packages_root.odin new file mode 100644 index 000000000..540f1952f --- /dev/null +++ b/asset/packages_root.odin @@ -0,0 +1,94 @@ +package asset + +import "core:fmt" +import "core:strings" +import "mirai:std/base" +import "mirai:store" + +// What a root lists from the packages it uses: a project, or a plugin, which +// keeps its packages as a project does. Every program that sets a registry up +// for a root that is not shipped gives it these entries. +Root_Packages :: struct { + listed:[]Package_Listed, // for registry_init or registry_mount of the root + problems:[]string, // one line per package it cannot list whole: which one, and how to fix it +} + +// packages_resolve of the list at root (PACKAGES_FILE) against the store at +// store_root, the machine's (store.default_root) when "". A root without the +// list lists nothing and opens no store. A list or a store that does not read +// is a problem, as is a package that is not here or not built here: none is an +// error, which is only that memory ran out. In allocator, an arena. +root_packages :: proc(root,store_root:string,allocator:=context.allocator) -> (resolved:Root_Packages,error:Error) { + problems:=make([dynamic]string,allocator) + list,found,load_error:=packages_load(root,context.temp_allocator) + switch load_error { + case .None: + case .Out_Of_Memory: return {},.Out_Of_Memory + case .Truncated,.Not_An_Asset,.Version,.Corrupt,.Hash,.Too_Large,.Schema,.Not_Found,.IO,.Invalid,.Codec_Missing: + problem_line(&problems,"%s/%s does not read (%v), so its packages are not listed: restore it from version control",root,PACKAGES_FILE,load_error) or_return + return {problems=problems[:]},.None + } + if !found || len(list.packages)==0 {return {problems=problems[:]},.None} + folder:=store_root!=""?store_root:store.default_root(context.temp_allocator) + if folder=="" { + problem_line(&problems,"this machine has no folder for the store of Mirai, so the packages of %s are not listed",root) or_return + return {problems=problems[:]},.None + } + s,open_error:=store.open(folder,context.temp_allocator) + switch open_error { + case .None: + case .Out_Of_Memory: return {},.Out_Of_Memory + case .Missing,.Not_Here_Yet,.Corrupt,.Invalid_Argument,.Io,.Unsupported_Layout: + problem_line(&problems,"the store at %s does not open (%v), so the packages of %s are not listed",folder,open_error,root) or_return + return {problems=problems[:]},.None + } + defer store.close(&s) + packages:=packages_resolve(&s,&list,allocator) or_return + unreadable:=make(map[base.Asset_Id]bool,context.temp_allocator) + for problem in packages.problems { + named:=package_named(&s,&list,problem.package_id) + switch problem.code { + case .Manifest_Missing: + problem_line(&problems,"%s is not in the store at %s: ingest its source on this machine, or fetch it from a peer that has it",named,folder) or_return + case .Manifest_Unreadable: + problem_line(&problems,"the manifest of %s in the store at %s is not one this engine reads: ingest its source again",named,folder) or_return + case .Build_Unreadable: + unreadable[problem.package_id]=true + problem_line(&problems,"the build of %s on this machine does not read, so its assets are not built: refresh it on a machine with its source",named) or_return + case .Asset_Unknown: + buffer:[base.ASSET_ID_TEXT]u8 + problem_line(&problems,"%s has no asset %s, which %s/%s uses: remove it from the list, or ingest the package again",named,base.asset_id_format(problem.asset,buffer[:]),root,PACKAGES_FILE) or_return + } + } + for &l in packages.listed { + if l.state!=.Not_Built || unreadable[l.package_id] {continue} + unreadable[l.package_id]=true + problem_line(&problems,"%s has no build on this machine that this engine reads, so its assets do not load: refresh it on a machine with its source, or fetch it from a peer that built it",package_named(&s,&list,l.package_id)) or_return + } + return {listed=packages.listed[:],problems=problems[:]},.None +} + +// "the package () of /": by its manifest when the store +// has it, else by its id. In the temp allocator. +@(private="file") +package_named :: proc(s:^store.Store,list:^Packages,id:base.Asset_Id) -> string { + buffer:[base.ASSET_ID_TEXT]u8 + text:=base.asset_id_format(id,buffer[:]) + for &entry in list.packages { + if entry.id!=id {continue} + folder:=entry.folder!=""?entry.folder:"the top of the assets" + if p,error:=package_get(s,entry.manifest,context.temp_allocator);error==.None && p.id==id { + return fmt.tprintf("the package %s (%s) listed in %s",p.name,text,folder) + } + return fmt.tprintf("the package %s listed in %s",text,folder) + } + return fmt.tprintf("the package %s",text) +} + +@(private="file") +problem_line :: proc(problems:^[dynamic]string,format:string,args:..any) -> Error { + line,no_line:=strings.clone(fmt.tprintf(format,..args),problems.allocator) + if no_line!=nil {return .Out_Of_Memory} + if _,no_room:=append(problems,line);no_room!=nil {return .Out_Of_Memory} + return .None +} diff --git a/asset/registry_test.odin b/asset/registry_test.odin index b9d08512b..ea77e5eca 100644 --- a/asset/registry_test.odin +++ b/asset/registry_test.odin @@ -2,6 +2,7 @@ package asset import "core:fmt" import "core:os" +import "core:strings" import "core:testing" import "core:time" import "mirai:std/base" @@ -559,9 +560,9 @@ registry_mounts_a_guest_folder_that_never_takes_an_id_the_registry_has :: proc(t } // What the packages a project and a plugin pin list of the cache is listed -// from the store, with no source, sidecar or imported folder anywhere: a built -// output is its blob and loads, one whose build is not here is listed -// Not_Built, a path a file of the folder has takes the id of the output after +// from the store through root_packages of each root, with no source, sidecar +// or imported folder anywhere: a built output is its blob and loads, one whose +// build is not here is listed Not_Built and named as a problem, a path a file of the folder has takes the id of the output after // its name, a refresh keeps every listed output and memory never holds one. @(test) registry_lists_what_a_project_pins_without_its_sources :: proc(t:^testing.T) { @@ -607,9 +608,18 @@ registry_lists_what_a_project_pins_without_its_sources :: proc(t:^testing.T) { append(&pack.assets,Package_Asset{kind=KIND_TEXTURE,name="atlas",id=atlas}) pack_list:=pin(t,&s,&pack,{blob(t,&types,&s,staging,atlas)},"") os.remove_all(staging) - resolved,resolve_error:=packages_resolve(&s,&crate_list,context.temp_allocator) - plugin_resolved,plugin_error:=packages_resolve(&s,&pack_list,context.temp_allocator) + // Each root lists from its own list and the store it is given; one without a list lists nothing. + game,pack_root:=fmt.tprintf("%s/game",folder),fmt.tprintf("%s/pack",folder) + testing.expect_value(t,packages_save(game,&crate_list),Error.None) + testing.expect_value(t,packages_save(pack_root,&pack_list),Error.None) + resolved,resolve_error:=root_packages(game,s.root,context.temp_allocator) + plugin_resolved,plugin_error:=root_packages(pack_root,s.root,context.temp_allocator) if !testing.expect(t,resolve_error==.None && plugin_error==.None && len(resolved.listed)==3 && len(plugin_resolved.listed)==1) {return} + // The output whose build is not here is a problem that names its package; the others load. + testing.expectf(t,len(resolved.problems)==1 && strings.contains(resolved.problems[0],"package crate") && strings.contains(resolved.problems[0],"no build"),"%v",resolved.problems) + testing.expect_value(t,len(plugin_resolved.problems),0) + none,none_error:=root_packages(staging,fmt.tprintf("%s/unopened",folder),context.temp_allocator) + testing.expect(t,none_error==.None && len(none.listed)==0 && len(none.problems)==0 && !os.exists(fmt.tprintf("%s/unopened",folder)),"a root without a list opens no store") // A file of the project at the path of a listed output keeps it. write_texture(t,&types,fmt.tprintf("%s/props/crate/lid.tex",root),own) write_material(t,&types,fmt.tprintf("%s/props/crate/Wood.mat",root),{9,9},wood) diff --git a/cook/author/command.odin b/cook/author/command.odin index 3f077384f..a05b6eb65 100644 --- a/cook/author/command.odin +++ b/cook/author/command.odin @@ -91,22 +91,28 @@ Project :: struct { untyped:string, // why it has none, with what gives them recipes:[]Recipe, // of the engine and of its active plugins scratch:Scratch_World, // where a recipe is run; made the first time + package_problems:[]string, // what the packages of the project and its plugins could not list (host.project_assets) } // What builds the library the types of a game come from. BUILD_FIX :: "mirai build editor builds it" -// Opens the project at root with the assets of its plugins. library false -// leaves the types of its game out. problem: why it does not open. -project_open :: proc(p:^Project,root,engine:string,library:=true) -> (problem:string) { +// Opens the project at root with the assets of its plugins and of the +// packages of both, from the store at store_root ("": the machine's). library +// false leaves the types of its game out. problem: why it does not open. +project_open :: proc(p:^Project,root,engine:string,library:=true,store_root:="") -> (problem:string) { opened,error:=project.open(root,engine,context.temp_allocator) if error.code!=.None {return project.error_text(root,error,context.temp_allocator)} p.root=strings.clone(opened.root) p.name=strings.clone(base.text_get(opened.manifest.name[:])) p.assets_root=project.folder_path(p.root,.Assets) - asset.registry_init(&p.registry,p.assets_root) - for mount in host.project_asset_mounts(&opened,context.temp_allocator) {asset.registry_mount(&p.registry,mount.folder,mount.prefix,mount.listed)} + found,listed:=host.project_assets(&opened,store_root,context.temp_allocator) + if !listed {return "out of memory to list its assets"} + asset.registry_init(&p.registry,p.assets_root,listed=found.listed) + for mount in found.mounts {asset.registry_mount(&p.registry,mount.folder,mount.prefix,mount.listed)} + p.package_problems=make([]string,len(found.problems)) + for problem,i in found.problems {p.package_problems[i]=strings.clone(problem)} asset.types_init(&p.types) p.types.codecs=asset.project_codecs(root,engine) p.assets={&p.registry,&p.types} @@ -130,6 +136,8 @@ project_close :: proc(p:^Project) { delete(p.assets_root) delete(p.source) delete(p.untyped) + for problem in p.package_problems {delete(problem)} + delete(p.package_problems) p^={} } @@ -369,6 +377,7 @@ command :: proc(args:[]string,output,errors:io.Writer) -> int { return 1 } defer project_close(&p) + for problem in p.package_problems {fmt.wprintfln(errors,"mirai %s %s: %s",rest[0],verb,problem)} if rest[0]=="texts" {return texts_command(&p,rest[1:],output,errors)} return project_run(&p,kind,verb,rest[2:],output,errors) } diff --git a/cook/build/build.odin b/cook/build/build.odin index a794c5a22..f34ba45d5 100644 --- a/cook/build/build.odin +++ b/cook/build/build.odin @@ -77,6 +77,7 @@ Desc :: struct { action:Action, // Check and Test: of the Editor target test_package:string, // Test: the folder of the package, absolute test:Test_Options, // Test: how its executable runs + store_root:string, // the machine store the packages of the project list from, for the paths of assets its code writes; "": the machine's allocator:runtime.Allocator, // of the job. Unset: context.allocator } @@ -225,7 +226,7 @@ job_start :: proc(desc:Desc) -> ^Job { for root,i in roots {job.imports[i]=strings.clone(root,a)} job.import_program=project_engine(p,desc.engine,a).import_program } - generated,written:=generated_write(p,desc.target) + generated,written:=generated_write(p,desc.target,desc.store_root) if !written { if len(generated.errors)>0 { job.result.text=strings.join(generated.errors,"\n",a) diff --git a/cook/build/generate.odin b/cook/build/generate.odin index ca5ab1ed5..89f625636 100644 --- a/cook/build/generate.odin +++ b/cook/build/generate.odin @@ -41,8 +41,8 @@ generate :: proc(p:^project.Project,target:Target,allocator:=context.allocator) } // Writes the generated packages. The errors of out are those of the attributes of the project's packages. -generated_write :: proc(p:^project.Project,target:Target) -> (out:Generated,ok:bool) { - return world_write_files(p,target==.Game) +generated_write :: proc(p:^project.Project,target:Target,store_root:="") -> (out:Generated,ok:bool) { + return world_write_files(p,target==.Game,store_root) } // The flags every compilation of a project's generated code needs. diff --git a/cook/build/world.odin b/cook/build/world.odin index e1da3d55d..325e98b22 100644 --- a/cook/build/world.odin +++ b/cook/build/world.odin @@ -303,8 +303,10 @@ generator_read :: proc(p:^project.Project) -> (g:Generator) { } // The source of the world packages and of the main packages. The world of -// the game that ships is generated only when shipped is asked for. -world_generate :: proc(p:^project.Project,allocator:=context.allocator,shipped:=false) -> (out:Generated) { +// the game that ships is generated only when shipped is asked for. The paths +// of assets the code writes are checked against the packages of the project +// and its plugins in the store at store_root ("": the machine's). +world_generate :: proc(p:^project.Project,allocator:=context.allocator,shipped:=false,store_root:="") -> (out:Generated) { context.allocator=allocator g:=generator_read(p) for pkg in g.packages { @@ -317,7 +319,7 @@ world_generate :: proc(p:^project.Project,allocator:=context.allocator,shipped:= for pkg in g.packages {draws_read(&g,pkg)} rules_check(&g) defaults_check(&g) - asset_paths_check(&g,p) + asset_paths_check(&g,p,store_root) out.errors=g.errors[:] if len(g.errors)>0 {return} out.game_world=world_write(&g,false) @@ -382,10 +384,10 @@ PROJECT_INFO_SYMBOL :: "mirai_project_info" // Writes the generated packages under /.mirai/gen when they changed. // out has what was generated, in the temp allocator. -world_write_files :: proc(p:^project.Project,shipped:=false) -> (out:Generated,ok:bool) { +world_write_files :: proc(p:^project.Project,shipped:=false,store_root:="") -> (out:Generated,ok:bool) { if out.errors=kinds_write(p);len(out.errors)>0 {return} if out.errors=records_write(p);len(out.errors)>0 {return} - out=world_generate(p,context.temp_allocator,shipped) + out=world_generate(p,context.temp_allocator,shipped,store_root) if len(out.errors)>0 {return} gen:=project.path(project.folder_path(p.root,.Generated,context.temp_allocator),"gen",allocator=context.temp_allocator) files:=[4][2]string{{"editor_world/world.odin",out.editor_world},{"shipped_world/world.odin",out.shipped_world},{"editor/main.odin",out.editor_main},{"game/main.odin",out.game_main}} @@ -3364,17 +3366,20 @@ defaults_check :: proc(g:^Generator) { // The paths of assets the code of the project writes: a string literal that // ends in the extension of an asset ("pets/Dog.scene") names an asset of the -// project or of its plugins, or the build fails where it is written, with the -// path that has the same name when there is one. +// project, of its plugins or of their packages, or the build fails where it +// is written, with the path that has the same name when there is one. @(private="file") -asset_paths_check :: proc(g:^Generator,p:^project.Project) { +asset_paths_check :: proc(g:^Generator,p:^project.Project,store_root:string) { assets:=project.folder_path(p.root,.Assets,context.temp_allocator) if !os.is_dir(assets) {return} registry:asset.Registry - asset.registry_init(®istry,assets,"",context.temp_allocator) + packages,_:=asset.root_packages(p.root,store_root,context.temp_allocator) + asset.registry_init(®istry,assets,"",context.temp_allocator,packages.listed) defer asset.registry_destroy(®istry) for folder in project.asset_folders(p,context.temp_allocator) { - if folder.prefix!="" {asset.registry_mount(®istry,folder.path,folder.prefix)} + if folder.prefix=="" {continue} + plugin,_:=asset.root_packages(folder.root,store_root,context.temp_allocator) + asset.registry_mount(®istry,folder.path,folder.prefix,plugin.listed) } if asset.registry_scan(®istry)!=.None {return} extensions:=make(map[string]bool,context.temp_allocator) diff --git a/cook/import/identity.odin b/cook/import/identity.odin index 31c301434..a72cd221f 100644 --- a/cook/import/identity.odin +++ b/cook/import/identity.odin @@ -58,7 +58,7 @@ writer_source :: proc(w:^Writer,path,folder:string) -> bool { w.taken=make([]bool,len(w.sidecar.outputs),context.temp_allocator) w.outputs=make([dynamic]Output,context.temp_allocator) w.paths=make([dynamic]string,context.temp_allocator) - w.imported=project.imported_path(w.options.root,context.temp_allocator) + w.imported=project.path(project.folder_path(w.options.root,.Generated,context.temp_allocator),"imported",allocator=context.temp_allocator) w.registry=w.options.registry if w.registry==nil { w.registry=new(asset.Registry,context.temp_allocator) diff --git a/cook/import/sources/sync.odin b/cook/import/sources/sync.odin index 4d94a7875..180fa0297 100644 --- a/cook/import/sources/sync.odin +++ b/cook/import/sources/sync.odin @@ -92,7 +92,7 @@ sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) - context.temp_allocator=base.dynamic_arena_allocator(&scratch) top:=project.path_absolute(root,context.temp_allocator) - imported:=project.imported_path(top,context.temp_allocator) + imported:=project.path(project.folder_path(top,.Generated,context.temp_allocator),"imported",allocator=context.temp_allocator) assets:=project.folder_path(top,.Assets,context.temp_allocator) registry:asset.Registry asset.registry_init(®istry,assets,"",context.temp_allocator) diff --git a/demo/game/animation_gallery_matching_test.odin b/demo/game/animation_gallery_matching_test.odin index 30ef35d53..736c9f658 100644 --- a/demo/game/animation_gallery_matching_test.odin +++ b/demo/game/animation_gallery_matching_test.odin @@ -1,7 +1,5 @@ package game -import "core:fmt" -import "core:os" import "core:testing" import "mirai:asset" import "mirai:core" @@ -22,8 +20,7 @@ gallery_mirai_matching_plays_idle_walk_and_jog_by_the_speed_it_is_asked :: proc( progress:core.Table(anim.Animation_Progress);defer core.table_destroy(&progress) panel:Gallery_UI core.entities_init(&f.entities) - root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) - if !testing.expect(t,found && host.assets_init(&f.assets,{root=fmt.tprintf("%s/demo/assets",root),imported=fmt.tprintf("%s/demo/.mirai/imported",root),capacity=512})) {return} + if !demo_assets_init(t,&f.assets) {return} f.app={assets=&f.assets,allocator=context.allocator} ids,ok:=mirai_assets(t,&f.assets,"characters/mirai/mirai_gallery.character") if !ok {return} diff --git a/demo/game/animation_motor_test.odin b/demo/game/animation_motor_test.odin index edfc3a639..92d8e75f0 100644 --- a/demo/game/animation_motor_test.odin +++ b/demo/game/animation_motor_test.odin @@ -30,7 +30,7 @@ animation_motor_preserves_world_root_direction_facing_and_physical_speed_limits_ } // The body of Mirai: the first mesh of the import of characters/mirai/mirai.glb, by the id its -// sidecar keeps. The sync of a build writes it to the imported folder. +// package keeps. The demo lists it from its packages in the machine store. @(private) MIRAI_BODY :: "f0c947702bb8141dc1d27444a341d52c" diff --git a/demo/game/mirai_matching_test.odin b/demo/game/mirai_matching_test.odin index b283cf6fa..bad40247c 100644 --- a/demo/game/mirai_matching_test.odin +++ b/demo/game/mirai_matching_test.odin @@ -8,6 +8,7 @@ import "core:testing" import "mirai:core" import render "mirai:engine/render" import "mirai:host" +import "mirai:project" import "mirai:std/base" import mm "mirai:std/math" import anim "plugins:animation/runtime" @@ -21,8 +22,7 @@ mirai_matching_selects_a_walking_clip :: proc(t:^testing.T) { graphs:anim.Character_Assets;defer anim.character_assets_free(&graphs) progress:core.Table(anim.Animation_Progress);defer core.table_destroy(&progress) core.entities_init(&f.entities) - root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) - if !testing.expect(t,found && host.assets_init(&f.assets,{root=fmt.tprintf("%s/demo/assets",root),imported=fmt.tprintf("%s/demo/.mirai/imported",root),capacity=512})) {return} + if !demo_assets_init(t,&f.assets) {return} f.app={assets=&f.assets,allocator=context.allocator} ids,ok:=mirai_assets(t,&f.assets,"characters/mirai/mirai_matching.character") if !ok {return} @@ -93,8 +93,7 @@ mirai_starting_to_its_side_turns_no_bone_faster_than_its_walk :: proc(t:^testing constraints:anim.Animation_Constraints;defer anim.constraints_free(&constraints) grounding:anim.Grounding_Runtime;defer anim.grounding_free(&grounding) core.entities_init(&f.entities) - root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) - if !testing.expect(t,found && host.assets_init(&f.assets,{root=fmt.tprintf("%s/demo/assets",root),imported=fmt.tprintf("%s/demo/.mirai/imported",root),capacity=512})) {return} + if !demo_assets_init(t,&f.assets) {return} f.app={assets=&f.assets,allocator=context.allocator} ids,ok:=mirai_assets(t,&f.assets,"characters/mirai/mirai_matching.character") if !ok {return} @@ -146,3 +145,15 @@ mirai_starting_to_its_side_turns_no_bone_faster_than_its_walk :: proc(t:^testing testing.expectf(t,start<=2*walk,"starting to its side no bone turns faster than twice its walk: %.2f against %.2f rad/s",start,walk) testing.expect_value(t,problems.count,0) } + +// The assets of the demo as its player sets them up: its own, those of its +// plugins and the outputs of the packages of both in the machine store. +demo_assets_init :: proc(t:^testing.T,a:^host.Assets) -> bool { + root,found:=os.lookup_env("MIRAI_ROOT",context.temp_allocator) + if !testing.expect(t,found,"MIRAI_ROOT names the engine") {return false} + demo:=fmt.tprintf("%s/demo",root) + p,error:=project.open(demo,root,context.temp_allocator) + if !testing.expect_value(t,error.code,project.Error_Code.None) {return false} + assets,listed:=host.project_assets(&p,"",context.temp_allocator) + return testing.expect(t,listed && host.assets_init(a,{root=fmt.tprintf("%s/assets",demo),listed=assets.listed,mounts=assets.mounts,package_problems=assets.problems,capacity=512})) +} diff --git a/editor/api_diagnostics.odin b/editor/api_diagnostics.odin index 87a98e653..63e0cd0c8 100644 --- a/editor/api_diagnostics.odin +++ b/editor/api_diagnostics.odin @@ -128,6 +128,7 @@ api_diagnostics :: proc(s:^Session,call:^api.Call) { append(&errors,entry) } api_duplicates_list(&app.assets.registry,&errors) + for problem in host.assets_package_problems(app.assets) {append(&errors,Api_Engine_Diagnostics_Result_Asset_Errors{kind="Package",problem="Not_Listed",message=problem})} } r.asset_errors=errors[:] r.build.phase="Idle" diff --git a/editor/editor.odin b/editor/editor.odin index 109d2489f..ba22f6c88 100644 --- a/editor/editor.odin +++ b/editor/editor.odin @@ -408,14 +408,12 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { maximized=!options.headless && options.width==0 && options.height==0, } if options.launcher && options.width==0 && options.height==0 {window.size,window.fixed_size,window.maximized=PROJECTS_WINDOW,true,false} - mounts:=make([dynamic]asset.Registry_Mount,context.temp_allocator) has_project:=false if p,error:=project.open(root,engine,context.temp_allocator);error.code==.None { has_project=true - append(&mounts,..host.project_asset_mounts(&p,context.temp_allocator)) if title:=project.window_title(&p.manifest);title!="" {window.title=strings.clone(title,context.temp_allocator)} } - append(&mounts,asset.Registry_Mount{folder=project.path(engine,EDITOR_ASSETS,allocator=context.temp_allocator),prefix=EDITOR_ASSETS_PREFIX}) + mounts:=[1]asset.Registry_Mount{{folder=project.path(engine,EDITOR_ASSETS,allocator=context.temp_allocator),prefix=EDITOR_ASSETS_PREFIX}} windowed:=!options.headless app,app_error:=host.app_create({ info=info,root=root,engine_root=engine,threads=jobs.THREADS_AUTO, @@ -423,7 +421,7 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { window_desc=window,vsync=windowed,fixed_dt=windowed?0:HEADLESS_DT, // Without a window the mixer plays to nowhere, in real time: what play.record mixes is there. audio=windowed || !options.no_gpu?.Device:.None,audio_driver=windowed?"":"dummy", - assets_root=os.is_dir(assets)?assets:"",assets_imported=project.imported_path(root,context.temp_allocator),assets_cache=project.cache_path(root,project.REGISTRY_CACHE,allocator=context.temp_allocator), + assets_root=os.is_dir(assets)?assets:"",assets_cache=project.cache_path(root,project.REGISTRY_CACHE,allocator=context.temp_allocator), animation_cache=project.cache_path(root,project.ANIMATION_CACHE,allocator=context.temp_allocator), collision_cache=project.content_cache_path(root,project.COLLISION_CACHE,allocator=context.temp_allocator), asset_mounts=mounts[:],shaders_root=root, diff --git a/editor/import.odin b/editor/import.odin index 63c8123e6..ab6c108c4 100644 --- a/editor/import.odin +++ b/editor/import.odin @@ -581,7 +581,9 @@ import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_I library,error:=host.project_library_load(path,0) if error!=.None {return {problem=host.library_error_text(error,path)}} held.library=library - asset.registry_init(&held.registry,project.folder_path(root,.Assets,context.temp_allocator),"",allocator) + // The type of sources.Library names no store: the packages are the machine's. + packages,_:=asset.root_packages(root,"",context.temp_allocator) + asset.registry_init(&held.registry,project.folder_path(root,.Assets,context.temp_allocator),"",allocator,packages.listed) asset.registry_scan(&held.registry) found:[PROVIDER_CAPACITY]Import_Format_Provider held.providers=slice.clone(found[:provided_in(&held.library.info,found[:])],allocator) diff --git a/host/app.odin b/host/app.odin index aad1d6114..8998e7768 100644 --- a/host/app.odin +++ b/host/app.odin @@ -60,12 +60,12 @@ App_Desc :: struct { engine_root:string, // the engine whose general settings lie under those of the project; empty: only the project's, as in a shipped game options:string, // folder of the options the player chose (settings of the machine and input bindings) over those of the project; "": none, as in the editor assets_root:string, // folder of cooked assets; its changes are reloaded - assets_imported:string, // the imported folder that goes with assets_root, of a project that is not shipped; "": none + store_root:string, // the machine store the packages of the project root and its plugins list from (project_assets); "": the machine's. A shipped game opens none assets_cache:string, // file where the registry of assets_root keeps what it saw; "": none animation_cache:string, // folder where the content keeps the .anim it compressed; "": none collision_cache:string, // folder where the convex pieces of meshes are kept (asset.mesh_collision_pieces); "": none shaders_root:string, // folder whose shader generation is loaded; "": the one that holds assets_root - asset_mounts:[]asset.Registry_Mount, + asset_mounts:[]asset.Registry_Mount, // besides those of the plugins of the project root, which the application mounts } // What the player was asked for at launch: libraries read it (the first @@ -315,7 +315,7 @@ app_create :: proc(desc:App_Desc) -> (app:^App,error:App_Error) { } } if desc.assets_root!="" { - if error=app_assets_create(app,desc.assets_root,desc.assets_cache,desc.asset_mounts,desc.assets_imported);error!=.None { + if error=app_assets_create(app,desc);error!=.None { app_destroy(app) return nil,error } diff --git a/host/assets.odin b/host/assets.odin index 29b1abbcd..f44769149 100644 --- a/host/assets.odin +++ b/host/assets.odin @@ -40,7 +40,8 @@ ASSETS_MESH_NONE :: render.Mesh_Handle(max(u32)) Assets_Desc :: struct { root:string, // folder of the asset registry - imported:string, // the imported folder that goes with root (asset.Registry); "" for none + listed:[]asset.Package_Listed, // the outputs of the packages of the project root is the assets of (project_assets) + package_problems:[]string, // what those packages and the ones of the mounts could not list: kept for whoever reports asset problems cache:string, // file where the registry keeps what it scanned; "" for none renderer:^render.Renderer, // nil: nothing is uploaded allocator:mem.Allocator, // context.allocator when unset @@ -221,6 +222,7 @@ Assets :: struct { missing_texture:Assets_Texture, missing_texture_ready:bool, errors:[dynamic]Assets_Error, + package_problems:[]string, // Assets_Desc.package_problems, owned revision:u64, // advances when something that was given out changes: who holds handles or rows asks again structure:u64, // advances when the instances of a scene are no longer what they need: who extracted it does it again structure_changes:[ASSETS_STRUCTURE_LOG]Assets_Structure_Change, @@ -272,7 +274,7 @@ assets_init :: proc(a:^Assets,desc:Assets_Desc) -> bool { allocator:=desc.allocator.procedure!=nil?desc.allocator:context.allocator capacity:=desc.capacity>0?desc.capacity:ASSETS_DEFAULT_CAPACITY a^={allocator=allocator,renderer=desc.renderer} - listed:=asset.registry_init(&a.registry,desc.root,desc.cache,allocator) + listed:=asset.registry_init(&a.registry,desc.root,desc.cache,allocator,desc.listed) // A copy of an asset made outside the editor gets an id of its own. a.registry.resolve_duplicates=true for mount in desc.mounts {listed&&=asset.registry_mount(&a.registry,mount.folder,mount.prefix,mount.listed)} @@ -300,6 +302,7 @@ assets_init :: proc(a:^Assets,desc:Assets_Desc) -> bool { a.free_scenes,errors[5]=make([dynamic]u32,0,start,allocator) a.errors,errors[6]=make([dynamic]Assets_Error,0,desc.error_capacity>0?desc.error_capacity:ASSETS_DEFAULT_ERRORS,allocator) for error in errors {ok&&=error==nil} + if ok {a.package_problems,ok=assets_texts_clone(desc.package_problems,allocator)} if desc.renderer!=nil {a.texture_assets=make([dynamic]base.Asset_Id,allocator)} ok=ok && cap(a.meshes)==capacity && cap(a.textures)==capacity && cap(a.materials)==capacity && cap(a.scenes)==capacity ok=ok && cap(a.material_descs)==capacity*2 && cap(a.row_marks)==capacity*2 && cap(a.row_hashes)==capacity*2 && cap(a.free_rows)==start && cap(a.free_meshes)==start && cap(a.rows_dirty)==ASSETS_ROWS_DIRTY @@ -330,9 +333,26 @@ assets_destroy :: proc(a:^Assets) { for &scope in a.scopes {delete(scope.ids)} asset.types_destroy(&a.types) asset.registry_destroy(&a.registry) + for problem in a.package_problems {delete(problem,a.allocator)} + delete(a.package_problems,a.allocator) a^={} } +@(private="file") +assets_texts_clone :: proc(texts:[]string,allocator:mem.Allocator) -> (kept:[]string,ok:bool) { + if len(texts)==0 {return nil,true} + no_room:runtime.Allocator_Error + if kept,no_room=make([]string,len(texts),allocator);no_room!=nil {return nil,false} + for text,i in texts { + if kept[i],no_room=strings.clone(text,allocator);no_room!=nil { + for done in kept[:i] {delete(done,allocator)} + delete(kept,allocator) + return nil,false + } + } + return kept,true +} + // Looks at the folder again: for files that appeared or went away. A folder // with a packed index is what its index says, and is not looked at. assets_scan :: proc(a:^Assets) -> bool {return a.indexed || asset.registry_scan(&a.registry)==.None} @@ -372,6 +392,8 @@ assets_unmount :: proc(a:^Assets,prefix:string) -> asset.Error { assets_errors :: proc(a:^Assets) -> []Assets_Error {return a.errors[:]} assets_errors_clear :: proc(a:^Assets) {clear(&a.errors)} +// What the packages of the project could not list, one line per package that names it. +assets_package_problems :: proc(a:^Assets) -> []string {return a.package_problems} // The id of the asset at a path of the folder, as its file is spelled // ("arena/Chrome.mat"): what code that names an asset by path resolves. diff --git a/host/assets_loading_test.odin b/host/assets_loading_test.odin index 6c15083cd..2b54e1b91 100644 --- a/host/assets_loading_test.odin +++ b/host/assets_loading_test.odin @@ -2,14 +2,18 @@ package host import "base:runtime" import "core:fmt" +import "core:os" +import "core:strings" import "core:testing" import "core:thread" import "mirai:asset" import "mirai:core" +import "mirai:project" import "mirai:render" import "mirai:std/base" import "mirai:std/jobs" import mm "mirai:std/math" +import "mirai:store" // Three loads share a frame: each gets a part of what the ones before it // left; one whose next asset does not fit its part, having taken nothing, @@ -356,3 +360,74 @@ what_a_load_read_is_freed_in_a_job :: proc(t:^testing.T) { app_release(&alone,{count,&freed}) testing.expect(t,freed==3 && alone.asset_releases.calls==nil,"called in the call") } + +// The dev player sets up the assets of a project with a list of packages and +// no imported folder or source: an output built in the store loads from its +// blob, a plugin's packages list under its prefix, and a build that is not on +// this machine is a problem that names its package, never a failure. +@(test) +the_assets_of_a_project_load_its_packages_from_the_store :: proc(t:^testing.T) { + folder:=base.test_folder(t,"host-packages") + if folder=="" {return} + root,staging:=fmt.tprintf("%s/game",folder),fmt.tprintf("%s/staging",folder) + plugin:=fmt.tprintf("%s/plugins/pack",root) + for made in ([]string{fmt.tprintf("%s/assets",root),fmt.tprintf("%s/assets",plugin),staging}) {testing.expect(t,os.make_directory_all(made)==nil)} + m:=project.manifest_default("Game") + base.text_set(m.plugins[0][:],"pack") + testing.expect(t,os.write_entire_file(fmt.tprintf("%s/%s",root,project.PROJECT_FILE),project.manifest_text(&m,context.temp_allocator))==nil) + pm:project.Plugin_Manifest + base.text_set(pm.name[:],"pack");base.text_set(pm.version[:],"1.0.0") + pm.parts={.Assets} + testing.expect(t,os.write_entire_file(fmt.tprintf("%s/%s",plugin,project.PLUGIN_FILE),project.plugin_text(&pm,context.temp_allocator))==nil) + + s,open_error:=store.open(fmt.tprintf("%s/store",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) + mesh:=base.Asset_Id{0x7061636b,1} + vertices:=[3]asset.Vertex{{position={0,0,0},normal=mm.normal_pack({0,1,0})},{position={1,0,0},normal=mm.normal_pack({0,1,0})},{position={0,0,1},normal=mm.normal_pack({0,1,0})}} + testing.expect_value(t,mesh_save_test(fmt.tprintf("%s/crate.mesh",staging),&types,mesh,vertices[:],{0,2,1}),asset.Error.None) + blob,put_error:=store.put_file(&s,fmt.tprintf("%s/crate.mesh",staging)) + testing.expect_value(t,put_error,store.Error.None) + os.remove_all(staging) + // The crate is built here; the atlas of the plugin's package is not. + crate:=asset.Package{id={20,20},name="crate"} + crate.assets=make([dynamic]asset.Package_Asset,context.temp_allocator) + append(&crate.assets,asset.Package_Asset{kind=asset.KIND_MESH,name="crate",id=mesh}) + pack:=asset.Package{id={30,30},name="pack"} + pack.assets=make([dynamic]asset.Package_Asset,context.temp_allocator) + append(&pack.assets,asset.Package_Asset{kind=asset.KIND_TEXTURE,name="atlas",id={31,31}}) + for entry in ([]struct{p:^asset.Package,root:string,built:bool}{{&crate,root,true},{&pack,plugin,false}}) { + manifest,error:=asset.package_put(&s,entry.p) + testing.expect_value(t,error,asset.Error.None) + if entry.built { + info,_:=asset.kind_info(asset.KIND_MESH) + b:=asset.Build{package_id=entry.p.id,final=true} + b.outputs=make([dynamic]asset.Build_Output,context.temp_allocator) + append(&b.outputs,asset.Build_Output{id=mesh,kind=asset.KIND_MESH,kind_version=info.version,blob=blob}) + built,_:=asset.build_put(&s,&b) + digits:[64]u8 + testing.expect_value(t,store.ref_set(&s,asset.REF_BUILDS,store.hex(asset.build_select(manifest,entry.p),&digits),built),store.Error.None) + } + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + append(&list.packages,asset.Project_Package{id=entry.p.id,manifest=manifest,folder="props"}) + testing.expect_value(t,asset.packages_save(entry.root,&list),asset.Error.None) + } + + p,error:=project.open(root,"",context.temp_allocator) + if !testing.expect_value(t,error.code,project.Error_Code.None) {return} + found,ok:=project_assets(&p,s.root,context.temp_allocator) + if !testing.expect(t,ok && len(found.listed)==1 && len(found.mounts)>=1) {return} + testing.expect(t,found.mounts[0].prefix=="plugins/pack" && len(found.mounts[0].listed)==1) + testing.expectf(t,len(found.problems)==1 && strings.contains(found.problems[0],"package pack"),"%v",found.problems) + a:Assets + if !testing.expect(t,assets_init(&a,{root=fmt.tprintf("%s/assets",root),listed=found.listed,mounts=found.mounts,package_problems=found.problems,capacity=8})) {return} + defer assets_destroy(&a) + testing.expect_value(t,len(assets_package_problems(&a)),1) + loaded:=assets_mesh(&a,mesh) + testing.expect(t,loaded!=nil && !loaded.failed && loaded.vertex_count==3,"the mesh loads from its blob") + testing.expect_value(t,asset.registry_file_path(&a.registry,"props/crate.mesh",context.temp_allocator),found.listed[0].file) + pending,listed:=asset.registry_find_path(&a.registry,"plugins/pack/props/atlas.tex") + testing.expect(t,listed && pending.state==.Not_Built) + testing.expect(t,!os.exists(fmt.tprintf("%s/.mirai/imported",root))) +} diff --git a/host/assets_watch.odin b/host/assets_watch.odin index 1356eb0bc..a7a3577ae 100644 --- a/host/assets_watch.odin +++ b/host/assets_watch.odin @@ -1,15 +1,33 @@ package host +import "core:log" import "core:strings" import "mirai:asset" import "mirai:platform" +import "mirai:project" import "mirai:std/jobs" @(private) -app_assets_create :: proc(app:^App,root,cache:string,mounts:[]asset.Registry_Mount=nil,imported:="") -> App_Error { +app_assets_create :: proc(app:^App,desc:App_Desc) -> App_Error { + root:=desc.assets_root + // The project lists its packages and the assets of its plugins; what its packages could not list is + // told, and never keeps it from starting. + listed:[]asset.Package_Listed + problems:[]string + mounts:=make([dynamic]asset.Registry_Mount,context.temp_allocator) + if desc.root!="" { + if p,open_error:=project.open(desc.root,desc.engine_root,context.temp_allocator);open_error.code==.None { + found,ok:=project_assets(&p,desc.store_root,context.temp_allocator) + if !ok {return .Out_Of_Memory} + listed,problems=found.listed,found.problems + if _,no_room:=append(&mounts,..found.mounts);no_room!=nil {return .Out_Of_Memory} + } + } + if _,no_room:=append(&mounts,..desc.asset_mounts);no_room!=nil {return .Out_Of_Memory} + for problem in problems {log.warnf("assets: %s",problem)} assets,error:=new(Assets,app.allocator) if error!=nil {return .Out_Of_Memory} - if !assets_init(assets,{root=root,imported=imported,cache=cache,renderer=app.renderer,allocator=app.allocator,mounts=mounts}) { + if !assets_init(assets,{root=root,listed=listed,package_problems=problems,cache=desc.assets_cache,renderer=app.renderer,allocator=app.allocator,mounts=mounts[:]}) { free(assets,app.allocator) return .Assets_Failed } diff --git a/host/content.odin b/host/content.odin index 1681263e0..e8193fbe7 100644 --- a/host/content.odin +++ b/host/content.odin @@ -808,18 +808,44 @@ content_collect :: proc(app:^App) -> (unloaded:int) { return len(gone) } -// The folders of assets of the active plugins of a project, as the registry -// mounts them: under plugins/, the path they have in a shipped game. -project_asset_mounts :: proc(p:^project.Project,allocator:=context.allocator) -> []asset.Registry_Mount { +// What the registry of a project lists besides the files of its assets. +Project_Assets :: struct { + listed:[]asset.Package_Listed, // the outputs of the packages of the project + mounts:[]asset.Registry_Mount, // the assets of each active plugin with the outputs of its packages, and the far meshes + problems:[]string, // what the packages of the project and its plugins could not list (asset.root_packages) +} + +// The assets of the project p besides its own files: the outputs of the +// packages of its root and of each active plugin with assets, from the store +// at store_root ("": the machine's), each plugin mounted under plugins/, +// the path its assets have in a shipped game. A project whose assets carry +// the index of a cook (a shipped game) opens no store and lists no package. +// False only when memory runs out. +project_assets :: proc(p:^project.Project,store_root:string,allocator:=context.allocator) -> (assets:Project_Assets,ok:bool) { mounts:=make([dynamic]asset.Registry_Mount,allocator) + problems:=make([dynamic]string,allocator) + own:=project.folder_path(p.root,.Assets,context.temp_allocator) + shipped:=os.exists(project.path(own,asset.REGISTRY_INDEX,allocator=context.temp_allocator)) + if !shipped { + packages,error:=asset.root_packages(p.root,store_root,allocator) + if _,no_room:=append(&problems,..packages.problems);error!=.None || no_room!=nil {return {},false} + assets.listed=packages.listed + } for folder in project.asset_folders(p,allocator) { - if folder.prefix!="" {append(&mounts,asset.Registry_Mount{folder=folder.path,prefix=folder.prefix})} + if folder.prefix=="" {continue} + mount:=asset.Registry_Mount{folder=folder.path,prefix=folder.prefix} + if !shipped { + packages,error:=asset.root_packages(folder.root,store_root,allocator) + if _,no_room:=append(&problems,..packages.problems);error!=.None || no_room!=nil {return {},false} + mount.listed=packages.listed + } + if _,no_room:=append(&mounts,mount);no_room!=nil {return {},false} } // The far meshes the editor made, where the assets are not cooked: a game that runs them shows them. - assets:=project.folder_path(p.root,.Assets,context.temp_allocator) far:=project.cache_path(p.root,project.FAR_CACHE,allocator=allocator) - if !os.exists(project.path(assets,asset.REGISTRY_INDEX,allocator=context.temp_allocator)) && (os.is_dir(far) || os.make_directory_all(far)==nil) { - append(&mounts,asset.Registry_Mount{folder=far,prefix=project.FAR_PREFIX}) + if !shipped && (os.is_dir(far) || os.make_directory_all(far)==nil) { + if _,no_room:=append(&mounts,asset.Registry_Mount{folder=far,prefix=project.FAR_PREFIX});no_room!=nil {return {},false} } - return mounts[:] + assets.mounts,assets.problems=mounts[:],problems[:] + return assets,true } diff --git a/host/player.odin b/host/player.odin index b25f67523..d57337a5c 100644 --- a/host/player.odin +++ b/host/player.odin @@ -354,10 +354,6 @@ player_run :: proc(info:^core.World_Info,project_root,engine_root:string,args:[] } } if surfaced {window.size={max(i32(options.surface[0]/options.surface[2]+0.5),1),max(i32(options.surface[1]/options.surface[2]+0.5),1)}} - mounts:[]asset.Registry_Mount - if p,open_error:=project.open(root,engine_root,context.temp_allocator);open_error.code==.None { - mounts=project_asset_mounts(&p,context.temp_allocator) - } gpu:=!options.no_gpu && !options.dedicated windowed:=!options.headless && gpu started:=time.tick_now() @@ -368,11 +364,9 @@ player_run :: proc(info:^core.World_Info,project_root,engine_root:string,args:[] fixed_dt=windowed || options.dedicated?0:1.0/60, audio=windowed || surfaced?.Device:.None, assets_root=os.is_directory(assets)?assets:"", - assets_imported=project.imported_path(root,context.temp_allocator), animation_cache=project.cache_path(root,project.ANIMATION_CACHE,allocator=context.temp_allocator), // A cooked game carries the pieces of its meshes: it decomposes none. collision_cache="" if os.exists(project.path(assets,asset.REGISTRY_INDEX,allocator=context.temp_allocator)) else project.content_cache_path(root,project.COLLISION_CACHE,allocator=context.temp_allocator), - asset_mounts=mounts, shaders_root=root, }) if app==nil { diff --git a/project/project.odin b/project/project.odin index 60a6e1767..1a5ea5efa 100644 --- a/project/project.odin +++ b/project/project.odin @@ -40,11 +40,6 @@ COLLISION_CACHE :: "collision" // in it: the convex pieces of meshes, by their FAR_CACHE :: "far" // in it: the far meshes of scenes the editor made, by what they are made of (asset.far_key) FAR_PREFIX :: "mirai-far" // where the registry of a project that is not cooked mounts the far cache THUMBNAIL_FOLDER :: "thumbs" // beside it: the previews of the editor -// Under the generated folder of a project or a plugin: what the imports of its -// sources build that nobody edits (meshes, textures, sounds, fonts), each at -// the path it has among the assets. Built again from the sources and their -// sidecars wherever it is missing (cook/import/sources.sync). -IMPORTED_FOLDER :: "imported" Plugin :: struct { manifest:Plugin_Manifest, @@ -342,11 +337,6 @@ cache_path :: proc(root:string,parts:..string,allocator:=context.allocator) -> s return strings.to_string(b) } -// The imported folder of the project or the plugin at root. -imported_path :: proc(root:string,allocator:=context.allocator) -> string { - return path(folder_path(root,.Generated,context.temp_allocator),IMPORTED_FOLDER,allocator=allocator) -} - folder_path :: proc(root:string,folder:Folder,allocator:=context.allocator) -> string { names:=FOLDER_NAMES return path(root,names[folder],allocator=allocator) @@ -417,9 +407,9 @@ recipe_folders :: proc(p:^Project,allocator:=context.allocator) -> []string { // A folder of assets of a project, and the path its files have among the // assets: "" for the ones of the project, plugins/ for those of a plugin. -// imported: the imported folder that goes with it, whose files are assets at -// the same paths. root: the project or the plugin the folder is of. -Asset_Folder :: struct {path,prefix,imported,root:string} +// root: the project or the plugin the folder is of, where the list of the +// packages it uses is (asset.root_packages). +Asset_Folder :: struct {path,prefix,root:string} // The assets folder of the project, when it has one, and the one of each // active plugin that declares the part, in the order of p.active. @@ -427,14 +417,13 @@ asset_folders :: proc(p:^Project,allocator:=context.allocator) -> []Asset_Folder folders:=make([dynamic]Asset_Folder,allocator) names:=FOLDER_NAMES parts:=PART_NAMES - if own:=folder_path(p.root,.Assets,allocator);os.is_dir(own) {append(&folders,Asset_Folder{own,"",imported_path(p.root,allocator),p.root})} + if own:=folder_path(p.root,.Assets,allocator);os.is_dir(own) {append(&folders,Asset_Folder{own,"",p.root})} for index in p.active { plugin:=&p.plugins[index] if .Assets not_in plugin.manifest.parts {continue} append(&folders,Asset_Folder{ path(plugin.folder,parts[.Assets],allocator=allocator), path(names[.Plugins],base.text_get(plugin.manifest.name[:]),allocator=allocator), - imported_path(plugin.folder,allocator), plugin.folder, }) } diff --git a/tools/mirai/asset.odin b/tools/mirai/asset.odin index c69f654a3..b111ab136 100644 --- a/tools/mirai/asset.odin +++ b/tools/mirai/asset.odin @@ -29,8 +29,8 @@ Asset_Command :: struct {name,usage,summary:string,run:proc(root:string,args:[]s ASSET_COMMANDS :: [?]Asset_Command{ {"info","info ","Header, chunks and tables",asset_info}, - {"ls","ls [--cache ]","Assets of a folder: id, kind, bytes, dependencies and path",asset_ls}, - {"deps","deps [--root ] [--all|--users]","Dependencies of an asset, or the assets that depend on it",asset_deps}, + {"ls","ls [--cache ] [--store ]","Assets of a folder: id, kind, bytes, dependencies and path",asset_ls}, + {"deps","deps [--root ] [--store ] [--all|--users]","Dependencies of an asset, or the assets that depend on it",asset_deps}, {"verify","verify ...","Checks the container, hash, tables and the rules of the kind; of a scene, what mirai scene check says",asset_verify}, {"dump","dump [--schema|--textconv]","The whole asset as JSON, a scene or a .ui as its document, or as the text git diffs",asset_dump}, {"get","get ","A table, a row or a value as JSON",asset_get}, @@ -240,21 +240,24 @@ asset_info :: proc(_:string,args:[]string) -> int { } asset_ls :: proc(_:string,args:[]string) -> int { - folder,cache:="","" + folder,cache,store_root:="","","" for i:=0;i int { } asset_deps :: proc(_:string,args:[]string) -> int { - file,folder:="","" + file,folder,store_root:="","","" all,users:=false,false for i:=0;i int { print(&w) return 0 } + packages:=packages_beside(folder,store_root) + for problem in packages.problems {fmt.eprintfln("mirai asset deps: %s",problem)} r:asset.Registry - asset.registry_init(&r,folder) + asset.registry_init(&r,folder,listed=packages.listed) if error:=asset.registry_scan(&r);error!=.None {return failed("deps",folder,error_text(error))} id:=a.blob.header.id missing:=0 @@ -738,14 +744,16 @@ asset_pack :: proc(_:string,args:[]string) -> int { return 0 } -// The imported folder that goes with a folder of assets: that of the project -// or the plugin whose assets folder it is; "" for any other folder. +// What the packages of the project or the plugin whose assets folder is folder +// list, from the store at store_root ("": the machine's); nothing for any other +// folder. In the temp allocator. @(private="file") -imported_beside :: proc(folder:string) -> string { +packages_beside :: proc(folder,store_root:string) -> asset.Root_Packages { absolute:=project.path_absolute(folder,context.temp_allocator) names:=project.FOLDER_NAMES - if os.base(absolute)!=names[.Assets] {return ""} - return project.imported_path(os.dir(absolute),context.temp_allocator) + if os.base(absolute)!=names[.Assets] {return {}} + packages,_:=asset.root_packages(os.dir(absolute),store_root,context.temp_allocator) + return packages } // Writes each asset again compressed as the project that holds it saves (build.compression), or diff --git a/tools/mirai/session.odin b/tools/mirai/session.odin index ffa24a536..c4ddd1fc9 100644 --- a/tools/mirai/session.odin +++ b/tools/mirai/session.odin @@ -348,15 +348,14 @@ SESSION_COPY_PREFIX :: "session-copy-" // Whether a path relative to the project, with forward slashes, stays out of // a copy: the history of git and the generated folder except its caches and -// what its imports built, so that neither the port file, the record, the locks -// nor the libraries of the editor that holds the project go with it, and the -// copy imports nothing again. +// thumbnails, so that neither the port file, the record, the locks nor the +// libraries of the editor that holds the project go with it. session_copy_skips :: proc(relative:string) -> bool { first,_,rest:=strings.partition(relative,"/") if first==".git" {return true} if first!=project.FOLDER_NAMES[.Generated] || rest=="" {return false} kept,_,_:=strings.partition(rest,"/") - if kept!=project.CACHE_FOLDER && kept!=project.THUMBNAIL_FOLDER && kept!=project.IMPORTED_FOLDER {return true} + if kept!=project.CACHE_FOLDER && kept!=project.THUMBNAIL_FOLDER {return true} return strings.has_suffix(relative,".lock") } diff --git a/tools/mirai/session_test.odin b/tools/mirai/session_test.odin index 9be9fed98..72e0efc5c 100644 --- a/tools/mirai/session_test.odin +++ b/tools/mirai/session_test.odin @@ -135,16 +135,16 @@ a_path_is_inside_a_folder_only_past_a_separator :: proc(t:^testing.T) { } } -// A copy for a session takes the sources, the assets, the caches and what the imports built, never +// A copy for a session takes the sources, the assets, the list of its packages and the caches, never // the files of the editor that holds the project; stop knows it by its place. @(test) a_session_copy_takes_the_project_and_its_caches_without_the_files_of_its_editor :: proc(t:^testing.T) { context.allocator=context.temp_allocator folder:=base.test_folder(t,"copy") if folder=="" {return} - kept:=[]string{project.PROJECT_FILE,"assets/scenes/main.scene","game/game.odin","assets/props/art.fbx", - ".mirai/cache/registry/index",".mirai/thumbs/main.png",".mirai/imported/props/art.mesh"} - left:=[]string{".git/HEAD",".mirai/api.json",".mirai/session.json",".mirai/session.lock", + kept:=[]string{project.PROJECT_FILE,"packages.mirai","assets/scenes/main.scene","game/game.odin","assets/props/art.fbx", + ".mirai/cache/registry/index",".mirai/thumbs/main.png"} + left:=[]string{".git/HEAD",".mirai/imported/props/art.mesh",".mirai/api.json",".mirai/session.json",".mirai/session.lock", ".mirai/session.log",".mirai/editor.dll",".mirai/editor-live-12-0.dll",".mirai/gen/world.odin", ".mirai/tmp/probe.scene",".mirai/cache/textures.lock"} for files in ([][]string{kept,left}) { From 644922c3b7cf42ce4d05aac4cb41b940aeadbfe6 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 19:07:50 -0300 Subject: [PATCH 014/123] store: a sweep safe beside writers, and leases The sweep keeps blobs younger than a grace age, claims an old one by renaming it away before judging it again, skips what will not delete, and treats the addresses of fresh leases as live. A put of a blob already here refreshes its time. Written with Codex. Co-Authored-By: Claude Fable 5.2 --- .agents/skills/mirai-store/SKILL.md | 9 +- store/blobs.odin | 87 +++++++++----- store/concurrent_test.odin | 62 ++++++++++ store/hostile_test.odin | 21 ++++ store/leases.odin | 144 ++++++++++++++++++++++ store/receive.odin | 43 ++++--- store/refs.odin | 2 +- store/rename_darwin.odin | 23 ++++ store/rename_linux.odin | 16 +++ store/rename_windows.odin | 36 ++++++ store/space.odin | 177 ++++++++++++++++++++-------- store/store.odin | 25 ++-- store/store_test.odin | 68 +++++++++-- store/sweep_test.odin | 148 +++++++++++++++++++++++ 14 files changed, 746 insertions(+), 115 deletions(-) create mode 100644 store/leases.odin create mode 100644 store/rename_darwin.odin create mode 100644 store/rename_linux.odin create mode 100644 store/rename_windows.odin create mode 100644 store/sweep_test.odin diff --git a/.agents/skills/mirai-store/SKILL.md b/.agents/skills/mirai-store/SKILL.md index 989c54613..f1829709a 100644 --- a/.agents/skills/mirai-store/SKILL.md +++ b/.agents/skills/mirai-store/SKILL.md @@ -60,7 +60,14 @@ derived data cache that use it decide what a ref means and what is live. path or hold it open with a `Reader`. - The root carries its layout version; a root of another layout is refused, never read. - The store never decides what is live: sweep removes the complete blobs outside the set it is - given, and partial blobs only go through remove. + given plus the addresses of every fresh lease, and partial blobs only go through remove. +- A sweep is safe beside writers of other processes: it never removes a blob younger than the + grace age, a put of a blob already here refreshes its time, and the sweep claims an old blob by + renaming it away before it judges its age again, giving it back if a put refreshed it. A put that + succeeds is never lost to a sweep; a reader may find a blob missing for the instant of a claim, + which is a miss like any other. A blob that will not delete is skipped and counted. +- A process that uses blobs holds a lease, a file it refreshes while it runs; a lease that stops + being refreshed stops protecting. - Ref namespaces and names are lowercase ASCII, digits, `.`, `_` and `-`, so a ref is the same file on every system. - No input terminates the process: damaged or truncated files of the root, random tables and diff --git a/store/blobs.odin b/store/blobs.odin index ee5c49d51..896ed5060 100644 --- a/store/blobs.odin +++ b/store/blobs.odin @@ -3,34 +3,45 @@ package store import "base:runtime" import "core:mem" import "core:os" +import "core:time" // Complete blobs are plain files of their bytes at blobs//
, // both in hex, and are never written in place. -put_bytes :: proc(s:^Store,data:[]u8) -> (hash:Hash,error:Error) { +put_bytes :: proc(s:^Store,data:[]u8,now:Maybe(time.Time)=nil) -> (hash:Hash,error:Error) { + if s==nil || s.root=="" {return {},.Invalid_Argument} pieces:=bytes_table(data,s.allocator) or_return defer delete(pieces,s.allocator) hash=address_value(i64(len(data)),pieces) - if has(s,hash) {return hash,.None} + refreshed:=blob_refresh(s,hash,now.? or_else time.now()) + return hash,put_bytes_after_refresh(s,data,pieces,hash,refreshed,now) +} + +@(private) +put_bytes_after_refresh :: proc(s:^Store,data:[]u8,pieces:[]Hash,hash:Hash, + refreshed:Error,now:Maybe(time.Time)) -> Error { + if refreshed!=.Missing {return refreshed} temp_buffer:[PATH_BYTES]u8 temp,file:=temp_create(s.root,&temp_buffer) or_return - _,write_error:=os.write(file,data) + written,write_error:=os.write(file,data) sync_error:=os.sync(file) os.close(file) - if write_error!=nil || sync_error!=nil { + if write_error!=nil || written!=len(data) || sync_error!=nil { os.remove(temp) - return {},os_error(write_error) if write_error!=nil else .Io + return os_error(write_error) if write_error!=nil else .Io } - if error=blob_place(s,temp,hash);error!=.None { + placed,place_error:=blob_place(s,temp,hash,now) + if place_error!=.None { os.remove(temp) - return hash,error + return place_error } - table_keep(s,hash,i64(len(data)),pieces) - return hash,.None + if placed {table_keep(s,hash,i64(len(data)),pieces)} + return .None } // Puts the file at path, read once piece by piece. -put_file :: proc(s:^Store,path:string) -> (hash:Hash,error:Error) { +put_file :: proc(s:^Store,path:string,now:Maybe(time.Time)=nil) -> (hash:Hash,error:Error) { + if s==nil || s.root=="" {return {},.Invalid_Argument} source,open_error:=os.open(path) if open_error!=nil {return {},os_error(open_error)} defer os.close(source) @@ -41,20 +52,15 @@ put_file :: proc(s:^Store,path:string) -> (hash:Hash,error:Error) { defer delete(pieces,s.allocator) temp_buffer:[PATH_BYTES]u8 temp,file:=temp_create(s.root,&temp_buffer) or_return + defer os.remove(temp) hash,error=file_hash(source,size,file,pieces) sync_error:=os.sync(file) if error==.None else nil os.close(file) if error==.Corrupt {error=.Io} if error==.None && sync_error!=nil {error=.Io} - if error!=.None || has(s,hash) { - os.remove(temp) - return hash if error==.None else {},error - } - if error=blob_place(s,temp,hash);error!=.None { - os.remove(temp) - return hash,error - } - table_keep(s,hash,size,pieces) + if error!=.None {return {},error} + placed:=blob_place(s,temp,hash,now) or_return + if placed {table_keep(s,hash,size,pieces)} return hash,.None } @@ -188,20 +194,45 @@ blob_file :: proc(s:^Store,hash:Hash,buffer:^[PATH_BYTES]u8) -> (string,Error) { return join(buffer,s.root,"blobs",text[:2],text) } -// Renames a finished file into the blob's place; a blob already there stays -// and the file goes. A failed rename leaves the file where it was. @(private) -blob_place :: proc(s:^Store,temp:string,hash:Hash) -> Error { +blob_refresh :: proc(s:^Store,hash:Hash,now:time.Time) -> Error { + buffer:[PATH_BYTES]u8 + path:=blob_file(s,hash,&buffer) or_return + at:=regular_time(s,path) or_return + if !older_than(at,now,BLOB_REFRESH_AGE) {return .None} + return blob_touch(s,path,now) +} + +@(private) +blob_touch :: proc(s:^Store,path:string,now:time.Time) -> Error { + os_error(os.change_times(path,now,now)) or_return + // A path-based time change can resolve the file just before it is claimed. + at:=regular_time(s,path) or_return + return .Missing if older_than(at,now,BLOB_REFRESH_AGE) else .None +} + +// Renames a finished file into place, or refreshes the blob another writer placed. +@(private) +blob_place :: proc(s:^Store,temp:string,hash:Hash,clock:Maybe(time.Time)) -> + (placed:bool,error:Error) { buffer,folder_buffer:[PATH_BYTES]u8 path:=blob_file(s,hash,&buffer) or_return name:[64]u8 text:=hex(hash,&name) - folder_make(join(&folder_buffer,s.root,"blobs",text[:2]) or_return) or_return - if os.is_file(path) || os.rename(temp,path)!=nil { - if !os.is_file(path) {return .Io} - os.remove(temp) + for _ in 0..size_of(Hash)) {return} + testing.expect(t,os.write_entire_file(truncated,valid[:len(valid)-size_of(Hash)])==nil) + random_lease:=path_in(folder,"leases","random") + testing.expect(t,os.write_entire_file(random_lease,test_bytes(len(valid),42))==nil) + oversized:=path_in(folder,"leases","oversized") + lease_bytes:=make([]u8,len(LEASE_MAGIC)+(LEASE_MAX_HASHES+2)*size_of(Hash), + context.temp_allocator) + copy(lease_bytes,valid) + testing.expect(t,os.write_entire_file(oversized,lease_bytes)==nil) + for file in ([3]string{truncated,random_lease,oversized}) {test_set_time(t,file,now)} + _,hostile_sweep_error:=sweep(&s,nil,now) + testing.expect_value(t,hostile_sweep_error,Error.None) + testing.expect(t,!has(&s,unleased),"invalid leases do not keep addresses") + // Memory that runs out. out:=s out.allocator=failing_allocator() diff --git a/store/leases.odin b/store/leases.odin new file mode 100644 index 000000000..1a6ac6c08 --- /dev/null +++ b/store/leases.odin @@ -0,0 +1,144 @@ +package store + +import "core:mem" +import "core:os" +import "core:time" + +LEASE_MAX_AGE :: 5*time.Minute +LEASE_MAX_HASHES :: 1<<20 // an editor with a large map open holds tens of thousands + +@(private) +LEASE_MAGIC :: "mirai lease 1\n" + +Lease :: struct { + store:^Store, + name:[NAME_MAX]u8, + name_len:int, + live:[]Hash, +} + +// The holder supplies a unique name and keeps the store open through release. +lease_open :: proc(s:^Store,name:string,live:[]Hash,now:Maybe(time.Time)=nil) -> + (lease:Lease,error:Error) { + if s==nil || s.root=="" || !name_valid(name) {return {},.Invalid_Argument} + lease.store=s + lease.name_len=copy(lease.name[:],name) + error=lease_update(&lease,live,now) + if error!=.None {return {},error} + return lease,.None +} + +lease_update :: proc(lease:^Lease,live:[]Hash,now:Maybe(time.Time)=nil) -> Error { + if lease==nil || lease.store==nil || len(live)>LEASE_MAX_HASHES {return .Invalid_Argument} + buffer:[PATH_BYTES]u8 + path:=lease_path(lease,&buffer) or_return + allocator:=lease.store.allocator + held,make_error:=make([]Hash,len(live),allocator) + if make_error!=nil {return .Out_Of_Memory} + copy(held,live) + if error:=lease_write(lease.store.root,path,held,now.? or_else time.now());error!=.None { + delete(held,allocator) + return error + } + delete(lease.live,allocator) + lease.live=held + return .None +} + +lease_refresh :: proc(lease:^Lease,now:Maybe(time.Time)=nil) -> Error { + buffer:[PATH_BYTES]u8 + path:=lease_path(lease,&buffer) or_return + return lease_write(lease.store.root,path,lease.live,now.? or_else time.now()) +} + +lease_release :: proc(lease:^Lease) -> Error { + buffer:[PATH_BYTES]u8 + path:=lease_path(lease,&buffer) or_return + error:=os_error(os.remove(path)) + delete(lease.live,lease.store.allocator) + lease^={} + return .None if error==.Missing else error +} + +@(private) +lease_path :: proc(lease:^Lease,buffer:^[PATH_BYTES]u8) -> (string,Error) { + if lease==nil || lease.store==nil || lease.store.root=="" || + lease.name_len<1 || lease.name_len>len(lease.name) { + return "",.Invalid_Argument + } + name:=string(lease.name[:lease.name_len]) + if !name_valid(name) {return "",.Invalid_Argument} + return join(buffer,lease.store.root,"leases",name) +} + +@(private) +lease_write :: proc(root,path:string,live:[]Hash,now:time.Time) -> Error { + if len(live)>LEASE_MAX_HASHES {return .Invalid_Argument} + payload:=mem.slice_to_bytes(live) + checksum:=address_of_bytes(payload) + return file_write(root,path,{transmute([]u8)string(LEASE_MAGIC),checksum[:],payload},now) +} + +@(private) +leases_collect :: proc(s:^Store,live:^Live_Set,now:time.Time) -> Error { + buffer:[PATH_BYTES]u8 + folder:=join(&buffer,s.root,"leases") or_return + entries,read_error:=os.read_all_directory_by_path(folder,s.allocator) + if read_error==.Not_Exist {return .None} + if read_error!=nil {return os_error(read_error)} + defer os.file_info_slice_delete(entries,s.allocator) + for entry in entries { + if entry.type!=.Regular || !name_valid(entry.name) {continue} + if older_than(entry.modification_time,now,LEASE_MAX_AGE) { + lease_collect_stale(s,live,entry.fullpath,now) or_return + } + lease_collect(s,live,entry.fullpath,now) or_return + } + return .None +} + +@(private) +lease_collect_stale :: proc(s:^Store,live:^Live_Set,path:string,now:time.Time) -> Error { + buffer:[PATH_BYTES]u8 + claimed,claim_error:=sweep_claim(s,path,&buffer) + if claim_error!=.None {return .None} + defer os.remove(os.dir(claimed)) + at,time_error:=regular_time(s,claimed) + if time_error==.None && older_than(at,now,LEASE_MAX_AGE) { + if os.remove(claimed)==nil {return .None} + } + collected:=lease_collect(s,live,claimed,now) + restored:=sweep_restore(path,claimed) + return collected if collected!=.None else restored +} + +@(private) +lease_collect :: proc(s:^Store,live:^Live_Set,path:string,now:time.Time) -> Error { + file,open_error:=os.open(path) + if open_error==.Not_Exist {return .None} + if open_error!=nil {return os_error(open_error)} + defer os.close(file) + info,stat_error:=os.fstat(file,s.allocator) + if stat_error!=nil {return os_error(stat_error)} + defer os.file_info_delete(info,s.allocator) + if info.type!=.Regular || older_than(info.modification_time,now,LEASE_MAX_AGE) {return .None} + + HEADER_BYTES :: len(LEASE_MAGIC)+size_of(Hash) + if info.sizeHEADER_BYTES+LEASE_MAX_HASHES*size_of(Hash) { + return .None + } + bytes:=info.size-HEADER_BYTES + if bytes%size_of(Hash)!=0 {return .None} + header:[HEADER_BYTES]u8 + if read_exact(file,header[:],0)!=.None || string(header[:len(LEASE_MAGIC)])!=LEASE_MAGIC { + return .None + } + hashes,make_error:=make([]Hash,int(bytes/size_of(Hash)),s.allocator) + if make_error!=nil {return .Out_Of_Memory} + defer delete(hashes,s.allocator) + payload:=mem.slice_to_bytes(hashes) + if read_exact(file,payload,HEADER_BYTES)!=.None {return .None} + checksum:=address_of_bytes(payload) + if mem.compare(checksum[:],header[len(LEASE_MAGIC):])!=0 {return .None} + return live_add(live,hashes) +} diff --git a/store/receive.odin b/store/receive.odin index a09fd1233..fb62f06cd 100644 --- a/store/receive.odin +++ b/store/receive.odin @@ -41,18 +41,19 @@ Claim :: struct { CLAIMED :: ".claimed" // Starts receiving a blob from its address and piece table; the table is -// checked against the address before anything is kept. A blob already -// complete or being received is left as it is; one being received whose -// pieces are all here is completed. -receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { +// checked against the address before anything is kept. +receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash,now:Maybe(time.Time)=nil) -> Error { + if s==nil || s.root=="" {return .Invalid_Argument} if size<0 || len(pieces)!=piece_count(size) {return .Invalid_Argument} if address_value(size,pieces)!=hash {return .Corrupt} - if has(s,hash) { + refreshed:=blob_refresh(s,hash,now.? or_else time.now()) + if refreshed==.None { partial_discard(s,hash) return .None } + if refreshed!=.Missing {return refreshed} if size==0 { - _,error:=put_bytes(s,nil) + _,error:=put_bytes(s,nil,now) return error } existing,existing_error:=partial_open(s,hash,{.Data,.Have}) @@ -61,7 +62,7 @@ receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { partial_close(&existing) if full_error!=.None {return full_error} if !full {return .None} - if finished:=partial_finish(s,hash);finished!=.Corrupt {return finished} + if finished:=partial_finish(s,hash,now);finished!=.Corrupt {return finished} } else if existing_error==.Corrupt { partial_discard(s,hash) } @@ -79,7 +80,8 @@ receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { // Keeps one received piece: checked against the table and written at its // offset; a duplicate is harmless. The piece that leaves none missing // completes the blob. -receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { +receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8,now:Maybe(time.Time)=nil) -> Error { + if s==nil || s.root=="" {return .Invalid_Argument} p,error:=partial_open(s,hash,{.Data,.Have}) if error==.Missing {return complete_piece(s,hash,index,piece)} if error!=.None {return error} @@ -94,12 +96,14 @@ receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { full:=partial_full(&p) or_return partial_close(&p) if !full {return .None} - return partial_finish(s,hash) + return partial_finish(s,hash,now) } // The indices of the pieces a blob being received still lacks; none for a // complete blob. One with none missing is completed first. -receive_missing :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (missing:[]int,error:Error) { +receive_missing :: proc(s:^Store,hash:Hash,allocator:=context.allocator,now:Maybe(time.Time)=nil) -> + (missing:[]int,error:Error) { + if s==nil || s.root=="" {return nil,.Invalid_Argument} for attempt in 0..<2 { if has(s,hash) {return nil,.None} p:Partial @@ -110,7 +114,7 @@ receive_missing :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (mis partial_close(&p) if error!=.None || len(missing)>0 || attempt==1 {return missing,error} delete(missing,allocator) - finished:=partial_finish(s,hash) + finished:=partial_finish(s,hash,now) if finished!=.None {return nil,finished} } return nil,.None @@ -215,31 +219,31 @@ partial_full :: proc(p:^Partial) -> (full:bool,error:Error) { // claims it, checks it and publishes it, or gives it back. Nothing to claim // is no error: another receiver holds it or it is gone. @(private) -partial_finish :: proc(s:^Store,hash:Hash) -> Error { +partial_finish :: proc(s:^Store,hash:Hash,now:Maybe(time.Time)) -> Error { c:Claim - claimed:=partial_claim(s,hash,&c) + claimed:=partial_claim(s,hash,&c,now.? or_else time.now()) if claimed==.Missing {return .None} if claimed!=.None {return claimed} defer claim_end(s,hash,&c) complete:=claim_verify(s,hash,&c) or_return if !complete {return .None} - return claim_publish(s,hash,&c) + return claim_publish(s,hash,&c,now) } // Renames the partial folder to its claimed name. A claim older than // TMP_STALE was left by a process that ended: it goes back first. @(private) -partial_claim :: proc(s:^Store,hash:Hash,c:^Claim) -> Error { +partial_claim :: proc(s:^Store,hash:Hash,c:^Claim,now:time.Time) -> Error { from_buffer:[PATH_BYTES]u8 from:=partial_folder(s,hash,&from_buffer) or_return claimed:=claimed_folder(s,hash,&c.buffer) or_return if !os.is_directory(from) { - if at,time_error:=os.modification_time_by_path(claimed);time_error==nil && time.since(at)>TMP_STALE { + at,time_error:=os.modification_time_by_path(claimed) + if time_error==nil && older_than(at,now,TMP_STALE) { os.rename(claimed,from) } if !os.is_directory(from) {return .Missing} } - now:=time.now() os.change_times(from,now,now) if os.rename(from,claimed)!=nil {return .Missing} c.partial=claimed @@ -280,10 +284,11 @@ claim_verify :: proc(s:^Store,hash:Hash,c:^Claim) -> (complete:bool,error:Error) // Moves the checked data into the blob's place, and its table to the kept tables. @(private) -claim_publish :: proc(s:^Store,hash:Hash,c:^Claim) -> Error { +claim_publish :: proc(s:^Store,hash:Hash,c:^Claim,now:Maybe(time.Time)=nil) -> Error { buffer,table_buffer,kept_buffer,folder_buffer:[PATH_BYTES]u8 - blob_place(s,join(&buffer,c.partial,"data") or_return,hash) or_return + placed:=blob_place(s,join(&buffer,c.partial,"data") or_return,hash,now) or_return c.published=true + if !placed {return .None} name:[64]u8 if folder_make(join(&folder_buffer,s.root,"tables",hex(hash,&name)[:2]) or_return)==.None { os.rename(join(&table_buffer,c.partial,"table") or_return,table_file(s,hash,&kept_buffer) or_return) diff --git a/store/refs.odin b/store/refs.odin index b7ff7ef36..e7f9504ef 100644 --- a/store/refs.odin +++ b/store/refs.odin @@ -26,7 +26,7 @@ ref_set :: proc(s:^Store,namespace,name:string,hash:Hash) -> Error { path:=ref_file(s,namespace,name,&buffer) or_return folder_make(join(&folder_buffer,s.root,"refs",namespace) or_return) or_return text:[64]u8 - return file_write(s.root,path,transmute([]u8)hex(hash,&text)) + return file_write(s.root,path,{transmute([]u8)hex(hash,&text)}) } ref_remove :: proc(s:^Store,namespace,name:string) -> Error { diff --git a/store/rename_darwin.odin b/store/rename_darwin.odin new file mode 100644 index 000000000..92255055f --- /dev/null +++ b/store/rename_darwin.odin @@ -0,0 +1,23 @@ +package store + +import "core:c" +import "core:sys/posix" + +foreign import libc "system:System" + +@(private) +foreign libc { + renamex_np :: proc "c" (from,to:cstring,flags:c.uint) -> c.int --- +} + +@(private) +rename_new :: proc(from,to:string) -> (exists:bool,error:Error) { + a,b:[PATH_BYTES+1]u8 + if len(from)>=len(a) || len(to)>=len(b) {return false,.Invalid_Argument} + copy(a[:],from) + copy(b[:],to) + RENAME_EXCL :: 0x00000004 + if renamex_np(cstring(&a[0]),cstring(&b[0]),RENAME_EXCL)==0 {return false,.None} + if posix.errno()==.EEXIST {return true,.None} + return false,.Io +} diff --git a/store/rename_linux.odin b/store/rename_linux.odin new file mode 100644 index 000000000..4e3cc4a66 --- /dev/null +++ b/store/rename_linux.odin @@ -0,0 +1,16 @@ +package store + +import "core:sys/linux" + +@(private) +rename_new :: proc(from,to:string) -> (exists:bool,error:Error) { + a,b:[PATH_BYTES+1]u8 + if len(from)>=len(a) || len(to)>=len(b) {return false,.Invalid_Argument} + copy(a[:],from) + copy(b[:],to) + result:=linux.renameat2(linux.AT_FDCWD,cstring(&a[0]), + linux.AT_FDCWD,cstring(&b[0]),{.NOREPLACE}) + if result==.NONE {return false,.None} + if result==.EEXIST {return true,.None} + return false,.Io +} diff --git a/store/rename_windows.odin b/store/rename_windows.odin new file mode 100644 index 000000000..30ef873ef --- /dev/null +++ b/store/rename_windows.odin @@ -0,0 +1,36 @@ +package store + +import "core:sys/windows" + +@(private) +rename_new :: proc(from,to:string) -> (exists:bool,error:Error) { + a,b:[PATH_BYTES+9]u16 + old:=rename_path(from,&a) or_return + new:=rename_path(to,&b) or_return + if windows.MoveFileExW(old,new,0) {return false,.None} + code:=windows.GetLastError() + if code==windows.ERROR_ALREADY_EXISTS || code==windows.ERROR_FILE_EXISTS {return true,.None} + return false,.Io +} + +@(private) +rename_path :: proc(path:string,buffer:^[PATH_BYTES+9]u16) -> (cstring16,Error) { + text:[PATH_BYTES+1]u16 + if len(path)==0 || len(path)>PATH_BYTES {return nil,.Invalid_Argument} + converted:=windows.MultiByteToWideChar(windows.CP_UTF8,windows.MB_ERR_INVALID_CHARS, + raw_data(path),i32(len(path)),raw_data(text[:]),i32(len(text)-1)) + if converted==0 {return nil,.Invalid_Argument} + target:=buffer[8:] + n:=windows.GetFullPathNameW(cstring16(&text[0]),u32(len(target)),cstring16(&target[0]),nil) + if n==0 || int(n)>=len(target) {return nil,.Io} + prefix:=[4]u16{'\\','\\','?','\\'} + if n<248 || (^[4]u16)(&target[0])^==prefix { + return cstring16(&target[0]),.None + } + if target[0]=='\\' && target[1]=='\\' { + copy(buffer[2:10],[]u16{'\\','\\','?','\\','U','N','C','\\'}) + return cstring16(&buffer[2]),.None + } + copy(buffer[4:8],prefix[:]) + return cstring16(&buffer[4]),.None +} diff --git a/store/space.odin b/store/space.odin index 250f964cb..a1cb1e7fb 100644 --- a/store/space.odin +++ b/store/space.odin @@ -1,82 +1,163 @@ package store -import "core:mem" import "core:os" -import "core:slice" +import "core:strings" import "core:time" import "mirai:std/base" -// Temporary files left by a process that ended before renaming them; a -// younger one may still be written. +BLOB_GRACE_AGE :: time.Hour +BLOB_REFRESH_AGE :: BLOB_GRACE_AGE/4 + @(private) TMP_STALE :: time.Hour +@(private) +SWEEP_PREFIX :: "sweep-" + Usage :: struct { bytes:i64, blobs:int, } -// What the complete blobs take. +Sweep_Result :: struct { + using freed:Usage, + skipped:int, +} + +@(private) +Live_Set :: map[Hash]struct{} + +@(private) +Sweep :: struct { + s:^Store, + live:Live_Set, + now:time.Time, + result:Sweep_Result, + tables:bool, +} + usage :: proc(s:^Store) -> (total:Usage,error:Error) { - error=blobs_walk(s,"blobs",proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { + error=blobs_walk(s,"blobs",proc(user:rawptr,file:os.File_Info,hash:Hash) { total:=(^Usage)(user) - total.bytes+=size + total.bytes+=file.size total.blobs+=1 - return .None },&total) return } -// Removes every complete blob whose address is not in live, with its kept -// table, and the stale files of tmp: what it freed. The caller decides what is live. -sweep :: proc(s:^Store,live:[]Hash) -> (freed:Usage,error:Error) { - Sweep :: struct {live:[]Hash,freed:Usage,tables:bool} - sorted,make_error:=make([]Hash,len(live),s.allocator) +sweep :: proc(s:^Store,live:[]Hash,now:Maybe(time.Time)=nil) -> (result:Sweep_Result,error:Error) { + if s==nil || s.root=="" {return {},.Invalid_Argument} + at:=now.? or_else time.now() + set,make_error:=make(Live_Set,len(live),s.allocator) if make_error!=nil {return {},.Out_Of_Memory} - defer delete(sorted,s.allocator) - copy(sorted,live) - slice.sort_by(sorted,proc(a,b:Hash) -> bool { - a,b:=a,b - return mem.compare(a[:],b[:])<0 - }) - state:=Sweep{live=sorted} - visit:=proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error { - state:=(^Sweep)(user) - _,found:=slice.binary_search_by(state.live,hash,proc(a,b:Hash) -> slice.Ordering { - a,b:=a,b - return slice.Ordering(mem.compare(a[:],b[:])) - }) - if found {return .None} - removed:=os.remove(path) - if removed==.Not_Exist {return .None} - if removed!=nil {return os_error(removed)} - if !state.tables { - state.freed.bytes+=size - state.freed.blobs+=1 - } - return .None - } - error=blobs_walk(s,"blobs",visit,&state) - if error!=.None {return state.freed,error} + defer delete(set) + live_add(&set,live) or_return + leases_collect(s,&set,at) or_return + state:=Sweep{s=s,live=set,now=at} + error=blobs_walk(s,"blobs",sweep_entry,&state) + if error!=.None {return state.result,error} state.tables=true - error=blobs_walk(s,"tables",visit,&state) - if error!=.None {return state.freed,error} + error=blobs_walk(s,"tables",sweep_entry,&state) + if error!=.None {return state.result,error} + buffer:[PATH_BYTES]u8 tmp:=join(&buffer,s.root,"tmp") or_return entries,read_error:=os.read_all_directory_by_path(tmp,s.allocator) - if read_error!=nil {return state.freed,os_error(read_error)} + if read_error!=nil {return state.result,os_error(read_error)} defer os.file_info_slice_delete(entries,s.allocator) - now:=time.now() for entry in entries { - if time.diff(entry.modification_time,now)>TMP_STALE {base.remove_all(entry.fullpath)} + if !older_than(entry.modification_time,at,TMP_STALE) {continue} + if entry.type==.Directory && strings.has_prefix(entry.name,SWEEP_PREFIX) { + claimed,path_error:=join(&buffer,entry.fullpath,"file") + if path_error!=.None {continue} + modified,stat_error:=regular_time(s,claimed) + if stat_error==.None && older_than(modified,at,BLOB_GRACE_AGE) {os.remove(claimed)} + os.remove(entry.fullpath) + } else { + base.remove_all(entry.fullpath) + } + } + return state.result,.None +} + +@(private) +live_add :: proc(set:^Live_Set,hashes:[]Hash) -> Error { + for hash in hashes { + if _,error:=map_insert(set,hash,{});error!=nil {return .Out_Of_Memory} + } + return .None +} + +@(private) +sweep_entry :: proc(user:rawptr,file:os.File_Info,hash:Hash) { + state:=(^Sweep)(user) + if hash in state.live || !older_than(file.modification_time,state.now,BLOB_GRACE_AGE) {return} + if state.tables && has(state.s,hash) {return} + buffer:[PATH_BYTES]u8 + claimed,error:=sweep_claim(state.s,file.fullpath,&buffer) + if error!=.None { + if !state.tables {state.result.skipped+=1} + return } - return state.freed,.None + defer os.remove(os.dir(claimed)) + keep:=state.tables && has(state.s,hash) + removed,finish_error:=sweep_finish(state.s,file.fullpath,claimed,state.now,keep) + if state.tables {return} + if finish_error!=.None {state.result.skipped+=1} + if removed { + state.result.bytes+=file.size + state.result.blobs+=1 + } +} + +@(private) +older_than :: proc(at,now:time.Time,age:time.Duration) -> bool { + return at._nsecu64(age) +} + +@(private) +regular_time :: proc(s:^Store,path:string) -> (time.Time,Error) { + info,error:=os.lstat(path,s.allocator) + if error!=nil {return {},os_error(error)} + defer os.file_info_delete(info,s.allocator) + if info.type!=.Regular {return {},.Io} + return info.modification_time,.None +} + +// The directory dates the claim independently of the file moved into it. +@(private) +sweep_claim :: proc(s:^Store,path:string,claim_buffer:^[PATH_BYTES]u8) -> + (claimed:string,error:Error) { + folder_buffer:[PATH_BYTES]u8 + folder:=temp_folder(s.root,SWEEP_PREFIX,&folder_buffer) or_return + claimed,error=join(claim_buffer,folder,"file") + if error==.None {error=os_error(os.rename(path,claimed))} + if error!=.None {os.remove(folder);return "",error} + return claimed,.None +} + +@(private) +sweep_finish :: proc(s:^Store,path,claimed:string,now:time.Time,keep:=false) -> (bool,Error) { + at,error:=regular_time(s,claimed) + if error==.None && !keep && older_than(at,now,BLOB_GRACE_AGE) { + error=os_error(os.remove(claimed)) + if error==.None {return true,.None} + } + restored:=sweep_restore(path,claimed) + return false,error if error!=.None else restored +} + +@(private) +sweep_restore :: proc(path,claimed:string) -> Error { + exists:=rename_new(claimed,path) or_return + if exists {return os_error(os.remove(claimed))} + return .None } -// Visits every file of // named by an address: the complete -// blobs, or the kept tables. +// Visits the regular files named by an address in blobs/ or tables/. @(private) -blobs_walk :: proc(s:^Store,kind:string,visit:proc(user:rawptr,path:string,hash:Hash,size:i64) -> Error,user:rawptr) -> Error { +blobs_walk :: proc(s:^Store,kind:string, + visit:proc(user:rawptr,file:os.File_Info,hash:Hash),user:rawptr) -> Error { buffer:[PATH_BYTES]u8 blobs:=join(&buffer,s.root,kind) or_return folders,folders_error:=os.read_all_directory_by_path(blobs,s.allocator) @@ -91,7 +172,7 @@ blobs_walk :: proc(s:^Store,kind:string,visit:proc(user:rawptr,path:string,hash: for file in files { hash,ok:=parse_hex(file.name) if !ok || file.type!=.Regular || file.name[:2]!=folder.name {continue} - visit(user,file.fullpath,hash,file.size) or_return + visit(user,file,hash) } } return .None diff --git a/store/store.odin b/store/store.odin index 7e7760179..37b3b4d9e 100644 --- a/store/store.odin +++ b/store/store.odin @@ -3,6 +3,7 @@ package store import "base:runtime" import "core:crypto" import "core:os" +import "core:time" // A content-addressed store of the machine: blobs named by their address and // refs, mutable names of an address. Several Store values and processes share @@ -42,7 +43,7 @@ open :: proc(root:string,allocator:=context.allocator) -> (s:Store,error:Error) } buffer:[PATH_BYTES]u8 folder_make(trimmed) or_return - for folder in ([5]string{"blobs","tables","partial","refs","tmp"}) { + for folder in ([6]string{"blobs","tables","partial","refs","leases","tmp"}) { folder_make(join(&buffer,trimmed,folder) or_return) or_return } layout_check(trimmed) or_return @@ -88,7 +89,7 @@ layout_check :: proc(root:string) -> Error { return .None } if open_error!=.Not_Exist {return .Io} - return file_write(root,path,transmute([]u8)string(LAYOUT)) + return file_write(root,path,{transmute([]u8)string(LAYOUT)}) } // Copies parts joined by '/' into buffer. @@ -131,18 +132,28 @@ os_error :: proc(error:os.Error) -> Error { // Writes chunks to a file of the root's tmp folder and renames it to path, // which then holds all of it or what it held before. @(private) -file_write :: proc(root:string,path:string,chunks:..[]u8) -> Error { +file_write :: proc(root,path:string,chunks:[][]u8,modified:Maybe(time.Time)=nil) -> Error { temp_buffer:[PATH_BYTES]u8 temp,file:=temp_create(root,&temp_buffer) or_return - write_error:os.Error + write_error:Error for chunk in chunks { - if write_error==nil {_,write_error=os.write(file,chunk)} + if write_error==.None { + n,error:=os.write(file,chunk) + write_error=os_error(error) + if error==nil && n!=len(chunk) {write_error=.Io} + } } sync_error:=os.sync(file) os.close(file) - if write_error!=nil || sync_error!=nil { + if write_error!=.None || sync_error!=nil { os.remove(temp) - return os_error(write_error) if write_error!=nil else .Io + return write_error if write_error!=.None else .Io + } + if at,ok:=modified.?;ok { + if time_error:=os.change_times(temp,at,at);time_error!=nil { + os.remove(temp) + return os_error(time_error) + } } if os.rename(temp,path)!=nil { os.remove(temp) diff --git a/store/store_test.odin b/store/store_test.odin index 5a3766ae6..410c1f890 100644 --- a/store/store_test.odin +++ b/store/store_test.odin @@ -3,6 +3,7 @@ package store import "core:os" import "core:strings" import "core:testing" +import "core:time" import "mirai:std/base" import "mirai:std/blake3" @@ -185,34 +186,58 @@ receiving_resumes_after_reopening_and_reads_the_pieces_it_has :: proc(t:^testing } @(test) -sweep_keeps_exactly_the_live_blobs :: proc(t:^testing.T) { +sweep_keeps_live_young_and_undeletable_blobs :: proc(t:^testing.T) { folder:=base.test_folder(t,"store-sweep");if folder=="" {return} s,ok:=test_store(t,folder);if !ok {return} defer close(&s) - hashes:[5]Hash + hashes:[7]Hash + now:=time.now() + old:=time.time_add(now,-2*BLOB_GRACE_AGE) + young:=time.time_add(now,-BLOB_GRACE_AGE/2) bytes:i64 for &hash,i in hashes { data:=test_bytes(1000*(i+1),u64(i)+20) hash,_=put_bytes(&s,data) bytes+=i64(len(data)) + test_blob_time(t,&s,hash,young if i==0 else old) + test_table_time(t,&s,hash,young if i==6 else old) } partial:=test_bytes(PIECE_BYTES+1,30) partial_hash:=test_address(partial) testing.expect_value(t,receive_begin(&s,partial_hash,i64(len(partial)),test_table(partial)),Error.None) total,_:=usage(&s) - testing.expect_value(t,total,Usage{bytes=bytes,blobs=5}) + testing.expect_value(t,total,Usage{bytes=bytes,blobs=len(hashes)}) + testing.expect_value(t,remove(&s,hashes[5]),Error.None) + table_keep(&s,hashes[5],6000,test_table(test_bytes(6000,25))) + test_table_time(t,&s,hashes[5],old) stale,fresh:=join_test(folder,"tmp/stale",0),join_test(folder,"tmp/fresh",0) testing.expect(t,os.write_entire_file(stale,[]u8{1})==nil && os.write_entire_file(fresh,[]u8{1})==nil) - old:=os.modification_time_by_path(stale) or_else {} - testing.expect(t,os.change_times(stale,old,{_nsec=old._nsec-2*i64(TMP_STALE)})==nil) + test_set_time(t,stale,time.time_add(now,-2*TMP_STALE)) unknown:Hash unknown[0]=0xee - freed,error:=sweep(&s,{hashes[3],unknown,hashes[1]}) + path,path_error:=blob_path(&s,hashes[2],context.temp_allocator) + if !testing.expect_value(t,path_error,Error.None) {return} + when ODIN_OS==.Windows { + reader,reader_error:=reader_open(&s,hashes[2]) + if !testing.expect_value(t,reader_error,Error.None) {return} + defer reader_close(&reader) + } else { + shard:=os.dir(path) + if !testing.expect(t,os.change_mode(shard,os.perm_number(0o555))==nil) {return} + defer os.change_mode(shard,os.perm_number(0o755)) + } + swept,error:=sweep(&s,{hashes[3],unknown,hashes[1]},now) testing.expect_value(t,error,Error.None) - testing.expect_value(t,freed,Usage{bytes=1000+3000+5000,blobs=3}) - for hash,i in hashes {testing.expectf(t,has(&s,hash)==(i==1 || i==3),"blob %d",i)} + testing.expect_value(t,swept.freed,Usage{bytes=5000+7000,blobs=2}) + testing.expect_value(t,swept.skipped,1) + for hash,i in hashes { + testing.expectf(t,has(&s,hash)==(i<4),"blob %d",i) + buffer:[PATH_BYTES]u8 + table,_:=table_file(&s,hash,&buffer) + testing.expectf(t,os.is_file(table)==(i<4 || i==6),"table %d",i) + } total,_=usage(&s) - testing.expect_value(t,total,Usage{bytes=2000+4000,blobs=2}) + testing.expect_value(t,total,Usage{bytes=1000+2000+3000+4000,blobs=4}) _,missing_error:=receive_missing(&s,partial_hash,context.temp_allocator) testing.expect_value(t,missing_error,Error.None) testing.expect(t,!os.exists(stale) && os.exists(fresh),"sweep removes only stale temporary files") @@ -223,6 +248,25 @@ sweep_keeps_exactly_the_live_blobs :: proc(t:^testing.T) { testing.expect_value(t,missing_error,Error.Missing) } +@(private) +test_set_time :: proc(t:^testing.T,path:string,at:time.Time,loc:=#caller_location) { + testing.expect(t,os.change_times(path,at,at)==nil,loc=loc) +} + +@(private) +test_blob_time :: proc(t:^testing.T,s:^Store,hash:Hash,at:time.Time,loc:=#caller_location) { + buffer:[PATH_BYTES]u8 + path,_:=blob_file(s,hash,&buffer) + test_set_time(t,path,at,loc) +} + +@(private="file") +test_table_time :: proc(t:^testing.T,s:^Store,hash:Hash,at:time.Time) { + buffer:[PATH_BYTES]u8 + path,_:=table_file(s,hash,&buffer) + test_set_time(t,path,at) +} + // Receives every piece but piece 0, whose bytes and mark are then written as // a receiver killed before completing would leave them. @(private) @@ -272,13 +316,15 @@ completion_publishes_only_the_bytes_it_checked :: proc(t:^testing.T) { data:=test_bytes(3*PIECE_BYTES+3,60) hash:=test_full_unfinished(t,&s,data) c:Claim - testing.expect_value(t,partial_claim(&s,hash,&c),Error.None) + now:=time.now() + testing.expect_value(t,partial_claim(&s,hash,&c,now),Error.None) complete,error:=claim_verify(&s,hash,&c) testing.expect(t,error==.None && complete) // Meanwhile another receiver completes it, a sweep removes it and a third begins again. _,put_error:=put_bytes(&s,data) testing.expect_value(t,put_error,Error.None) - _,sweep_error:=sweep(&s,nil) + test_blob_time(t,&s,hash,time.time_add(now,-2*BLOB_GRACE_AGE)) + _,sweep_error:=sweep(&s,nil,now) testing.expect_value(t,sweep_error,Error.None) testing.expect_value(t,receive_begin(&s,hash,i64(len(data)),test_table(data)),Error.None) testing.expect_value(t,receive_piece(&s,hash,2,test_piece(data,2)),Error.None) diff --git a/store/sweep_test.odin b/store/sweep_test.odin new file mode 100644 index 000000000..478765bca --- /dev/null +++ b/store/sweep_test.odin @@ -0,0 +1,148 @@ +package store + +import "core:os" +import "core:testing" +import "core:time" +import "mirai:std/base" + +@(test) +existing_blobs_are_refreshed_only_after_a_quarter_of_grace :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-refresh");if folder=="" {return} + s,ok:=test_store(t,folder);if !ok {return} + defer close(&s) + now:=time.unix(time.time_to_unix(time.now()),0) + Put :: enum {Bytes,File,Begin,Complete} + for route,i in Put { + for age,j in ([2]time.Duration{2*BLOB_GRACE_AGE,BLOB_REFRESH_AGE/2}) { + data:=test_bytes(PIECE_BYTES+1,u64(100+2*i+j)) + hash:=test_address(data) + if route==.Complete {test_full_unfinished(t,&s,data)} + _,put_error:=put_bytes(&s,data,now) + if !testing.expect_value(t,put_error,Error.None) {return} + before:=time.time_add(now,-age) + test_blob_time(t,&s,hash,before) + error:Error + switch route { + case .Bytes: _,error=put_bytes(&s,data,now) + case .File: + source:=join_test(folder,"source",0) + if !testing.expect(t,os.write_entire_file(source,data)==nil) {return} + _,error=put_file(&s,source,now) + case .Begin: + error=receive_begin(&s,hash,i64(len(data)),test_table(data),now) + case .Complete: + error=receive_piece(&s,hash,0,test_piece(data,0),now) + } + testing.expect_value(t,error,Error.None) + path:=[PATH_BYTES]u8{} + name,_:=blob_file(&s,hash,&path) + changed,time_error:=os.modification_time_by_path(name) + testing.expect(t,time_error==nil) + expected:=now if j==0 else before + testing.expectf(t,changed==expected,"%v with age %v",route,age) + } + } + _,sweep_error:=sweep(&s,nil,now) + testing.expect_value(t,sweep_error,Error.None) + total,_:=usage(&s) + testing.expect_value(t,total.blobs,8) +} + +@(test) +sweep_and_put_resolve_both_claim_interleavings :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-claim-put");if folder=="" {return} + s,ok:=test_store(t,folder);if !ok {return} + defer close(&s) + other,ok_other:=test_store(t,folder);if !ok_other {return} + defer close(&other) + now:=time.now() + old:=time.time_add(now,-2*BLOB_GRACE_AGE) + for put_first,i in ([2]bool{true,false}) { + data:=test_bytes(123,u64(160+i)) + hash,error:=put_bytes(&s,data,now) + if !testing.expect_value(t,error,Error.None) {return} + test_blob_time(t,&s,hash,old) + path,claim_buffer:[PATH_BYTES]u8 + blob,_:=blob_file(&s,hash,&path) + entry,stat_error:=os.stat(blob,s.allocator) + if !testing.expect(t,stat_error==nil) {return} + defer os.file_info_delete(entry,s.allocator) + testing.expect(t,older_than(entry.modification_time,now,BLOB_GRACE_AGE)) + if put_first { + _,error=put_bytes(&other,data,now) + testing.expect_value(t,error,Error.None) + state:=Sweep{s=&s,now=now} + sweep_entry(&state,entry,hash) + testing.expect_value(t,state.result.blobs,0) + testing.expect_value(t,state.result.skipped,0) + } else { + claimed,claim_error:=sweep_claim(&s,blob,&claim_buffer) + if !testing.expect_value(t,claim_error,Error.None) {return} + defer os.remove(os.dir(claimed)) + testing.expect(t,!has(&other,hash)) + refreshed:=blob_touch(&other,blob,now) + testing.expect_value(t,refreshed,Error.Missing) + error=put_bytes_after_refresh(&other,data,test_table(data),hash,refreshed,now) + testing.expect_value(t,error,Error.None) + removed,finish_error:=sweep_finish(&s,blob,claimed,now) + testing.expect_value(t,finish_error,Error.None) + testing.expect(t,removed) + } + testing.expect_value(t,verify(&s,hash),Error.None) + } +} + +@(test) +fresh_leases_keep_their_current_addresses :: proc(t:^testing.T) { + folder:=base.test_folder(t,"store-leases");if folder=="" {return} + s,ok:=test_store(t,folder);if !ok {return} + defer close(&s) + other,ok_other:=test_store(t,folder);if !ok_other {return} + defer close(&other) + now:=time.now() + old:=time.time_add(now,-2*BLOB_GRACE_AGE) + stale:=time.time_add(now,-2*LEASE_MAX_AGE) + hashes:[4]Hash + for &hash,i in hashes { + hash,_=put_bytes(&s,test_bytes(123,u64(180+i)),now) + test_blob_time(t,&s,hash,old) + } + fresh,open_error:=lease_open(&s,"editor",{hashes[0]},now) + if !testing.expect_value(t,open_error,Error.None) {return} + defer lease_release(&fresh) + updated:=[1]Hash{hashes[1]} + fresh_buffer,claim_buffer:[PATH_BYTES]u8 + fresh_path,_:=lease_path(&fresh,&fresh_buffer) + claimed,claim_error:=sweep_claim(&other,fresh_path,&claim_buffer) + if !testing.expect_value(t,claim_error,Error.None) {return} + defer os.remove(os.dir(claimed)) + testing.expect_value(t,lease_update(&fresh,updated[:],stale),Error.None) + testing.expect_value(t,sweep_restore(fresh_path,claimed),Error.None) + modified,time_error:=regular_time(&other,fresh_path) + testing.expect_value(t,time_error,Error.None) + testing.expect(t,older_than(modified,now,LEASE_MAX_AGE),"the replacement stays in place") + updated[0]=hashes[0] + testing.expect_value(t,lease_refresh(&fresh,now),Error.None) + collected,make_error:=make(Live_Set,1,s.allocator) + if !testing.expect(t,make_error==nil) {return} + defer delete(collected) + testing.expect_value(t,lease_collect_stale(&other,&collected,fresh_path,now),Error.None) + testing.expect(t,hashes[1] in collected && !(hashes[0] in collected)) + expired,expired_error:=lease_open(&s,"expired",{hashes[2]},stale) + if !testing.expect_value(t,expired_error,Error.None) {return} + defer lease_release(&expired) + released,released_error:=lease_open(&s,"released",{hashes[3]},now) + if !testing.expect_value(t,released_error,Error.None) {return} + testing.expect_value(t,lease_release(&released),Error.None) + + swept,sweep_error:=sweep(&other,nil,now) + testing.expect_value(t,sweep_error,Error.None) + testing.expect_value(t,swept.blobs,3) + for hash,i in hashes {testing.expectf(t,has(&s,hash)==(i==1),"blob %d",i)} + + buffer:[PATH_BYTES]u8 + path,_:=lease_path(&expired,&buffer) + testing.expect(t,!os.exists(path)) + testing.expect_value(t,lease_refresh(&expired,now),Error.None) + testing.expect(t,os.is_file(path)) +} From 23afa4e1bc50a5a895916f3017da40baa2aeec7a Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 19:16:44 -0300 Subject: [PATCH 015/123] editor: the session owns the path of its program strings.replace_all gives back its input when nothing changes, so the session freed the temp allocator's path on close and the editor aborted. Co-Authored-By: Claude Opus 5.5 --- editor/editor.odin | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/editor/editor.odin b/editor/editor.odin index ba22f6c88..18377d941 100644 --- a/editor/editor.odin +++ b/editor/editor.odin @@ -443,7 +443,9 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { s.project_root=strings.clone(root) s.engine_root=strings.clone(engine) program,_:=os.get_executable_path(context.temp_allocator) - s.program,_=strings.replace_all(program,"\\","/") + // replace_all gives back its input when nothing changes: the clone owns it either way. + forward,_:=strings.replace_all(program,"\\","/",context.temp_allocator) + s.program=strings.clone(forward) s.engine_stamp=engine_stamp_here(s) s.library_stamp=library_stamp_here(s) s.assets_root=strings.clone(assets) From a4c3f7e35ee156be2adba03f9ddcf57885479261 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 19:28:06 -0300 Subject: [PATCH 016/123] import: the sync of a project becomes a machine library of packages cook/import/sources/library.odin replaces sync.odin. ingest takes a source from anywhere into the store as a package, finds an origin ingested before by the ref origins/ and keeps its ids, skips one whose identity and build are current, and takes turns with a lock per origin. project_add and project_ingest pin a package in packages.mirai (project_pin, its only writer) and follow its documents and, for a model whose source is here, its prefab. project_refresh takes each pinned package to a build this engine reads: local refs, the caller's cache.Source, an ingest of an origin on this machine; else it reports by package name, also when only Fast is here and Final is required. It never removes and never builds without the source. Source hashes are kept per origin in the store (sources/). mirai import ingests and adds to a project (--project, --folder); mirai import --refresh replaces --sync, and the build calls it (imports_refresh). The identity of a package includes its id, so two packages of the same bytes no longer share a build key; an explicit parts setting decides the owner of documents again; cache_key_build hashes empty bytes without asserting. Co-Authored-By: Claude Opus 5.5 --- .../skills/mirai-assets/references/formats.md | 4 +- .../skills/mirai-editor/references/panels.md | 2 +- .agents/skills/mirai-imports/SKILL.md | 73 ++- .../mirai-imports/references/imports.md | 129 ++-- .../skills/mirai-projects/references/build.md | 6 +- .../mirai-projects/references/projects.md | 2 + asset/packages.odin | 8 + cook/build/build.odin | 2 +- cook/build/imports.odin | 8 +- cook/import/documents.odin | 10 +- cook/import/identity.odin | 6 +- cook/import/model.odin | 1 + cook/import/outputs_test.odin | 4 +- cook/import/package.odin | 133 +++- cook/import/reader_test.odin | 4 +- cook/import/sources/bench_test.odin | 6 +- cook/import/sources/cache_test.odin | 96 +-- cook/import/sources/cli_test.odin | 106 +-- cook/import/sources/cluster_test.odin | 2 +- cook/import/sources/command.odin | 189 ++++-- cook/import/sources/cook_test.odin | 55 +- cook/import/sources/fuzz_test.odin | 55 +- cook/import/sources/library.odin | 614 ++++++++++++++++++ cook/import/sources/library_test.odin | 343 ++++++++++ cook/import/sources/reimport_test.odin | 402 ++++-------- cook/import/sources/settings_test.odin | 113 ++-- cook/import/sources/support_test.odin | 76 ++- cook/import/sources/sync.odin | 247 ------- cook/import/sources/write_test.odin | 104 +-- cook/textures/cache.odin | 3 +- tools/demo_content/main.odin | 2 +- tools/mirai/main.odin | 2 +- tools/mirai/package.odin | 2 +- 33 files changed, 1815 insertions(+), 994 deletions(-) create mode 100644 cook/import/sources/library.odin create mode 100644 cook/import/sources/library_test.odin delete mode 100644 cook/import/sources/sync.odin diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index 913f9c859..61b8831ab 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -180,8 +180,8 @@ One rule for all types, of the engine and of plugins: plugin's type, in the `version` of its `asset_kinds`, which the build generates as `_VERSION` ([manifests](../../mirai-projects/references/projects.md#manifests)). - **A change in what a type stores or in what a stored value means raises its version**, and every - file of the type is made again: the [sync](../../mirai-imports/references/imports.md#sync) of - the next build imports again the source of an imported one. `version_message` says which asset + file of the type is made again: the [refresh](../../mirai-imports/references/imports.md#library) + of the next build ingests again the source of an imported one. `version_message` says which asset it is, its type, its version and the one the build reads and, if it was imported, the `mirai import` that rebuilds it in its folder with the root where its source is; for a newer file, a newer engine. `mirai asset verify` diff --git a/.agents/skills/mirai-editor/references/panels.md b/.agents/skills/mirai-editor/references/panels.md index 09758db3e..098c68e3e 100644 --- a/.agents/skills/mirai-editor/references/panels.md +++ b/.agents/skills/mirai-editor/references/panels.md @@ -854,7 +854,7 @@ is refused. The source is versioned with its sidecar beside what it makes, and w that nobody edits goes to the imported folder ([identity](../../mirai-imports/references/imports.md#identity-and-reimport)). The imported folder is brought up to date with the sources by the build that opens the editor -([sync](../../mirai-imports/references/imports.md#sync)). +([refresh](../../mirai-imports/references/imports.md#library)). File > Import, the content browser, a file dropped on the window and `asset.import` call it. It imports in a thread of its own, with its own job system, while the editor keeps drawing; the content browser shows the dot of what works. `import_poll`, between frames, takes what it wrote on diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index 6a5eeb2e0..7b23b6a45 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -25,8 +25,9 @@ the assets. [Identity](references/imports.md#identity-and-reimport). - The records of the store (package and build manifests, refs, `packages.mirai`): [packages](../mirai-assets/references/formats.md#packages). -- A checkout without its imported folder, what a build imports and when a source is imported - again: [Sync](references/imports.md#sync). +- Ingesting a source from anywhere, adding it to a project (`packages.mirai`), what a build + refreshes, a peer without sources, and when a source is imported again: + [Library](references/imports.md#library). - Adding or changing a format (`Format`, `Reader`, `Source_Space`, version): [Formats](references/imports.md#formats). - Import cache and versions: [Cache](references/imports.md#import-cache), @@ -58,8 +59,10 @@ the assets. 1. **Importing a file.** An import takes the store (`Import_Options.store`) and puts in it the package of the source and a build of its blobs; given `documents`, it also writes or follows the documents in that folder of a project. Review the assets with `mirai asset verify`, `ls` - and `info`; commit the documents, never the blobs. The editor, the CLI, the sync of builds and - the content tools import with the same `sources.import_source`: same assets, prefab included. + and `info`; commit `packages.mirai` and the documents, never the blobs. The editor, the CLI, + the library and the content tools import with the same `sources.import_source`: same assets, + prefab included. `mirai import ` ingests it and adds it to a project + ([library](references/imports.md#library)). 2. **From code.** `sources.import_source(path,options)` writes everything; for the model without writing it, `importer.import_model(path,sources.FORMATS[:],options)` reads it into an `Imported_Model` (plain data, no engine components) and `importer.import_write` writes its @@ -75,8 +78,10 @@ the assets. model format) or an `Import_Source_Provider` (a source that imports whole); the editor and the import program of the engine add its formats to `sources.FORMATS`, which holds only the engine's own ([plugins](references/imports.md#plugin-formats)). -6. **Regenerating imported assets.** `mirai build` builds what is missing or stale of a project - and its plugins once its target stands ([sync](references/imports.md#sync)). +6. **Regenerating imported assets.** `mirai build` refreshes the packages a project and its + plugins pin once its target stands (`mirai import --refresh `): builds from the store, + from a peer, or by ingesting a source this machine has + ([library](references/imports.md#library)). ## Invariants @@ -91,6 +96,10 @@ the assets. place after the read: from the `front` its settings state, or else, for a file that is a rig, from where its body is found facing ([front](references/imports.md#front)). - **The source is not modified.** The same bytes and options give the same assets. +- **The library never removes and never builds without the source.** A refresh takes a build + from the store, the caller's source or an ingest of an origin on this machine; else it names + the package and what gives it. Nothing is deleted because a source is missing. Decided by the + owner. - **A source imports from anywhere.** In a project, in a plugin or in no project: what it makes goes to the store, and its documents to the folder of the project given. In a project, it lives beside what it makes, in folders by theme, never a folder per asset type. @@ -110,7 +119,8 @@ the assets. the editor edits, are also documents of a project, followed from the baseline. Decided by the owner. - **The identity holds no engine version; the key holds the identity and the versions.** The - identity (source, inputs, settings, plugin code) is in the manifest; the key + identity (package, source, inputs, settings, plugin code: two packages of the same bytes never + share a build, since its blobs hold their ids) is in the manifest; the key (`asset.import_key`) adds the writer, format, kind and texture builder versions, so any machine with the manifest finds a build without the source. A Final build is named under `imports`, which peers share; a Fast one only under `imports-fast`; `builds` names the one this machine @@ -174,10 +184,10 @@ the assets. - **Textures in blocks are encoded with a perceptual metric and RDO** at `RDO_LAMBDA_DEFAULT` unless the setting `rdo_lambda` says otherwise. - **The editor never waits for the final encode** (`textures.Encode_Speed`): its imports take the - Final of the texture cache, else build Fast (no RDO, low effort, `Texture_Flag.Fast`). The sync - of a build that does not cook (the editor's) builds Fast too and takes Fast or Final as current; - only the sync of a build that cooks, or of `mirai package engine`, builds Final. `mirai import` - and tools build Final, and so does the editor for a texture that is a document. + Final of the texture cache, else build Fast (no RDO, low effort, `Texture_Flag.Fast`). The + refresh of a build that does not cook (the editor's) builds Fast too and takes Fast or Final; + only the refresh of a build that cooks, or of `mirai package engine`, requires Final. `mirai + import` and tools build Final, and so does the editor for a texture that is a document. - **One algorithm, once.** Mips, color rules and block compression in `cook/textures` (the encoders are `compressor:blocks`); PBR conversions in `cook/import/pbr.odin`; triangulation, simplification, hulls and convex decomposition in `std/geom`; transforms with `std/math`. @@ -202,7 +212,7 @@ the assets. key records what it is built from now is taken without writing it, and one that records other encoders or another builder is built again); `cook/build/cook_cache_test.odin` (a cook takes the blocks of a texture from the cache of its - content) and `cook/import/sources/cook_test.odin` (the sync of a cook builds a fast texture + content) and `cook/import/sources/cook_test.odin` (the refresh of a cook builds a fast texture final from its source; that of a build that does not cook keeps either). - Parallel: import a map with `mirai import --rebuild` over a copy of an import made without the job system (or before the change) and compare the trees with `diff -r`: the same bytes. @@ -214,24 +224,25 @@ the assets. `prompts_kenney` pack again ([command](../mirai-plugins/references/plugins.md#first-party-plugins)) and verify it with `mirai asset verify`. -- Identity, the imported folder and sync: `cook/import/sources/reimport_test.odin` (the parts of - an import name their prefab and keep their ids where they were moved, a source keeps the ids of - its import with its sidecar, a part renamed in the source is a new one, two models with the same - image get a texture each and neither touches the other, a checkout without its imported folder - gets the same assets byte for byte and only what changed is imported again, the sources of a - plugin format build in the cache and sync reads them with the formats of the plugins, a source - outside the assets is refused, when a material is a part of a model, sync runs the importer of - a plugin for a whole source whose inputs changed), `cook/import/outputs_test.odin` (the outputs - of a source are the assets its package lists, the models of a whole source share their parts by - name, inputs count in the identity, importing again keeps ids and blobs, a source outside any - project imports, a setting changes the identity and an engine version only the key, a Fast - build is not shared, two stores hold the same blobs), - `asset/registry_test.odin` (the listing of the cache by where the sidecar is, never by the order - of a scan), `cook/import/sources/settings_test.odin` (a sidecar that is not one is refused) and - `cook/import/sources/write_test.odin` (importing again keeps what was changed in a material and - takes the rest from the source). After changing what an import writes, prove it on the demo: - delete `demo/.mirai/imported`, `mirai build editor demo`, and `git status` shows no sidecar - or document changed. +- Identity: `cook/import/sources/reimport_test.odin` (the parts of an import name their prefab and + keep their ids where they were moved, a source keeps the ids of its package with the manifest a + project pins and a copy is another package, a part renamed in the source is a new one, two + models with the same image get a texture each and neither touches the other, when a material is + a part of a model), `cook/import/outputs_test.odin` (the outputs of a source are the assets its + package lists, the models of a whole source share their parts by name, inputs count in the + identity, importing again keeps ids and blobs, a source outside any project imports, a setting + changes the identity and an engine version only the key, a Fast build is not shared, two stores + hold the same blobs), `cook/import/sources/settings_test.odin` (the package keeps the settings; + settings that are not this build's are refused) and `cook/import/sources/write_test.odin` + (importing again keeps what was changed in a material and takes the rest from the source). +- Library: `cook/import/sources/library_test.odin` (a peer without sources fetches, lists and + loads what a project pins, and without a source of builds reports by package and neither + removes nor builds; a source outside any project ingests once and two projects list the same + blobs; an engine change ingests again where the source is and is reported by name elsewhere, + also when only Fast is here and Final is required; hostile `packages.mirai` and manifests are + reported; two ingests of one source at once import it once) and + `cook/import/sources/reimport_test.odin` (a refresh reads a source again with the formats of + the plugins, and ingests a whole source again when what it is built from changes). - `cook/import/sources/cli_test.odin` needs `MIRAI_ROOT` (defined by `mirai test`); run it when touching `cook/import/sources/command.odin`. It imports in-process with `command`; the whole path (`mirai import` → `mirai-import`) needs the built engine. @@ -246,7 +257,7 @@ the assets. rejected. The binding is regenerated with `uv run scripts/ufbx_binding.py` ([ufbx](../../../third_party/ufbx/UPSTREAM.md)). - `fuzz_test.odin` truncates each fixture to all its lengths and changes bytes (OBJ, glTF, GLB, - USDA, binary and ASCII FBX, WAV, Ogg, PNG and the `.import` text): each attempt is imported or + USDA, binary and ASCII FBX, WAV, Ogg, PNG, fonts and prompts): each attempt is imported or rejected, without crashes, hangs or leaks. A new format extends those tests. - A change that must give the same bytes is compared by importing with the CLI before and after (`--rebuild` on the second pass, or in new folders) and with `cmp` of each asset. diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index da1eb4c77..02f900cc1 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -10,8 +10,8 @@ Packages, all *engine-time*: one format each, with its `FORMAT`. They import the core and nothing from the other formats. - `cook/import/sources`: the `FORMATS` table (the engine's formats), `source_kind` and `import_source`, used by the editor (`editor/import.odin`), `mirai import` (`command.odin`, in the - engine's `mirai-import` program) and `sources.sync`. The editor adds the formats of plugins - ([plugin formats](#plugin-formats)). + engine's `mirai-import` program) and the [library](#library) (`ingest`, `project_add`, + `project_refresh`). The editor adds the formats of plugins ([plugin formats](#plugin-formats)). - `cook/textures`: images to `.tex` levels. ## API @@ -19,8 +19,8 @@ Packages, all *engine-time*: - `sources.import_source` imports a file of the assets folder of a project or a plugin: a model writes its assets and, if it has nodes, its prefab; an image, its `.tex`; a sound, a font or a prompt pack, its asset. The [package](#identity-and-reimport) of the source in the store keeps - its settings and the id of each asset: importing it again gives the same assets. `sources.sync` - imports every source of a root whose outputs are missing or stale ([sync](#sync)). + its settings and the id of each asset: importing it again gives the same assets. The + [library](#library) ingests sources, pins them in projects and refreshes their builds. - `sources.source_kind` says what a file is from its first bytes ([sniffing](#formats)), except what the signature does not settle: TGA by extension and prompts by `.prompts.json`. It carries the formats it searches with (`formats`, `FORMATS` by default), like `model_folder` @@ -156,8 +156,8 @@ part ([extension](../../mirai-editor/references/extensions.md#importers-of-a-plu dialog. Its `prepare` gives the format's options for an import (`Format.data`) from the assets of the project and the library of its plugins, and `release` frees them when it ends. - The editor imports with the providers of its library; the import program of the engine - (`mirai import`, the [sync](#sync) of every build) loads the editor library of the project and - takes the same providers from it (`editor.import_library`, `sources.Library`), only when a + (`mirai import`, the [refresh](#library) of every build) loads the editor library of the project + and takes the same providers from it (`editor.import_library`, `sources.Library`), only when a source is read by no format of the engine or a model imports, whose prefab is written in the types of the game the library has. A clone without that library built fails those sources and says to build the editor. @@ -165,9 +165,8 @@ part ([extension](../../mirai-editor/references/extensions.md#importers-of-a-plu is the single source of every asset it makes (a scene, the models it reads with `output_model`), and the files it reads besides it are its `inputs`, like the `.bin` of a glTF. It appears with its name as a filter of the dialog and claims a file with `claims(path, head)` - before the formats. One from outside the assets is brought in by its `bring`, with the files it - reads, else copied alone. The import program takes these importers from the same library and - [sync](#sync) runs them. The Unreal content importer is the `unreal` plugin of the catalog + before the formats. The import program takes these importers from the same library and the + [refresh](#library) runs them. The Unreal content importer is the `unreal` plugin of the catalog ([git plugins and catalog](../../mirai-plugins/references/plugins.md#git-plugins-and-catalog)). ## Import cache @@ -204,12 +203,12 @@ the asset: the [package](#identity-and-reimport) is. An import ingests its source as a package of the store ([packages](../../mirai-assets/references/formats.md#packages)), from wherever the source is: in the assets of a project, of a plugin or in no project. Its manifest is what identifies the -import: the origin (where the source was, a hint), the identity (`identity_of`: the bytes of the -source and its inputs, its settings and the code of a plugin that imports it whole, no engine -version), the settings, the inputs and one asset per thing the source gives, by kind, name and -index, with its id, owner and dependencies. Nothing else decides an id: the first ingest gives -each part a random one; importing again (`Import_Options.manifest`, else the manifest the ref -`packages` of this machine names for the path of the source) gives each part the asset it had. A +import: the origin (where the source was, a hint), the identity (`identity_of`: the package, the +bytes of the source and its inputs, its settings and the code of a plugin that imports it whole, +no engine version), the settings, the inputs and one asset per thing the source gives, by kind, +name and index, with its id, owner and dependencies. Nothing else decides an id: the first ingest +gives each part a random one; importing again (`Import_Options.manifest`, else the manifest the ref +`origins` of this machine names for the path of the source) gives each part the asset it had. A manifest that does not read is an error that names it, never the defaults nor new ids. What an import writes is a blob of its build (`writer_put`), whatever the format that reads the @@ -277,51 +276,57 @@ what surely belongs to that model is a part; what is doubtful stays independent an id of its own: no import writes it ([repeated ids](../../mirai-assets/references/formats.md#registry)). -## Sync - -`sources.sync(root, options)` brings the imported folder of a project or a plugin up to date with -the sources of its assets folder (`cook/import/sources/sync.odin`): it is what makes a checkout -that was just cloned, a worktree that was just made or a branch that changed a source run. - -- It imports each source whose sidecar lists an output of the cache with no file named by the key - of what the import builds from now (`importer.import_key`): the bytes of the source and of every - file it read (the buffers and images of a glTF, the material library of an OBJ, the images of a - pack of prompts), its settings, the versions of the importer and of what it writes and the block - encoders of the program, with textures Final. An import of documents alone (clips of a glTF - without meshes) leaves instead an empty stamp named the same way (`importer.STAMP_EXTENSION`). - A sync with `fast_textures` (`--fast`, what a build that does not cook runs) also takes outputs - built Fast as current and imports with Fast textures; Final stays current there. One notion of - current, the bytes: a source that only moved is not imported, and one whose `.bin` or image - changed is. The hash of a file whose size and modification time are those the cache of the root - keeps for it (`importer.Source_Hashes`, in `.mirai/cache/sources`) is taken from there, so a - build reads only the files that changed. A USD is composed by `usdcat`, which does not say which - layers it read: after changing one alone, import the stage. -- It also imports a source with a document of a version of its kind this build does not read (the - engine's in `asset.KINDS`, a plugin's in the `version` of its `asset_kinds`): the import writes - it again in the current version, with its id and what was kept of it. -- A source no format of the engine reads goes to the importer of a plugin that claims it whole, - else to the formats the plugins add (`Sync_Options.library`); one nothing reads is a failure - that says so. A whole source is keyed like any source of no format (`import_version(0)`) plus - the code of the plugin that claims it (`sources.code_identity` of its folder: its Odin sources - but tests, with the plugins it holds, as `Whole_Source.code` and - `Import_Options.importer_code`), so a change of that code imports it again with no version to - raise; its importer reads the settings its sidecar keeps, as every import does. -- It finds what to import first, then imports it, saying as it goes (`Sync_Options.progress`) a - line per source with why and how many of the sources it is, and under it, every - `importer.PROGRESS_EVERY`, the assets written of those its sidecar listed - (`Import_Options.progress`). `mirai import` passes its output, and the build shows the lines of - the import program as it prints them. -- A file of the imported folder that no sidecar with its source next to it lists under its - current key is removed. A source that does not import is a failure of the report that names it: - the others are imported, and nothing is removed. -- **Who runs it.** Every build of a project once its target stands (`build.job_imports`, with - `imports_sync`): `mirai build` of the editor, the game or the server, `mirai play`, and the - builds the editor starts, with the engine's import program, which reads the formats of plugins - from the editor library just built. `mirai package engine` runs it for the plugins. A plugin - with assets is a root of its own. -- What costs comes from the [texture cache](#texture-cache) and the [mesh cache](#mesh-cache), - which the worktrees of a repository share: a new worktree reads its sources and writes its - imported folder without compressing or simplifying anything. +## Library + +The library of the machine is what its store keeps of imports and what a project does with it +(`cook/import/sources/library.odin`). A project names the packages it uses in its +`packages.mirai` ([packages](../../mirai-assets/references/formats.md#packages)); the store, +shared by every checkout, worktree and project of the machine, keeps their manifests and builds. + +- **`ingest(origin, options, settings)`** imports a file or a folder from wherever it is: in the + assets of a project or a plugin, or in no project. Nothing is copied: the package goes to the + store, named by `packages/` and by its origin (`origins/`, of this machine + alone), so ingesting the same origin again keeps its ids. `settings`, when given, replace those + the package keeps. A source whose identity is that of its package and whose build this engine + reads is not imported again: the report names its package and lists no assets. Two processes + or worktrees that ingest one origin take turns with a lock per origin (`cook/workspace`, under + `/locks/`), and the second finds the work done. +- **`project_add(root, manifest, folder, assets, options)`** pins a package in the project + (`project_pin`, the only writer of `packages.mirai`) and follows it there. **Following** writes + or follows its documents under `folder` of the assets: from the baselines of its build + (`documents_follow`), and for a model whose source is here and whose game types the library + of the project's editor gives, by importing it with that folder, which also writes its prefab + (`prefab.model_prefab`). Without the source or those types a model's prefab cannot be written, + and the report says so. +- **`project_refresh(root, options)`** takes each package the project pins to a build this + engine reads: the key (`asset.import_key` of the identity and the versions of what reads its + origin, an engine importer or a format of the engine or of the plugins) is looked up in + `imports` (and `imports-fast` with `fast_textures`), then asked of the caller's source + (`cache.Source`, the same procedure value the caches of `cook/cache` take: given + `(imports, )` it makes the build and every blob it lists present, given + `(packages, )` the manifest), then built by ingesting the origin when it exists on this + machine. Otherwise the report says, by package name, that a machine with its source must ingest + it with this engine; when only a Fast build is here and Final is required, it says that. It + then names the build under `builds/` and, when that changed, follows the + documents. A source here whose identity changed (its bytes, an input, the code of the plugin + that imports it whole) is ingested again with the ids its manifest gives, and the project pins + the new manifest. +- **It never removes** a blob, a ref or a file of a project because a source is missing, and + never builds without the source. What the store does not have stays listed as not built. +- A source a plugin imports whole has a key only its importer knows: the refresh takes the build + this machine chose for its manifest, and a machine without its source reports it. +- **Source hashes.** The address of each file an identity reads is kept per origin by size and + modification time (`importer.Source_Hashes`, `sources/` in the store), so a + refresh reads only the files that changed. A USD is composed by `usdcat`, which does not say + which layers it read: after changing one alone, ingest the stage. +- **Progress.** `Library_Options.progress` gets a line per package and per source imported, and + the assets written every `importer.PROGRESS_EVERY`. +- **Commands.** `mirai import ` ingests and adds to the project of `--project`, else the + one whose assets hold the file, under `--folder` (else the folder of the file); `mirai import + --refresh ` refreshes a project or a plugin, `--fast` taking Fast. Every build of a + project refreshes it and its plugins once its target stands (`build.job_imports`, + `imports_refresh`) with the engine's import program, which reads the formats of plugins from + the editor library just built. ## In parallel @@ -372,8 +377,8 @@ machine that shares blobs with another finds what that one built. - With `Import_Options.fast_textures` (the editor's imports) a texture of the cache takes the Final the texture cache keeps for its key; without one it is built Fast under the key of Fast settings, and a texture kept Fast is written again as soon as the cache holds its Final, or by the - [sync](#sync) of a build that cooks. `Import_Info.settings_hash` says which of the two the file - holds. + [refresh](#library) of a build that cooks. `Import_Info.settings_hash` says which of the two + the file holds. - [Cooking](../../mirai-projects/references/build.md#asset-cooking) stores there the blocks of an RGBA8 texture by `textures.cache_key_blocks`. - A key names what made the blocks (`textures.encoders_id`), so machines with other encoders diff --git a/.agents/skills/mirai-projects/references/build.md b/.agents/skills/mirai-projects/references/build.md index f19494941..c17edc4c7 100644 --- a/.agents/skills/mirai-projects/references/build.md +++ b/.agents/skills/mirai-projects/references/build.md @@ -229,10 +229,10 @@ Settings > Input changed, and SDL3, which the executable links. The meshes, textures, sounds and fonts an import builds are not versioned: they are in `.mirai/imported/` of the project and of each plugin with assets, and are built again from the -sources and their sidecars ([sync](../../mirai-imports/references/imports.md#sync)). +sources and their sidecars ([library](../../mirai-imports/references/imports.md#library)). -- **A build imports, and says so.** Every build of a project runs the sync of the project and of - its active plugins once its target stands (`job_imports`, `build.imports_sync`): the editor, +- **A build imports, and says so.** Every build of a project runs the refresh of the project and of + its active plugins once its target stands (`job_imports`, `build.imports_refresh`): the editor, the game, the server, `mirai play` and the builds the editor starts. It uses the engine's import program, since the tool builds without the importers, which reads the formats the plugins of the project add from its editor library; the game, compiled from the engine's sources, diff --git a/.agents/skills/mirai-projects/references/projects.md b/.agents/skills/mirai-projects/references/projects.md index c449748bd..5f09746ff 100644 --- a/.agents/skills/mirai-projects/references/projects.md +++ b/.agents/skills/mirai-projects/references/projects.md @@ -508,6 +508,8 @@ by its absolute path, which hands it to Git Bash ([launcher](#tool-and-machine)) | Build only the engine | `mirai build engine [--release] [--force]`|--release] [--force]` | | Generate the world without building | `mirai generate [project]` | | Run the content tool of a project | `mirai content [project] [-- arguments]` | +| Ingest a source from anywhere and add it to a project | `mirai import [--project P] [--folder F] [settings]` | +| Bring the packages a project pins to builds of this engine | `mirai import --refresh [--fast]` | | Documents of interfaces and scenes, no editor | `mirai ui\|scene read\|write\|check ... [--project P]` | | Git plugins of a project | `mirai plugin add\|update\|remove\|sync\|list\|catalog\|check ...` | | Check types | `mirai check [ [-flag...]]`: a package relative to the working folder or to the engine, or absolute; none inside a project: its game, editor and plugins. In a runtime package it also fails on an allocation whose error nothing receives (`tools/mirai/alloc_lint.odin` says which calls and which packages), and in the runtime or editor part of a plugin on an import of `render` or `rhi` outside its tests or of the physics solver (`tools/mirai/solver_lint.odin`) | diff --git a/asset/packages.odin b/asset/packages.odin index 8fae37651..070d0f75b 100644 --- a/asset/packages.odin +++ b/asset/packages.odin @@ -41,6 +41,8 @@ REF_IMPORTS :: "imports" // : a build with Final textures, REF_IMPORTS_FAST :: "imports-fast" // : a build with Fast textures, never shared REF_BUILDS :: "builds" // : the build the refresh of this machine chose REF_CACHE_PREFIX :: "cache-" // and the name of a folder of cook/cache: - +REF_ORIGINS :: "origins" // : the manifest of the package its source was ingested as on this machine +REF_SOURCES :: "sources" // : what the files of that source were when last read, never shared IMPORT_KEY_VERSIONS :: 32 // the most versions an import key combines // Where the source of a package was on the machine that ingested it: a hint @@ -146,6 +148,12 @@ cache_namespace :: proc(name:string,buffer:^[store.NAME_MAX]u8) -> string { return string(buffer[:n]) } +// The name of the refs REF_ORIGINS and REF_SOURCES of the source at origin +// (an absolute path with forward slashes), in buffer. +origin_name :: proc(origin:string,buffer:^[64]u8) -> string { + return store.hex(store.address_of_bytes(transmute([]u8)origin),buffer) +} + @(private="file") word_put :: proc(out:[]u8,value:u64,size:int) -> int { for i in 0..>uint(8*i))} diff --git a/cook/build/build.odin b/cook/build/build.odin index 22b85b9ca..6c94717b3 100644 --- a/cook/build/build.odin +++ b/cook/build/build.odin @@ -893,7 +893,7 @@ job_run :: proc(desc:Desc) -> ^Job { @(private) job_imports :: proc(job:^Job) -> bool { if len(job.imports)==0 {return true} - ok,message:=imports_sync(job.import_program,job.imports,fast=job.cook.assets=="") + ok,message:=imports_refresh(job.import_program,job.imports,fast=job.cook.assets=="") if !ok { job.result.text=strings.clone(message,virtual.arena_allocator(&job.arena)) job_fail(job,.Import_Failed) diff --git a/cook/build/imports.odin b/cook/build/imports.odin index 4df2749c6..81c10cbe6 100644 --- a/cook/build/imports.odin +++ b/cook/build/imports.odin @@ -6,8 +6,8 @@ import "core:strings" import "mirai:project" import "mirai:std/subprocess" -// Brings the imported folder of each root (a project, a plugin) up to date -// with the sources of its assets folder (cook/import/sources.sync), with the +// Brings the packages each root (a project, a plugin) pins in its +// packages.mirai to builds this engine reads (cook/import/sources.project_refresh), with the // import program of the engine (Engine_Paths.import_program, of the defines // the engine was built with), which reads the formats the plugins of the // project add from the library of its editor: the tool builds without the @@ -16,12 +16,12 @@ import "mirai:std/subprocess" // says of each source it imports goes to the output of this process as it // says it: a large source imports for minutes. message, in the temp // allocator: why it failed. -imports_sync :: proc(program:string,roots:[]string,fast:bool) -> (ok:bool,message:string) { +imports_refresh :: proc(program:string,roots:[]string,fast:bool) -> (ok:bool,message:string) { if len(roots)==0 {return true,""} command:=make([dynamic]string,context.temp_allocator) append(&command,program) if fast {append(&command,"--fast")} - for root in roots {append(&command,"--sync",root)} + for root in roots {append(&command,"--refresh",root)} state,_,errors,error:=subprocess.exec({command=command[:]},context.temp_allocator,echo=os.to_writer(os.stdout)) if error!=nil {return false,fmt.tprintf("the import program could not run: %v",error)} if state.exit_code!=0 {return false,strings.trim_space(string(errors))} diff --git a/cook/import/documents.odin b/cook/import/documents.odin index 11f20ad6d..f32572cb1 100644 --- a/cook/import/documents.odin +++ b/cook/import/documents.odin @@ -83,16 +83,18 @@ documents_follow :: proc(s:^store.Store,p:^asset.Package,b:^asset.Build,folder:s // Writes the document whose baseline is the file at baseline to target, or // follows the file target has. skeletons: the baselines the clips of an -// animation may play on, else registry has them. The warning, in the temp -// allocator, says what of the file could not be kept. -document_follow :: proc(types:^asset.Types,kind:asset.Fourcc,baseline,target:string,skeletons:[]Followed_Skeleton,registry:^asset.Registry) -> (warning:string,error:asset.Error) { +// animation may play on, else registry has them. decided: the owner of the +// baseline wins over the one of the file (Import_Settings.parts decides it +// again). The warning, in the temp allocator, says what of the file could not +// be kept. +document_follow :: proc(types:^asset.Types,kind:asset.Fourcc,baseline,target:string,skeletons:[]Followed_Skeleton,registry:^asset.Registry,decided:=false) -> (warning:string,error:asset.Error) { given:asset.Asset asset.asset_load(&given,baseline,context.temp_allocator) or_return defer asset.asset_close(&given) previous:asset.Asset existed:=os.exists(target) && asset.asset_load(&previous,target,context.temp_allocator)==.None defer if existed {asset.asset_close(&previous)} - owner:=previous.blob.header.owner if existed else given.blob.header.owner + owner:=previous.blob.header.owner if existed && !decided else given.blob.header.owner id:=given.blob.header.id switch kind { case asset.KIND_MATERIAL: diff --git a/cook/import/identity.odin b/cook/import/identity.odin index dc7c318d2..f675b67e7 100644 --- a/cook/import/identity.odin +++ b/cook/import/identity.odin @@ -188,7 +188,7 @@ writer_take :: proc(w:^Writer,output:int,hash:store.Hash) -> bool { } if target:=w.targets[output];target!="" { existed:=os.exists(target) - warning,follow_error:=document_follow(&w.types,o.kind,blob,target,w.followed[:],w.registry) + warning,follow_error:=document_follow(&w.types,o.kind,blob,target,w.followed[:],w.registry,decided=w.report.settings.parts!=.Auto) if follow_error!=.None {return save_failed(w,follow_error,target)} if warning!="" {writer_warn(w,"%s",warning)} if !existed {append(&w.created,strings.clone(target,w.keep))} @@ -250,8 +250,8 @@ fresh_path :: proc(w:^Writer,file:string) -> string { @(private) writer_id :: proc(w:^Writer) -> base.Asset_Id {return base.asset_id_random(&w.rng)} -// An absolute path with forward slashes: how files are compared. -@(private) +// An absolute path with forward slashes: how files are compared, and the +// origin of a package (asset.Package_Origin.path). path_key :: proc(path:string) -> string { absolute,error:=filepath.abs(path,context.temp_allocator) if error!=nil {absolute=path} diff --git a/cook/import/model.odin b/cook/import/model.odin index 062abb943..c3055d4e9 100644 --- a/cook/import/model.odin +++ b/cook/import/model.odin @@ -81,6 +81,7 @@ Import_Options :: struct { // (sources.code_identity), 0 for the engine's: part of what its outputs // are made from (identity_of). importer_code:u64, + hashes:^Source_Hashes, // the files of the source known by size and time (identity_of); nil: every one is read material_index:int, // the index of the first material among those of the source: imports that write its materials apart give each its own // Where the import cooks: the worker that calls it (the owner of its // job system, or the job that imports), which runs its builds in diff --git a/cook/import/outputs_test.odin b/cook/import/outputs_test.odin index eb0ac32ec..18a17e13b 100644 --- a/cook/import/outputs_test.odin +++ b/cook/import/outputs_test.odin @@ -207,10 +207,10 @@ the_models_of_an_import_share_their_parts_by_name :: proc(t:^testing.T) { // An input that changed changes what the import is made from. testing.expect(t,len(manifest.inputs)==1 && manifest.inputs[0]=="props.pkg","the input is listed") settings,_:=settings_parse(manifest.settings) - before:=identity_of(path_key(source),manifest.inputs[:],settings,0) + before:=identity_of(manifest.id,path_key(source),manifest.inputs[:],settings,0) testing.expect(t,before==manifest.identity,"the identity is that of the source, its input and its settings") testing.expect(t,os.write_entire_file(package_file,"other props")==nil) - testing.expect(t,identity_of(path_key(source),manifest.inputs[:],settings,0)!=before,"the input counts in the identity") + testing.expect(t,identity_of(manifest.id,path_key(source),manifest.inputs[:],settings,0)!=before,"the input counts in the identity") } // An image only its format decodes is keyed by its data: a texture the diff --git a/cook/import/package.odin b/cook/import/package.odin index ed767e3ee..be08b8e6e 100644 --- a/cook/import/package.odin +++ b/cook/import/package.odin @@ -3,7 +3,9 @@ package importer import "core:fmt" import "core:os" import "core:slice" +import "core:strconv" import "core:strings" +import "core:time" import "mirai:asset" import "mirai:cook/textures" import "mirai:std/base" @@ -56,28 +58,31 @@ settings_valid :: proc(settings:Import_Settings) -> bool { settings.rig_crotch>=0 && settings.rig_neck>=0 && (settings.rig==.Humanoid || settings.rig_crotch==0 && settings.rig_neck==0) } -// What the outputs of an import are made from, without any version of the -// engine (asset.Package.identity): the bytes of the source at path and of each -// input, named from the folder of the source, the text of its settings and the -// code of the plugin that imports it whole (Import_Options.importer_code). A -// file that cannot be read, such as a folder, counts as no bytes. -identity_of :: proc(path:string,inputs:[]string,settings:Import_Settings,code:u64) -> store.Hash { +// What the outputs of the package package_id are made from, without any +// version of the engine (asset.Package.identity): the package, whose ids its +// blobs hold, so two packages of the same bytes have builds of their own; the +// bytes of the source at path and of each input, named from the folder of the +// source; the text of its settings and the code of the plugin that imports it +// whole (Import_Options.importer_code). A file that cannot be read, such as a +// folder, counts as no bytes. With hashes, a file whose size and modification +// time it knows is not read. +identity_of :: proc(package_id:base.Asset_Id,path:string,inputs:[]string,settings:Import_Settings,code:u64,hashes:^Source_Hashes=nil) -> store.Hash { TAG :: "mirai import identity" bytes:=make([dynamic]u8,context.temp_allocator) append(&bytes,TAG) - address :: proc(bytes:^[dynamic]u8,file:string) { + package_id:=package_id + append(&bytes,..([^]u8)(&package_id)[:size_of(package_id)]) + address :: proc(bytes:^[dynamic]u8,file:string,hashes:^Source_Hashes) { hash:=store.address_of_bytes(nil) - if !unsafe_path(file) { - if read,_,error:=store.address_of_file(file);error==.None {hash=read} - } + if !unsafe_path(file) {hash=file_address(hashes,file)} append(bytes,..hash[:]) } - address(&bytes,path) + address(&bytes,path,hashes) folder:=path[:max(strings.last_index_byte(path,'/'),0)] for input in inputs { append(&bytes,input) append(&bytes,0) - address(&bytes,fmt.tprintf("%s/%s",folder,input)) + address(&bytes,fmt.tprintf("%s/%s",folder,input),hashes) } append(&bytes,settings_text(settings,context.temp_allocator)) append(&bytes,0) @@ -86,6 +91,82 @@ identity_of :: proc(path:string,inputs:[]string,settings:Import_Settings,code:u6 return store.address_of_bytes(bytes[:]) } +// The addresses of the files the identity of one source read, by path, as +// they were when read: kept by the library of the machine per source +// (asset.REF_SOURCES), so finding whether a source changed reads only the +// files whose size or modification time changed. The bytes still decide: a +// file that changed is read again. +Source_Hashes :: struct { + files:map[string]Source_Hash, + changed:bool, +} +Source_Hash :: struct {size,modified:i64,hash:store.Hash} + +// A file modified this close to when it was read may change again within the +// same modification time: it is read again the next time. +@(private="file") SETTLED_NANOSECONDS :: 2_000_000_000 + +// The hashes kept for the source at origin (path_key), in allocator; none +// when the store has none or they do not read. +source_hashes_load :: proc(s:^store.Store,origin:string,allocator:=context.allocator) -> (hashes:Source_Hashes) { + hashes.files=make(map[string]Source_Hash,allocator=allocator) + name:[64]u8 + hash,ref_error:=store.ref_get(s,asset.REF_SOURCES,asset.origin_name(origin,&name)) + if ref_error!=.None {return} + size,size_error:=store.blob_size(s,hash) + if size_error!=.None || size>asset.RECORD_BYTES {return} + data:=make([]u8,int(size),context.temp_allocator) + if store.read(s,hash,0,data)!=.None {return} + text:=string(data) + for line in strings.split_lines_iterator(&text) { + parts:=strings.split_n(line," ",4,context.temp_allocator) + if len(parts)!=4 {continue} + address,address_ok:=store.parse_hex(parts[0]) + file_size,size_ok:=strconv.parse_i64(parts[1]) + modified,modified_ok:=strconv.parse_i64(parts[2]) + if address_ok && size_ok && modified_ok {hashes.files[strings.clone(parts[3],allocator)]={file_size,modified,address}} + } + return +} + +// Keeps the hashes of the source at origin when reading it changed them. +source_hashes_save :: proc(s:^store.Store,origin:string,hashes:^Source_Hashes) { + if !hashes.changed {return} + files:=make([dynamic]string,0,len(hashes.files),context.temp_allocator) + for file in hashes.files {append(&files,file)} + slice.sort(files[:]) + b:=strings.builder_make(context.temp_allocator) + digits:[64]u8 + for file in files { + known:=hashes.files[file] + fmt.sbprintf(&b,"%s %d %d %s\n",store.hex(known.hash,&digits),known.size,known.modified,file) + } + hash,error:=store.put_bytes(s,transmute([]u8)strings.to_string(b)) + if error!=.None {return} + name:[64]u8 + _=store.ref_set(s,asset.REF_SOURCES,asset.origin_name(origin,&name),hash) +} + +// The address of the bytes of a file, from hashes when its size and +// modification time are those it knows; that of no bytes for a file that +// cannot be read. +@(private="file") +file_address :: proc(hashes:^Source_Hashes,file:string) -> store.Hash { + info,stat_error:=os.stat(file,context.temp_allocator) + size,modified:=info.size,time.time_to_unix_nano(info.modification_time) + if hashes!=nil && stat_error==nil { + if known,found:=hashes.files[file];found && known.size==size && known.modified==modified {return known.hash} + } + hash,_,error:=store.address_of_file(file) + if error!=.None {return store.address_of_bytes(nil)} + if hashes!=nil && stat_error==nil && time.time_to_unix_nano(time.now())-modified>SETTLED_NANOSECONDS { + if _,found:=hashes.files[file];!found {hashes.files[strings.clone(file,hashes.files.allocator)]={}} + hashes.files[file]={size,modified,hash} + hashes.changed=true + } + return hash +} + // The versions of what reads a source with a format of version // (import_version) and writes its assets, as asset.import_key combines them: // the writer and the format, every kind of asset and what builds textures. @@ -97,7 +178,7 @@ import_versions :: proc(version:u32) -> [3]u64 { // The manifest of the package the source at origin (an absolute path with // forward slashes) was ingested as: the one manifest names, else the one the -// ref REF_PACKAGES of this machine names whose origin is origin. found is +// ref REF_ORIGINS of this machine names for origin. found is // false without one. In allocator, an arena. package_previous :: proc(s:^store.Store,manifest:store.Hash,origin:string,allocator:=context.allocator) -> (p:asset.Package,hash:store.Hash,found:bool,error:Error) { digits:[64]u8 @@ -108,15 +189,12 @@ package_previous :: proc(s:^store.Store,manifest:store.Hash,origin:string,alloca if read_error!=.None {return {},{},false,error_make(.Malformed,"the manifest %s of the package of %s does not read (%v): import the source as a new package",store.hex(manifest,&digits),origin,read_error)} return p,manifest,true,{} } - refs,_:=store.ref_list(s,asset.REF_PACKAGES,context.temp_allocator) - for ref in refs { - candidate,read_error:=asset.package_get(s,ref.hash,context.temp_allocator) - if read_error!=.None || candidate.origin.path!=origin {continue} - p,read_error=asset.package_get(s,ref.hash,allocator) - if read_error!=.None {continue} - return p,ref.hash,true,{} - } - return + name:[64]u8 + named,ref_error:=store.ref_get(s,asset.REF_ORIGINS,asset.origin_name(origin,&name)) + if ref_error!=.None {return} + read_error:asset.Error + if p,read_error=asset.package_get(s,named,allocator);read_error!=.None || p.origin.path!=origin {return {},{},false,{}} + return p,named,true,{} } // The settings of the package the source at path was ingested as @@ -146,7 +224,7 @@ output_id :: proc(s:^store.Store,path:string,kind:asset.Fourcc,name:string) -> ( } // Puts the manifest of the package and its build in the store and names -// them: the package with REF_PACKAGES, the build with REF_IMPORTS when its +// them: the package with REF_PACKAGES and by its origin (REF_ORIGINS), the build with REF_IMPORTS when its // textures are Final, else REF_IMPORTS_FAST, and as the build this machine // chose for the manifest (REF_BUILDS). @(private) @@ -169,14 +247,15 @@ writer_publish :: proc(w:^Writer,name,importer:string) -> bool { build,build_error:=asset.build_put(s,&b) if build_error!=.None {return writer_fail(w,.Write,"the build of the package of %s could not be put in the store at %s: %v",w.path,s.root,build_error)} id_text:[base.ASSET_ID_TEXT]u8 - key_text,select_text:[64]u8 - refs:=[3]struct{namespace,name:string}{ + key_text,select_text,origin_text:[64]u8 + refs:=[4]struct{namespace,name:string}{ {asset.REF_PACKAGES,base.asset_id_format(w.package_id,id_text[:])}, + {asset.REF_ORIGINS,asset.origin_name(w.origin,&origin_text)}, {asset.REF_IMPORTS if b.final else asset.REF_IMPORTS_FAST,store.hex(w.key,&key_text)}, {asset.REF_BUILDS,store.hex(asset.build_select(manifest,&p),&select_text)}, } for ref,at in refs { - if error:=store.ref_set(s,ref.namespace,ref.name,manifest if at==0 else build);error!=.None { + if error:=store.ref_set(s,ref.namespace,ref.name,manifest if at<2 else build);error!=.None { return writer_fail(w,.Write,"the ref %s/%s of the package of %s could not be set in the store at %s: %v",ref.namespace,ref.name,w.path,s.root,error) } } @@ -191,7 +270,7 @@ writer_identity :: proc(w:^Writer) { if w.fixed {return} w.fixed=true slice.sort(w.inputs[:]) - w.identity=identity_of(w.origin,w.inputs[:],w.report.settings,w.options.importer_code) + w.identity=identity_of(w.package_id,w.origin,w.inputs[:],w.report.settings,w.options.importer_code,w.options.hashes) versions:=import_versions(w.version) w.key,_=asset.import_key(w.identity,versions[:]) } diff --git a/cook/import/reader_test.odin b/cook/import/reader_test.odin index 5cdf81e13..af9637f68 100644 --- a/cook/import/reader_test.odin +++ b/cook/import/reader_test.odin @@ -44,8 +44,8 @@ a_reader_sees_the_settings_kept_with_its_source :: proc(t:^testing.T) { ingested:=asset.Package{id={5,6},name="model",origin={path=path_key(path)},settings=settings_text({skeleton=kept},context.temp_allocator)} manifest,put_error:=asset.package_put(s,&ingested) if !testing.expect_value(t,put_error,asset.Error.None) {return} - id_text:[base.ASSET_ID_TEXT]u8 - testing.expect_value(t,store.ref_set(s,asset.REF_PACKAGES,base.asset_id_format(ingested.id,id_text[:]),manifest),store.Error.None) + origin:[64]u8 + testing.expect_value(t,store.ref_set(s,asset.REF_ORIGINS,asset.origin_name(ingested.origin.path,&origin),manifest),store.Error.None) model,error=import_model(path,formats[:],{store=s,settings={skeleton=given}}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} model_destroy(&model) diff --git a/cook/import/sources/bench_test.odin b/cook/import/sources/bench_test.odin index 6945b6946..caf41f02d 100644 --- a/cook/import/sources/bench_test.odin +++ b/cook/import/sources/bench_test.odin @@ -23,7 +23,7 @@ best_of :: proc(body:proc(data:rawptr),data:rawptr) -> time.Duration { } @(private="file") -Large :: struct {path,out:string,model_bytes,asset_bytes:int,triangles,vertices:int} +Large :: struct {t:^testing.T,path,out:string,model_bytes,asset_bytes:int,triangles,vertices:int} @(test) bench_import_and_write_of_a_large_obj :: proc(t:^testing.T) { @@ -42,7 +42,7 @@ bench_import_and_write_of_a_large_obj :: proc(t:^testing.T) { fmt.sbprintf(&text,"f %d/%d/%d %d/%d/%d %d/%d/%d %d/%d/%d\n",a,a,a,b,b,b,c,c,c,d,d,d) } } - large:=Large{path=write_text(folder,"grid.obj",string(text.buf[:])),out=fmt.tprintf("%s/out",folder),model_bytes=len(text.buf)} + large:=Large{t=t,path=write_text(folder,"grid.obj",string(text.buf[:])),out=fmt.tprintf("%s/out",folder),model_bytes=len(text.buf)} read:=best_of(proc(data:rawptr) { large:=(^Large)(data) model,error:=importer.import_model(large.path,FORMATS[:]) @@ -58,7 +58,7 @@ bench_import_and_write_of_a_large_obj :: proc(t:^testing.T) { model,_:=importer.import_model(large.path,FORMATS[:]) defer importer.model_destroy(&model) // Every round builds the mesh: the one before left the same file. - report,error:=write_into(&model,large.out,{rebuild=true}) + report,error:=write_into(large.t,&model,large.out,{rebuild=true}) assert(error.code==.None) large.asset_bytes=report.assets[0].bytes importer.report_destroy(&report) diff --git a/cook/import/sources/cache_test.odin b/cook/import/sources/cache_test.odin index 66a19438c..b0a58d86e 100644 --- a/cook/import/sources/cache_test.odin +++ b/cook/import/sources/cache_test.odin @@ -11,6 +11,7 @@ import importer "mirai:cook/import" import "mirai:cook/prefab" import "mirai:cook/textures" import "mirai:std/base" +import "mirai:store" @(test) sources_are_known_by_their_content :: proc(t:^testing.T) { @@ -33,18 +34,18 @@ sources_are_known_by_their_content :: proc(t:^testing.T) { testing.expectf(t,source_kind(path)==c.kind,"%s is a %v source, not %v",c.name,c.kind,source_kind(path)) } // A GLB is read as one whatever its name says. - report,error:=import_source(fmt.tprintf("%s/model.bin",folder),{world=prefab.import_world(t)}) + report,error:=import_test(t,fmt.tprintf("%s/model.bin",folder),{world=prefab.import_world(t)}) testing.expectf(t,error.code==.None && len(report.assets)>0,"%v %s",error.code,error.message) importer.report_destroy(&report) - _,refused:=import_source(fmt.tprintf("%s/mesh.txt",folder)) + _,refused:=import_test(t,fmt.tprintf("%s/mesh.txt",folder)) testing.expect(t,refused.code==.Unsupported && strings.contains(refused.message,"glTF, USD, FBX and OBJ"),refused.message) - _,missing:=import_source(fmt.tprintf("%s/none.obj",folder)) + _,missing:=import_test(t,fmt.tprintf("%s/none.obj",folder)) testing.expect_value(t,missing.code,importer.Error_Code.Not_Found) - // Every source has a sidecar, next to it, which keeps its id. - picture,picture_error:=import_source(fmt.tprintf("%s/picture.dat",folder)) - testing.expect(t,picture_error.code==.None && len(picture.assets)==1 && picture.assets[0].kind==asset.KIND_TEXTURE && os.exists(fmt.tprintf("%s/picture.dat.import",folder))) + // An image is known by its bytes, and nothing is written beside it. + picture,picture_error:=import_test(t,fmt.tprintf("%s/picture.dat",folder)) + testing.expect(t,picture_error.code==.None && len(picture.assets)==1 && picture.assets[0].kind==asset.KIND_TEXTURE && !os.exists(fmt.tprintf("%s/picture.dat.import",folder))) importer.report_destroy(&picture) - _,collection:=import_source(fmt.tprintf("%s/family.ttc",folder)) + _,collection:=import_test(t,fmt.tprintf("%s/family.ttc",folder)) testing.expect(t,collection.code==.Unsupported && strings.contains(collection.message,"collection"),collection.message) } @@ -85,26 +86,30 @@ built_with :: proc(t:^testing.T,path:string,settings:u64) -> (was:u64) { return } -// What is current is decided by the name of a file of the cache: the key of -// what the import builds from now. A file under another key is taken as it is -// when what it records it was built from is what it would be built from now. +// What is current is a blob a build of the package has: the build of the key +// of what the import is made from now, else the one this machine chose for +// its previous manifest, when what the blob records it was built from is what +// it would be built from now. @(test) -an_import_keeps_what_its_cache_says_is_current_and_builds_again_what_changed :: proc(t:^testing.T) { +an_import_keeps_what_its_builds_say_is_current_and_builds_again_what_changed :: proc(t:^testing.T) { root,_:=test_project(t,"cache") if root=="" {return} source:=copy_in(root,DEMO,"gallery/vase.obj","gallery/vase.mtl","gallery/vase_color.png") world:=prefab.import_world(t) - Kept :: struct {meshes,textures,others:int,mesh,texture:string,id:base.Asset_Id} + s:=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return} + Kept :: struct {meshes,textures,others:int,mesh,texture:string,id:base.Asset_Id,manifest,build:store.Hash} import_kept :: proc(t:^testing.T,source:string,options:importer.Import_Options,loc:=#caller_location) -> (k:Kept,ok:bool) { - report,error:=import_source(source,options) + report,error:=import_test(t,source,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None,"%v %s",error.code,error.message,loc=loc) {return} k.meshes,k.textures,k.others=kept_of(&report) k.mesh,k.id=written_of(&report) k.texture,_=written_of(&report,asset.KIND_TEXTURE) + k.manifest,k.build=report.manifest,report.build return k,true } - first,ok:=import_kept(t,source,{world=world}) + first,ok:=import_kept(t,source,{world=world,store=s}) if !ok {return} testing.expect(t,first.meshes==0 && first.textures==0 && first.others==0,"a first import builds everything") built,_:=os.read_entire_file(first.mesh,context.temp_allocator) @@ -116,33 +121,46 @@ an_import_keeps_what_its_cache_says_is_current_and_builds_again_what_changed :: // Nothing changed: the mesh and the texture are kept; materials and the prefab, cheap, are written again. again:Kept - again,ok=import_kept(t,source,{world=world}) + again,ok=import_kept(t,source,{world=world,store=s}) testing.expect(t,ok && again.meshes==1 && again.textures==1 && again.others==0 && again.mesh==first.mesh && same(first.mesh,built),"an unchanged source keeps its mesh and its texture") // Another code that imports it (that of a plugin that imports it whole) is another key, and - // the same mesh and image: their files are taken under it as they are, not built nor written. + // the same mesh and image: the blobs of the build before are taken, not built. coded:Kept - coded,ok=import_kept(t,source,{world=world,importer_code=1}) - testing.expect(t,ok && coded.meshes==1 && coded.textures==1 && coded.id==first.id && coded.mesh!=first.mesh && !os.exists(first.mesh) && !os.exists(first.texture),"a file under another key is taken") - testing.expect(t,same(coded.mesh,built) && same(coded.texture,texture),"as it is") + coded,ok=import_kept(t,source,{world=world,store=s,importer_code=1}) + testing.expect(t,ok && coded.meshes==1 && coded.textures==1 && coded.id==first.id && coded.mesh==first.mesh && coded.texture==first.texture,"a blob of another key is taken") // One that records other encoders (a texture) or another builder (a mesh) is built again. - texture_settings:=built_with(t,coded.texture,1) - mesh_settings:=built_with(t,coded.mesh,1) + b,read_error:=asset.build_get(s,coded.build,context.temp_allocator) + p,_:=asset.package_get(s,coded.manifest,context.temp_allocator) + if !testing.expect_value(t,read_error,asset.Error.None) {return} + recorded:[2]u64 + for &o in b.outputs { + if o.kind!=asset.KIND_MESH && o.kind!=asset.KIND_TEXTURE {continue} + blob,_:=store.blob_path(s,o.blob,context.temp_allocator) + data,_:=os.read_entire_file(blob,context.temp_allocator) + other:=fmt.tprintf("%s/other",root) + _=os.write_entire_file(other,data) + recorded[int(o.kind==asset.KIND_TEXTURE)]=built_with(t,other,1) + o.blob,_=store.put_file(s,other) + } + forged,_:=asset.build_put(s,&b) + digits:[64]u8 + store.ref_set(s,asset.REF_BUILDS,store.hex(asset.build_select(coded.manifest,&p),&digits),forged) other:Kept - other,ok=import_kept(t,source,{world=world,importer_code=2}) + other,ok=import_kept(t,source,{world=world,store=s,importer_code=2}) testing.expect(t,ok && other.meshes==0 && other.textures==0,"what other encoders or another builder made is built again") - testing.expect(t,same(other.mesh,built) && same(other.texture,texture) && texture_settings!=1 && mesh_settings!=1,"to the bytes of now") + testing.expect(t,same(other.mesh,built) && same(other.texture,texture) && recorded[0]!=1 && recorded[1]!=1,"to the bytes of now") - // A setting of the mesh builds it again, under a new name, with the same id; the texture is + // A setting of the mesh builds it again, another blob with the same id; the texture is // built from the same image and settings and is taken. turned:Kept - turned,ok=import_kept(t,source,{world=world,front=.X,ignore_stored=true}) - testing.expect(t,ok && turned.meshes==0 && turned.textures==1 && turned.id==first.id && turned.mesh!=other.mesh && !os.exists(other.mesh),"a setting builds again, and the file of before goes") + turned,ok=import_kept(t,source,{world=world,store=s,front=.X,ignore_stored=true}) + testing.expect(t,ok && turned.meshes==0 && turned.textures==1 && turned.id==first.id && turned.mesh!=other.mesh && os.exists(other.mesh),"a setting builds again, and the blob of before stays in the store") // --rebuild keeps nothing; with the settings of the first import, it gives its bytes. rebuilt:Kept - rebuilt,ok=import_kept(t,source,{world=world,rebuild=true,ignore_stored=true}) + rebuilt,ok=import_kept(t,source,{world=world,store=s,rebuild=true,ignore_stored=true}) testing.expect(t,ok && rebuilt.meshes==0 && rebuilt.textures==0 && rebuilt.others==0 && rebuilt.mesh==first.mesh,"rebuild builds everything") testing.expect(t,same(first.mesh,built)) } @@ -159,7 +177,7 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr imported :: proc(t:^testing.T,types:^asset.Types,folder,name:string,bytes:[]u8,options:importer.Import_Options) -> (format:asset.Texture_Format,mips:int,texels:string,flags:asset.Texture_Flags) { image:=fmt.tprintf("%s/%s.png",folder,name) _=os.write_entire_file(image,bytes) - report,error:=import_source(image,options) + report,error:=import_test(t,image,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None && len(report.assets)==1,"%v %s",error.code,error.message) {return} a:asset.Asset @@ -230,7 +248,7 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ return len(view.colliders) } imported :: proc(t:^testing.T,source:string,options:importer.Import_Options) -> (mesh:string,ok:bool) { - report,error:=import_source(source,options) + report,error:=import_test(t,source,options) defer importer.report_destroy(&report) mesh,_=written_of(&report) return mesh,testing.expectf(t,error.code==.None,"%v %s",error.code,error.message) @@ -260,24 +278,24 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ hulled,bytes_error:=asset.document_bytes(&doc,context.temp_allocator) testing.expect(t,bytes_error==.None && cache.put({store=kept},importer.MESH_CACHE,key,importer.MESH_CACHE_VERSION,hulled)) testing.expect(t,os.remove(mesh_path)==nil) - if _,ok=imported(t,source,options);!ok {return} - testing.expect_value(t,colliders(t,&types,mesh_path),1) + hulled_path:string + if hulled_path,ok=imported(t,source,options);!ok {return} + testing.expect_value(t,colliders(t,&types,hulled_path),1) // A store that misses asks its source before building: what a peer built is taken as it is. peer:=options peer.store=test_store(t,fmt.tprintf("%s/peer",root)) peer.cache_source=copy_source(kept) - testing.expect(t,os.remove(mesh_path)==nil) - if _,ok=imported(t,source,peer);!ok {return} - testing.expect_value(t,colliders(t,&types,mesh_path),1) + peer_path:string + if peer_path,ok=imported(t,source,peer);!ok {return} + testing.expect_value(t,colliders(t,&types,peer_path),1) // Rebuilding does not read the cache, and builds it again. rebuilt:=options rebuilt.rebuild=true - if _,ok=imported(t,source,rebuilt);!ok {return} - testing.expect_value(t,colliders(t,&types,mesh_path),0) - testing.expect(t,os.remove(mesh_path)==nil) - if _,ok=imported(t,source,options);!ok {return} - last,_:=os.read_entire_file(mesh_path,context.temp_allocator) + rebuilt_path:string + if rebuilt_path,ok=imported(t,source,rebuilt);!ok {return} + testing.expect_value(t,colliders(t,&types,rebuilt_path),0) + last,_:=os.read_entire_file(rebuilt_path,context.temp_allocator) testing.expect(t,string(last)==string(built)) } diff --git a/cook/import/sources/cli_test.odin b/cook/import/sources/cli_test.odin index 22fbdc05c..8e05d4f57 100644 --- a/cook/import/sources/cli_test.odin +++ b/cook/import/sources/cli_test.odin @@ -6,6 +6,7 @@ import "core:os" import "core:slice" import "core:strings" import "core:testing" +import "mirai:asset" import "mirai:cook/prefab" import importer "mirai:cook/import" import "mirai:project" @@ -22,10 +23,15 @@ import "mirai:std/subprocess" @(private="file") Result :: struct {code:int,out,err:string} +// mirai import with args, with --store store_root when it has any: the +// store of a test is never the machine's. @(private="file") -import_run :: proc(args:..string) -> Result { +import_run :: proc(store_root:string,args:..string) -> Result { output,errors:=strings.builder_make(context.temp_allocator),strings.builder_make(context.temp_allocator) - code:=command(args,strings.to_writer(&output),strings.to_writer(&errors),test_library) + all:=make([dynamic]string,context.temp_allocator) + append(&all,..args) + if len(args)>0 {append(&all,"--store",store_root)} + code:=command(all[:],strings.to_writer(&output),strings.to_writer(&errors),test_library) return {code,strings.to_string(output),strings.to_string(errors)} } @@ -83,80 +89,96 @@ mirai_import_writes_assets_that_mirai_asset_verifies :: proc(t:^testing.T) { if !testing.expectf(t,error==nil && state.exit_code==0,"the tool does not build: %v\n%s%s",error,string(stdout),string(stderr)) {return} } - // A model with a texture in the assets of a project: mesh, three materials, the image of the leaves and the prefab. + kept:=fmt.tprintf("%s/store",root_folder) + // A model with a texture in the assets of a project: ingested (mesh, three materials, the + // image of the leaves) and added to the project, with its documents and prefab beside it. plant:=copy_in(root_folder,DEMO,"gallery/plant.gltf","gallery/leaves_color.png") - first:=import_run(plant,"--uncompressed") + first:=import_run(kept,plant,"--uncompressed") testing.expectf(t,first.code==0 && first.err=="","import fails: %d\n%s%s",first.code,first.out,first.err) paths:=reported_paths(first.out) testing.expect_value(t,len(paths),6) - testing.expect(t,strings.contains(first.out,"6 assets\n") && !strings.contains(first.out,"warning"),first.out) + testing.expect(t,strings.contains(first.out,"6 assets\n") && strings.contains(first.out,"\npackage ") && !strings.contains(first.out,"warning"),first.out) for kind in ([]string{"mesh ","material ","texture ","scene "}) {testing.expectf(t,strings.contains(first.out,kind),"the report names a %s",kind)} + list,_,list_error:=asset.packages_load(root_folder,context.temp_allocator) + testing.expect(t,list_error==.None && len(list.packages)==1 && list.packages[0].folder=="gallery","the project pins the package under the folder of the source") // The scene is checked against the game of a project, which this folder is not. verify:=make([dynamic]string,context.temp_allocator) append(&verify,"asset","verify") for path in paths {if !strings.has_suffix(path,".scene") {append(&verify,path)}} checked:=run(t,exe,..verify[:]) testing.expectf(t,checked.code==0,"asset verify fails: %s%s",checked.out,checked.err) - listing:=run(t,exe,"asset","ls",folder) - testing.expect(t,listing.code==0 && strings.contains(listing.out," gallery/plant/Leaves.mat") && strings.contains(listing.out," gallery/plant/Plant.mesh") && strings.contains(listing.out," gallery/plant/leaves_color.tex"),listing.out) // Importing again keeps every id, and the mesh and the texture are not built again. - again:=import_run(plant,"--uncompressed") + again:=import_run(kept,plant,"--uncompressed") testing.expect(t,again.code==0 && strings.contains(again.out,"6 assets, 2 were already up to date") && slice.equal(reported_paths(again.out),paths),again.out) - rebuilt:=import_run(plant,"--uncompressed","--rebuild") + rebuilt:=import_run(kept,plant,"--uncompressed","--rebuild") testing.expect(t,rebuilt.code==0 && slice.equal(reported_assets(rebuilt.out),reported_assets(first.out)),"--rebuild builds everything again, to the same bytes") // An image on its own. bricks:=copy_in(root_folder,DEMO,"surfaces/bricks/bricks_color.png") - texture:=import_run(bricks,"--srgb") - testing.expectf(t,texture.code==0 && strings.contains(texture.out,"texture") && strings.contains(texture.out,".tex"),"%d: %s%s",texture.code,texture.out,texture.err) + texture:=import_run(kept,bricks,"--srgb") + testing.expectf(t,texture.code==0 && strings.contains(texture.out,"texture"),"%d: %s%s",texture.code,texture.out,texture.err) verify_texture:=make([dynamic]string,context.temp_allocator) append(&verify_texture,"asset","verify") append(&verify_texture,..reported_paths(texture.out)) verified:=run(t,exe,..verify_texture[:]) testing.expect_value(t,verified.code,0) + current:=import_run(kept,bricks) + testing.expectf(t,current.code==0 && strings.contains(current.out,"0 assets"),"a current source is not imported again: %s%s",current.out,current.err) - // The sidecar of an import, next to its source, keeps its settings: the options change them, and importing again uses them. + // The package of a source keeps its settings: the options change them, and importing again uses them. copy:=copy_in(root_folder,DEMO,"gallery/vase.obj","gallery/vase.mtl","gallery/vase_color.png") - turned:=import_run(copy,"--front","x") - testing.expectf(t,turned.code==0 && strings.contains(turned.out," triangles") && !strings.contains(turned.out,"hull") && strings.contains(turned.out,"sidecar ") && strings.contains(turned.out,"vase.obj.import"),"%d: %s%s",turned.code,turned.out,turned.err) - sidecar:=fmt.tprintf("%s.import",copy) - stored,_:=os.read_entire_file(sidecar,context.temp_allocator) - testing.expect(t,strings.contains(string(stored),"\"front\": \"X\""),string(stored)) + turned:=import_run(kept,copy,"--front","x","--rebuild") + testing.expectf(t,turned.code==0 && strings.contains(turned.out," triangles") && !strings.contains(turned.out,"hull"),"%d: %s%s",turned.code,turned.out,turned.err) + stored,_,_:=importer.settings_load({store=test_store(t,kept)},copy) + testing.expect_value(t,stored.front,importer.Front.X) ids :: proc(out:string) -> []string { found:=make([dynamic]string,context.temp_allocator) for line in reported_assets(out) {append(&found,line[:32])} return found[:] } - mesh_path:string - for line in reported_assets(turned.out) {if strings.contains(line," mesh ") {mesh_path=reported_paths(line)[0]}} - with_front,_:=os.read_entire_file(mesh_path,context.temp_allocator) - repeated:=import_run(copy) - same,_:=os.read_entire_file(mesh_path,context.temp_allocator) - testing.expect(t,repeated.code==0 && slice.equal(reported_paths(repeated.out),reported_paths(turned.out)) && string(same)==string(with_front) && len(same)>0,"importing again without options gives the same assets") - bare:=import_run(copy,"--front","negative_z","--lods","none") + repeated:=import_run(kept,copy,"--rebuild") + testing.expect(t,repeated.code==0 && slice.equal(reported_paths(repeated.out),reported_paths(turned.out)),"importing again without options gives the same assets") + bare:=import_run(kept,copy,"--front","negative_z","--lods","none") testing.expect(t,bare.code==0 && slice.equal(ids(bare.out),ids(turned.out)) && strings.contains(bare.out," 3232 triangles"),bare.out) - _=os.write_entire_file(sidecar,"{\"lods\": \"Never\"}") - refused:=import_run(copy) - testing.expect(t,refused.code==1 && strings.contains(refused.err,"vase.obj.import"),refused.err) + + // A source outside any project: ingested into the library alone. + outside:=fmt.tprintf("%s/outside",root_folder) + os.make_directory_all(outside) + data,_:=os.read_entire_file(DEMO+"surfaces/bricks/bricks_color.png",context.temp_allocator) + loose:=fmt.tprintf("%s/loose.png",outside) + _=os.write_entire_file(loose,data) + alone:=import_run(kept,loose) + testing.expectf(t,alone.code==0 && strings.contains(alone.out,"texture") && strings.contains(alone.out,"package "),"%d: %s%s",alone.code,alone.out,alone.err) + list,_,_=asset.packages_load(root_folder,context.temp_allocator) + testing.expect_value(t,len(list.packages),3) + // Added to a project when it is named, under the folder given. + named:=import_run(kept,loose,"--project",root_folder,"--folder","loose") + testing.expectf(t,named.code==0,"%d: %s%s",named.code,named.out,named.err) + list,_,_=asset.packages_load(root_folder,context.temp_allocator) + testing.expect(t,len(list.packages)==4 && list.packages[3].folder=="loose") + + // A refresh of the project: everything is here, nothing is built. + refreshed:=import_run(kept,"--refresh",root_folder) + testing.expectf(t,refreshed.code==0 && strings.contains(refreshed.out,"4 packages, 0 ingested"),"%d: %s%s",refreshed.code,refreshed.out,refreshed.err) // Misuse and failures. - testing.expect_value(t,import_run(copy,"--lods","some").code,2) - testing.expect_value(t,import_run(copy,"--front","up").code,2) - testing.expect_value(t,import_run(bricks,"--front","x").code,2) - testing.expect_value(t,import_run().code,2) - testing.expect_value(t,import_run(copy,"--srgb").code,2) - testing.expect_value(t,import_run(copy,"--alpha-coverage","0.5").code,2) - testing.expect_value(t,import_run(bricks,"--alpha-coverage","1.5").code,2) - testing.expect_value(t,import_run(bricks,"--rdo-lambda","101").code,2) - testing.expect_value(t,import_run(bricks,"--rdo-lambda","-1").code,2) - testing.expect_value(t,import_run(copy,"--nope").code,2) - testing.expect_value(t,import_run(copy,"--out",folder).code,2) - missing:=import_run(fmt.tprintf("%s/missing.obj",folder)) + testing.expect_value(t,import_run(kept,copy,"--lods","some").code,2) + testing.expect_value(t,import_run(kept,copy,"--front","up").code,2) + testing.expect_value(t,import_run(kept,bricks,"--front","x").code,2) + testing.expect_value(t,import_run(kept).code,2) + testing.expect_value(t,import_run(kept,copy,"--srgb").code,2) + testing.expect_value(t,import_run(kept,copy,"--alpha-coverage","0.5").code,2) + testing.expect_value(t,import_run(kept,bricks,"--alpha-coverage","1.5").code,2) + testing.expect_value(t,import_run(kept,bricks,"--rdo-lambda","101").code,2) + testing.expect_value(t,import_run(kept,bricks,"--rdo-lambda","-1").code,2) + testing.expect_value(t,import_run(kept,copy,"--nope").code,2) + testing.expect_value(t,import_run(kept,copy,"--out",folder).code,2) + testing.expect_value(t,import_run(kept,loose,"--folder","loose").code,2) + testing.expect_value(t,import_run(kept,"--refresh",root_folder,copy).code,2) + missing:=import_run(kept,fmt.tprintf("%s/missing.obj",folder)) testing.expect(t,missing.code==1 && strings.contains(missing.err,"mirai import:") && strings.contains(missing.err,"missing.obj"),missing.err) - outside:=import_run(fmt.tprintf("%s/elsewhere.obj",root_folder)) - testing.expect(t,outside.code==1 && strings.contains(outside.err,"assets"),outside.err) write_text(folder,"broken.obj","v 0 0 0\nf 1 2 3\n") - broken:=import_run(fmt.tprintf("%s/broken.obj",folder)) + broken:=import_run(kept,fmt.tprintf("%s/broken.obj",folder)) testing.expect(t,broken.code==1 && strings.contains(broken.err,"outside"),broken.err) } diff --git a/cook/import/sources/cluster_test.odin b/cook/import/sources/cluster_test.odin index 7eb115793..5c369de63 100644 --- a/cook/import/sources/cluster_test.odin +++ b/cook/import/sources/cluster_test.odin @@ -19,7 +19,7 @@ demo_level0 :: proc(t:^testing.T,folder,relative:string,file:^asset.Asset,types: source,name:=filepath.dir(relative),filepath.base(relative) copied:=copy_in(folder,DEMO,source) if filepath.ext(source)==".obj" {copy_in(folder,DEMO,"gallery/vase.mtl","gallery/vase_color.png")} - report,import_error:=import_source(copied,{world=prefab.import_world(t),textures=.Uncompressed}) + report,import_error:=import_test(t,copied,{world=prefab.import_world(t),textures=.Uncompressed}) if !testing.expectf(t,import_error.code==.None,"%s: %s",source,import_error.message) {return} defer importer.report_destroy(&report) written,found:=written_of(&report,asset.KIND_MESH,name) diff --git a/cook/import/sources/command.odin b/cook/import/sources/command.odin index b3da23e8c..8384fd966 100644 --- a/cook/import/sources/command.odin +++ b/cook/import/sources/command.odin @@ -10,36 +10,39 @@ import "core:time" import "mirai:asset" import importer "mirai:cook/import" import "mirai:cook/textures" +import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" import "mirai:store" // The command mirai import, which the import program of the engine runs: it -// turns a model (OBJ, glTF, GLB, USD, FBX, or a format a plugin of the +// ingests a model (OBJ, glTF, GLB, USD, FBX, or a format a plugin of the // project adds), an image, a sound (WAV, Ogg Vorbis), a font (TrueType, // OpenType), a manifest of prompts (.prompts.json) or a source a plugin -// of the project imports whole in the assets folder of a project or a plugin -// into assets with import_source, as the editor does: a model also gets the -// prefab of its hierarchy. -// A source is imported with the settings its sidecar keeps (.import), -// changed by the options given; the result is written back there with the id -// of each asset, so importing again without options gives the same assets. +// of the project imports whole, from wherever it is, into the library of the +// machine store (ingest), and adds it to a project (project_add): the one of +// --project, else the one whose assets hold the file, else none. Its cache +// is listed under --folder of the project's assets (else the folder of the +// file under them, else the top), and its documents and the prefab of a +// model are written there. +// A source is ingested with the settings its package keeps, changed by the +// options given; ingesting again without options gives the same assets, and +// a source whose package is current is not imported again, unless --rebuild. // --rdo-lambda 0 encodes the blocks of textures for the least error and a -// higher lambda for smaller files. An asset whose file already is the one the -// import gives is kept as it is, unless --rebuild. An image imported for the -// first time without --srgb, --data or --normal holds what its name says -// (image_use). Only models and images have settings. -// --sync , once per project or plugin, imports instead every source -// under it whose outputs in its imported folder are missing or stale (sync): -// what a checkout needs before it runs. --fast builds the textures of a sync -// or of an import Fast, as the editor imports them, unless the texture cache -// keeps them Final: what a build that does not cook takes. What imports build -// is kept in the caches of the machine store (cook/cache), or of the store at -// --store. +// higher lambda for smaller files. An image ingested for the first time +// without --srgb, --data or --normal holds what its name says (image_use). +// Only models and images have settings. +// --refresh , once per project or plugin, instead brings every package +// its packages.mirai pins to a build this engine reads (project_refresh): what +// a checkout needs before it runs. It removes nothing and builds only from a +// source on this machine; a package nothing here gives is a failure that names +// it. --fast builds textures Fast, as the editor imports them, unless the +// texture cache keeps them Final, and takes a Fast build: what a build that +// does not cook takes. The store is that of the machine, or the one at --store. // The report goes to output, one asset per line: id, kind, bytes and path, and -// under it, indented, what the asset holds; then the settings file and the +// under it, indented, what the asset holds; then the package and the // warnings. How far a long import is goes to errors while it runs, and a -// failure as "mirai import: : ". A sync says each source it +// failure as "mirai import: : ". A refresh says each source it // imports, and how far it is, in output. // --profile adds the seconds each zone of the import took, summed over the // threads that cooked it and on the thread that imports, and the time no zone @@ -48,7 +51,7 @@ import "mirai:store" // Exit codes: 0 done, 1 failed, 2 misuse. @(private="file") -IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--rebuild] [--fast] [--store ] [--profile] [--release]" +IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--project ] [--folder ] [--rebuild] [--fast] [--store ] [--profile] [--release] | mirai import --refresh [--refresh ...] [--fast] [--store ] [--release]" // library reads the formats and the importers of whole sources the plugins of // a project add and the types of its game (in the import program of the @@ -56,7 +59,10 @@ IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lam command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> int { file:string options:importer.Import_Options - roots:=make([dynamic]string,context.temp_allocator) // of --sync + roots:=make([dynamic]string,context.temp_allocator) // of --refresh + root:string // of --project + folder:string // of --folder + foldered:bool profiled:bool store_root:string // of the store whose caches keep what imports build; "": that of the machine // What the options change of the settings the source has. @@ -72,9 +78,11 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in for ;at in } options.store=&kept } + library_options:=Library_Options{store=options.store,fast_textures=options.fast_textures,rebuild=options.rebuild,plugins=library,worker=options.worker} + if options.store==nil {fmt.wprintln(errors,"mirai import: this machine has no folder of user data for its store: pass one with --store");return 1} if len(roots)>0 { - return sync_command(roots[:],options,library,output,errors) + if file!="" || root!="" || foldered || changed!={} {fmt.wprintln(errors,IMPORT_USAGE);return 2} + return refresh_command(roots[:],library_options,output,errors) } if file=="" {fmt.wprintln(errors,IMPORT_USAGE);return 2} - root,in_assets:=source_root(file,context.temp_allocator) - if !in_assets {fmt.wprintfln(errors,"mirai import: %s is in no assets folder of a project or a plugin: move it beside what it makes, under assets/",file);return 1} - options.root=root + origin:=importer.path_key(file) + // The project it is added to, and where under its assets. + if root=="" { + if inside,in_assets:=source_root(origin,context.temp_allocator);in_assets { + root=inside + if !foldered { + assets:=project.folder_path(root,.Assets,context.temp_allocator) + folder=strings.trim_prefix(filepath.dir(origin),assets) + folder=strings.trim_prefix(folder,"/") + } + } + } + if root!="" { + root=project.path_absolute(root,context.temp_allocator) + library_options.root=root + } else if foldered { + fmt.wprintln(errors,"mirai import: --folder is a folder of the assets of a project: give it with --project") + return 2 + } formats:=FORMATS[:] - kind:=source_kind(file) + kind:=source_kind(origin) plugins:Library_Imports defer if plugins.release!=nil {plugins.release(plugins.state)} - if (kind==.None || kind==.Model) && library!=nil { + if (kind==.None || kind==.Model) && library!=nil && root!="" { plugins=library(root,context.temp_allocator) - options.world=plugins.world all:=make([dynamic]importer.Format,0,len(FORMATS)+len(plugins.formats),context.temp_allocator) append(&all,..FORMATS[:]) append(&all,..plugins.formats) formats=all[:] - kind=source_kind(file,formats) - if _,claimed:=whole_find(plugins.wholes,file);claimed {kind=.None} + kind=source_kind(origin,formats) + if _,claimed:=whole_find(plugins.wholes,origin);claimed {kind=.None} } if changed&{.Image,.Alpha_Coverage}!={} && kind!=.Image {fmt.wprintln(errors,"mirai import: --srgb, --normal, --normal-green-down, --data and --alpha-coverage apply only to images");return 2} if changed&{.Lods,.Parts,.Front,.Rig,.Rig_Crotch,.Rig_Neck,.Skeleton}!={} && kind!=.Model {fmt.wprintln(errors,"mirai import: --lods, --parts, --front, --skeleton and the --rig options apply only to models");return 2} + // The settings the package keeps, changed by the options: given only when they change it. + settings:^importer.Import_Settings if kind==.Model || kind==.Image { - stored,found,settings_error:=importer.settings_load(options,file) + stored,found,settings_error:=importer.settings_load(options,origin) if settings_error.code!=.None { fmt.wprintfln(errors,"mirai import: %s: %s",file,settings_error.message) return 1 } - options.settings,options.ignore_stored=stored,true + chosen:=stored // An image with no settings yet holds what its name says, as in the editor. - if kind==.Image && !found {options.image=image_use(file)} - if .Textures in changed {options.textures=given.textures} - if .Rdo_Lambda in changed {options.rdo_lambda=given.rdo_lambda} - if .Image in changed {options.image=given.image} - if .Alpha_Coverage in changed {options.alpha_coverage=given.alpha_coverage} - if .Lods in changed {options.lods=given.lods} - if .Parts in changed {options.parts=given.parts} - if .Front in changed {options.front=given.front} - if .Rig in changed {options.rig=given.rig} - if .Rig_Crotch in changed {options.rig_crotch=given.rig_crotch} - if .Rig_Neck in changed {options.rig_neck=given.rig_neck} - if .Skeleton in changed {options.skeleton=given.skeleton} + if kind==.Image && !found {chosen.image=image_use(origin)} + if .Textures in changed {chosen.textures=given.textures} + if .Rdo_Lambda in changed {chosen.rdo_lambda=given.rdo_lambda} + if .Image in changed {chosen.image=given.image} + if .Alpha_Coverage in changed {chosen.alpha_coverage=given.alpha_coverage} + if .Lods in changed {chosen.lods=given.lods} + if .Parts in changed {chosen.parts=given.parts} + if .Front in changed {chosen.front=given.front} + if .Rig in changed {chosen.rig=given.rig} + if .Rig_Crotch in changed {chosen.rig_crotch=given.rig_crotch} + if .Rig_Neck in changed {chosen.rig_neck=given.rig_neck} + if .Skeleton in changed {chosen.skeleton=given.skeleton} // The heights tell the fit of a person where the body is. - if options.rig!=.Humanoid { + if chosen.rig!=.Humanoid { if changed&{.Rig_Crotch,.Rig_Neck}!={} {fmt.wprintln(errors,"mirai import: --rig-crotch and --rig-neck go with --rig humanoid");return 2} - options.rig_crotch,options.rig_neck=0,0 + chosen.rig_crotch,chosen.rig_neck=0,0 + } + if chosen!=stored || !found { + settings=new(importer.Import_Settings,context.temp_allocator) + settings^=chosen } } profile:base.Profile started:=time.tick_now() if profiled && !importer.profile_begin(&profile,&options) {fmt.wprintln(errors,"mirai import: no memory for --profile");return 1} defer if profiled {profile_print(&profile,started,output);base.profile_destroy(&profile)} - progress:importer.Progress - importer.progress_begin(&progress,errors,filepath.base(file)) - options.progress=&progress - report,error:=import_source(file,options,formats,plugins.wholes) + library_options.worker=options.worker + library_options.progress=errors + if root=="" { + report,error:=ingest(origin,library_options,settings) + if error.code!=.None { + fmt.wprintfln(errors,"mirai import: %s: %s",file,error.message) + return 1 + } + defer importer.report_destroy(&report) + ingest_print(&report,output) + return 0 + } + report,added,error:=project_ingest(root,origin,folder,library_options,settings) + defer library_report_destroy(&added) if error.code!=.None { fmt.wprintfln(errors,"mirai import: %s: %s",file,error.message) return 1 } defer importer.report_destroy(&report) + ingest_print(&report,output) + return library_print(&added,root,output,errors) +} + +// The assets an ingest wrote, its warnings and its package. +@(private="file") +ingest_print :: proc(report:^importer.Import_Report,output:io.Writer) { written_print(output,report.assets[:]) for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} - return 0 + id:[base.ASSET_ID_TEXT]u8 + digits:[64]u8 + fmt.wprintfln(output,"package %s %s",base.asset_id_format(report.package_id,id[:]),store.hex(report.manifest,&digits)) } -// --sync: every root in turn, one line each with what its sync did. +// --refresh: every root in turn, one line each with what its refresh did. @(private="file") -sync_command :: proc(roots:[]string,options:importer.Import_Options,library:Library,output,errors:io.Writer) -> int { +refresh_command :: proc(roots:[]string,options:Library_Options,output,errors:io.Writer) -> int { code:=0 for root in roots { started:=time.tick_now() - report:=sync(root,{store=options.store,rebuild=options.rebuild,fast_textures=options.fast_textures,worker=options.worker,library=library,progress=output}) - defer sync_report_destroy(&report) - for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} - for failure in report.failures {fmt.wprintfln(errors,"mirai import: %s",failure)} - if len(report.failures)>0 {code=1} - if report.sources>0 || report.removed>0 { - fmt.wprintfln(output,"%s: %d of %d imported sources built again, %d stale files removed (%.1f s).",root,report.imported,report.sources,report.removed,time.duration_seconds(time.tick_since(started))) + options:=options + options.root=project.path_absolute(root,context.temp_allocator) + options.progress=output + report:=project_refresh(root,options) + defer library_report_destroy(&report) + if library_print(&report,root,output,errors)!=0 {code=1} + if report.packages>0 { + fmt.wprintfln(output,"%s: %d packages, %d ingested, %d builds fetched (%.1f s).",root,report.packages,report.ingested,report.fetched,time.duration_seconds(time.tick_since(started))) } } return code } +// The warnings, failures and missing packages of a report; 1 with any failure or package missing. +@(private="file") +library_print :: proc(report:^Library_Report,root:string,output,errors:io.Writer) -> int { + for warning in report.warnings {fmt.wprintfln(output,"warning: %s",warning)} + for failure in report.failures {fmt.wprintfln(errors,"mirai import: %s",failure)} + for missing in report.missing {fmt.wprintfln(errors,"mirai import: %s: %s",root,missing)} + return 1 if len(report.failures)>0 || len(report.missing)>0 else 0 +} + // The zones of a profiled import (importer.profile_totals), each with its // seconds summed over the threads and those on the thread that imports, and // the wall time no zone of that thread covers. diff --git a/cook/import/sources/cook_test.odin b/cook/import/sources/cook_test.odin index f08c2ba1b..633febe19 100644 --- a/cook/import/sources/cook_test.odin +++ b/cook/import/sources/cook_test.odin @@ -9,10 +9,11 @@ import "mirai:asset" import importer "mirai:cook/import" import "mirai:cook/build" import "mirai:std/base" +import "mirai:store" -// A texture the editor imported Fast ships Final: the sync of a build that +// A texture the editor imported Fast ships Final: the refresh of a build that // cooks builds it again from its source with the id it has, a cook without it -// says so, and the sync of a build that does not cook takes either as it is. +// says so, and the refresh of a build that does not cook takes either as it is. @(test) a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) @@ -31,12 +32,17 @@ a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { } source,final:=fmt.tprintf("%s/noise.tga",assets),fmt.tprintf("%s/final/noise.tga",assets) testing.expect(t,os.write_entire_file(source,image)==nil && os.write_entire_file(final,image)==nil) - imported :: proc(t:^testing.T,types:^asset.Types,source:string,options:importer.Import_Options) -> (id:base.Asset_Id,flags:asset.Texture_Flags,blocks:[]u8,path:string) { - report,error:=import_source(source,options) + s:=test_store(t,fmt.tprintf("%s/store",folder)) + if s==nil {return} + fast:=Library_Options{store=s,root=folder,fast_textures=true} + final_options:=fast + final_options.fast_textures=false + imported :: proc(t:^testing.T,types:^asset.Types,source:string,options:Library_Options) -> (id:base.Asset_Id,flags:asset.Texture_Flags,blocks:[]u8,manifest:store.Hash) { + report,error:=ingest(source,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None && len(report.assets)==1,"%v %s",error.code,error.message) {return} id,flags,blocks=level_of(t,types,report.assets[0].path) - return id,flags,blocks,strings.clone(report.assets[0].path,context.temp_allocator) + return id,flags,blocks,report.manifest } level_of :: proc(t:^testing.T,types:^asset.Types,path:string) -> (id:base.Asset_Id,flags:asset.Texture_Flags,blocks:[]u8) { a:asset.Asset @@ -45,30 +51,25 @@ a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { if !testing.expect_value(t,error,asset.Error.None) {return} return a.blob.header.id,view.info.flags,slice.clone(asset.texture_mip(view,0),context.temp_allocator) } - fast_id,fast_flags,fast_blocks,fast_path:=imported(t,&types,source,{fast_textures=true}) - _,final_flags,final_blocks,_:=imported(t,&types,final,{}) + fast_id,fast_flags,fast_blocks,manifest:=imported(t,&types,source,fast) + _,final_flags,final_blocks,_:=imported(t,&types,final,final_options) testing.expect(t,.Fast in fast_flags && .Fast not_in final_flags && !slice.equal(fast_blocks,final_blocks)) - manifest:=fmt.tprintf("%s/%s",folder,build.COOK_FILE) - testing.expect(t,os.write_entire_file(manifest,`{"roots":["noise.tex"]}`)==nil) + added:=project_add(folder,manifest,"",nil,fast,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) - testing.expect(t,strings.contains(fast_path,"/.mirai/imported/"),"the texture of a source of the project is of its imported folder, not of its assets") - - // The cook reads the folder as it is: it says who builds the texture Final. - report:=build.cook_assets({assets=assets,output=fmt.tprintf("%s/program",folder),manifest=manifest,root=folder,allocator=context.temp_allocator}) - testing.expectf(t,report.error==.Source_Failed && strings.contains(report.detail,"assets/noise.tga") && strings.contains(report.detail,"mirai build game"),"%v %s",report.error,report.detail) - - caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",folder))} - fast:=caches - fast.fast_textures=true - kept:=sync(folder,fast,context.temp_allocator) - testing.expectf(t,kept.imported==0 && kept.removed==0 && len(kept.failures)==0,"Fast is current for a build that does not cook: %v",kept) - - // The sync of a build that cooks does, with the id the texture has. - synced:=sync(folder,caches,context.temp_allocator) - testing.expectf(t,synced.imported==1 && len(synced.failures)==0,"%v",synced) - kept=sync(folder,fast,context.temp_allocator) - testing.expectf(t,kept.imported==0 && kept.removed==0 && len(kept.failures)==0,"Final is current for a build that does not cook: %v",kept) - report=build.cook_assets({assets=assets,output=fmt.tprintf("%s/game",folder),manifest=manifest,root=folder,allocator=context.temp_allocator}) + // The refresh of a build that does not cook takes Fast; that of one that cooks builds Final, with the id it has. + kept:=project_refresh(folder,fast,context.temp_allocator) + testing.expectf(t,kept.ingested==0 && len(kept.failures)==0 && len(kept.missing)==0,"Fast is current for a build that does not cook: %v",kept) + cook_manifest:=fmt.tprintf("%s/%s",folder,build.COOK_FILE) + testing.expect(t,os.write_entire_file(cook_manifest,`{"roots":["noise.tex"]}`)==nil) + // The cook reads the project as it is: it says who builds the texture Final. + report:=build.cook_assets({assets=assets,output=fmt.tprintf("%s/program",folder),manifest=cook_manifest,root=folder,store=s,allocator=context.temp_allocator}) + testing.expectf(t,report.error==.Source_Failed && strings.contains(report.detail,"noise") && strings.contains(report.detail,"mirai build game"),"%v %s",report.error,report.detail) + refreshed:=project_refresh(folder,final_options,context.temp_allocator) + testing.expectf(t,refreshed.ingested==1 && len(refreshed.failures)==0 && len(refreshed.missing)==0,"%v",refreshed) + kept=project_refresh(folder,fast,context.temp_allocator) + testing.expectf(t,kept.ingested==0 && len(kept.failures)==0,"Final is current for a build that does not cook: %v",kept) + report=build.cook_assets({assets=assets,output=fmt.tprintf("%s/game",folder),manifest=cook_manifest,root=folder,store=s,allocator=context.temp_allocator}) testing.expectf(t,report.error==.None,"%v %s",report.error,report.detail) shipped_id,shipped_flags,shipped_blocks:=level_of(t,&types,fmt.tprintf("%s/game/noise.tex",folder)) testing.expect(t,shipped_id==fast_id && .Fast not_in shipped_flags && slice.equal(shipped_blocks,final_blocks),"Final, with the id it had") diff --git a/cook/import/sources/fuzz_test.odin b/cook/import/sources/fuzz_test.odin index 14233c6e2..d360f5b4a 100644 --- a/cook/import/sources/fuzz_test.odin +++ b/cook/import/sources/fuzz_test.odin @@ -125,55 +125,25 @@ damaged_images_and_settings_are_imported_or_refused :: proc(t:^testing.T) { png:=png_bytes(8,8,{200,120,60},true) path:=fmt.tprintf("%s/subject.png",folder) imported,refused:=0,0 - attempt :: proc(path,root:string,data:[]u8,imported,refused:^int) { + options:=test_options(t,path,{root=root,image=.Color,ignore_stored=true}) + attempt :: proc(path:string,options:importer.Import_Options,data:[]u8,imported,refused:^int) { if os.write_entire_file(path,data)!=nil {return} - report,error:=importer.import_texture(path,{root=root,image=.Color,ignore_stored=true}) + report,error:=importer.import_texture(path,options) if error.code==.None { importer.report_destroy(&report) imported^+=1 } else {refused^+=1} } - attempt(path,root,png,&imported,&refused) + attempt(path,options,png,&imported,&refused) testing.expect_value(t,imported,1) - for length in 0..len(png)/2,"%d of %d damaged images are refused",refused,imported+refused) - - // The sidecar of a source: any text is the settings and the outputs it says or is refused, and never the defaults or other ids by accident. - wanted:=importer.Import_Settings{textures=.Uncompressed,rdo_lambda=30,image=.Color,alpha_coverage=0.25,lods=.None,front=.X} - listed:=make([dynamic]importer.Output,context.temp_allocator) - append(&listed,importer.Output{kind=asset.KIND_MESH,name="Crate \"big\"",index=2,id={3,4},part=true},importer.Output{kind=asset.KIND_MATERIAL,name="Wood",id={5,6}},importer.Output{kind=asset.KIND_TEXTURE,name="wood",id={7,8}}) - text:=importer.sidecar_text({settings=wanted,folder="crate",outputs=listed},context.temp_allocator) - parsed,valid:=importer.sidecar_parse(text,context.temp_allocator) - testing.expect(t,valid && parsed.settings==wanted && parsed.folder=="crate" && len(parsed.outputs)==3) - if valid && len(parsed.outputs)==3 { - for output,at in parsed.outputs {testing.expectf(t,output==listed[at],"output %d is read as written: %v",at,output)} - } - read,rejected:=0,0 - for length in 0..>33)%u64(len(work)))]=marks[int((state>>20)%u64(len(marks)))] - if changed,ok:=importer.sidecar_parse(string(work),context.temp_allocator);ok { - read+=1 - // What is read is settings an import can use, and outputs it can write. - changed_settings:=changed.settings - testing.expect(t,changed_settings.alpha_coverage>=0 && changed_settings.alpha_coverage<=1 && changed_settings.lods<=max(importer.Lods) && changed_settings.front<=max(importer.Front)) - for output in changed.outputs {testing.expect(t,output.id!={} && output.kind!=0,"an output has an id and a kind")} - } else {rejected+=1} - } - testing.expectf(t,rejected>len(text)/2 && read>0,"%d texts are sidecars and %d are refused",read,rejected) } // Fonts and manifests of prompts, cut at lengths spread over them or with @@ -191,26 +161,27 @@ damaged_fonts_and_prompts_are_imported_or_refused :: proc(t:^testing.T) { for subject in subjects { path:=fmt.tprintf("%s/%s",folder,subject.file) imported,refused:=0,0 - attempt :: proc(path,folder:string,data:[]u8,imported,refused:^int) { + options:=test_options(t,path) + attempt :: proc(path:string,options:importer.Import_Options,data:[]u8,imported,refused:^int) { if os.write_entire_file(path,data)!=nil {return} - report,error:=import_source(path,{}) + report,error:=import_source(path,options) if error.code==.None { importer.report_destroy(&report) imported^+=1 } else {refused^+=1} } - attempt(path,folder,subject.data,&imported,&refused) + attempt(path,options,subject.data,&imported,&refused) testing.expectf(t,imported==1,"%s as it is is imported",subject.file) step:=max(len(subject.data)/200,1) - for length:=0;length0 && imported>1,"%s: %d damaged files imported and %d refused",subject.file,imported-1,refused) } diff --git a/cook/import/sources/library.odin b/cook/import/sources/library.odin new file mode 100644 index 000000000..4b86b360f --- /dev/null +++ b/cook/import/sources/library.odin @@ -0,0 +1,614 @@ +package sources + +import "core:fmt" +import "core:io" +import "core:mem" +import "core:os" +import "core:path/filepath" +import "core:strings" +import "core:time" +import "mirai:asset" +import "mirai:core" +import "mirai:cook/cache" +import importer "mirai:cook/import" +import "mirai:cook/workspace" +import "mirai:project" +import "mirai:std/base" +import "mirai:std/jobs" +import "mirai:store" + +// The library of the machine: the packages its store keeps (asset/packages.odin) +// and what a project does with them. +// +// - ingest: a source anywhere on disk becomes a package of the store, where +// it is; nothing is copied into a project. Ingesting the same origin again +// finds its package (asset.REF_ORIGINS) and keeps its ids; one whose +// identity and build are current is not imported again. Two processes that +// ingest one origin at once take turns (a lock per origin), and the second +// finds the work done. +// - project_add: pins a package in the project's packages.mirai +// (project_pin, its only writer) and follows it into the project. +// - project_refresh: for each package a project pins, the build this engine +// reads, from the local refs, else from the caller's source, else by +// ingesting its origin when it is on this machine (also when the source +// changed); else the report says, by package name, that a machine with its +// source must ingest it. It never removes a blob, a ref or a file, and never +// builds without the source. +// +// Following a package writes or follows its documents in the project's folder +// of it: from the build's baselines (importer.documents_follow), and for a +// model whose source is here, by importing it with the project's folder, +// which also writes its prefab in the game's types (prefab.model_prefab). + +// What the library of the editor of the project at root gives its imports: +// the formats of models and the importers of whole sources its plugins add, +// with what they read prepared for its assets, and the types of its game, +// which the prefab of a model is written in; release(state) lets them go. +// problem: why there are none. Only a program that links the editor reads it +// (editor.import_library). +Library :: #type proc(root:string,allocator:mem.Allocator) -> Library_Imports +Library_Imports :: struct { + formats:[]importer.Format, + wholes:[]Whole_Source, + world:^core.World_Info, + release:proc(state:rawptr), + state:rawptr, + problem:string, +} + +Library_Options :: struct { + store:^store.Store, + // Asked for what the store does not have before building: a manifest by + // (asset.REF_PACKAGES, ), a build by (asset.REF_IMPORTS, ) with + // every blob it lists, a result of cook/cache. + source:cache.Source, + // Fast textures allowed: imports build them Fast unless the texture cache + // has them Final, and a Fast build is taken; false: Final required, what a + // cook ships. + fast_textures:bool, + rebuild:bool, // ingest: imports the source again, also when it is current + // The project or plugin the library works for: its plugins read sources + // (plugins) and its game's types hold prefabs. "": the formats of the engine alone. + root:string, + plugins:Library, // nil: no formats of plugins, and no prefabs of models + worker:jobs.Worker_Ref, + progress:io.Writer, // where each source imported is said as it goes; nil: nowhere +} + +Library_Report :: struct { + packages:int, // pinned by the project + ingested:int, // imported now + fetched:int, // builds the source gave + missing:[dynamic]string, // a package with no build here and no source on this machine, by name, and what gives it + failures:[dynamic]string, // what could not be read or imported, with the reason + warnings:[dynamic]string, + allocator:mem.Allocator, +} + +library_report_destroy :: proc(report:^Library_Report) { + for text in report.missing {delete(text,report.allocator)} + for text in report.failures {delete(text,report.allocator)} + for text in report.warnings {delete(text,report.allocator)} + delete(report.missing) + delete(report.failures) + delete(report.warnings) + report^={} +} + +// Ingests the source at origin, a file or a folder anywhere on disk, as a +// package of options.store. settings, when given, replace those its package +// keeps. A source whose package is current (same identity, a build this +// engine reads) is not imported: the report names its package and lists no +// assets. In allocator. +ingest :: proc(origin:string,options:Library_Options,settings:^importer.Import_Settings=nil,allocator:=context.allocator) -> (report:importer.Import_Report,error:importer.Error) { + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + return library_ingest(&l,origin,{},settings,"",allocator) +} + +// Pins the package whose manifest the store keeps under manifest in the +// project at root, its cache listed under folder of its assets, using assets +// (none: all), and follows it there (project_refresh). In allocator. +project_add :: proc(root:string,manifest:store.Hash,folder:string,assets:[]base.Asset_Id,options:Library_Options,allocator:=context.allocator) -> (report:Library_Report) { + report_begin(&report,allocator) + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + context.temp_allocator=l.temp + top:=project.path_absolute(root,context.temp_allocator) + digits:[64]u8 + p,error:=asset.package_get(options.store,manifest,context.temp_allocator) + if error!=.None { + report_line(&report.failures,"%s: the manifest %s is not a package in the store at %s (%v): ingest its source, or fetch it from a peer that has it",top,store.hex(manifest,&digits),options.store.root,error) + return + } + entry:=asset.Project_Package{id=p.id,manifest=manifest,folder=folder} + entry.assets=make([dynamic]base.Asset_Id,context.temp_allocator) + for id in assets { + known:=false + for &a in p.assets {known||=a.id==id} + if !known { + report_line(&report.failures,"%s: package %s has no asset %v: use one its manifest lists",top,p.name,id) + return + } + append(&entry.assets,id) + } + if pin_error:=project_pin(top,entry);pin_error!=.None { + report_line(&report.failures,"%s: %s could not be written (%v): a folder is relative to the assets, with forward slashes, and never leaves them",top,asset.PACKAGES_FILE,pin_error) + return + } + report.packages=1 + project_follow_begin(&l,top) + package_refresh(&l,top,entry,&report,true) + return +} + +// Ingests the source at origin as ingest does and pins its package in the +// project at root under folder, with what the project used of it before: one +// import, which for a model writes its documents and its prefab there. The +// report of the import and that of the project, in allocator. +project_ingest :: proc(root,origin,folder:string,options:Library_Options,settings:^importer.Import_Settings=nil,allocator:=context.allocator) -> (report:importer.Import_Report,added:Library_Report,error:importer.Error) { + report_begin(&added,allocator) + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + caller:=context.temp_allocator + context.temp_allocator=l.temp + top:=project.path_absolute(root,context.temp_allocator) + project_follow_begin(&l,top) + entry:=asset.Project_Package{folder=folder} + documents:=l.assets if folder=="" else fmt.tprintf("%s/%s",l.assets,folder) + r,_:=reader_of(&l,importer.path_key(origin)) + model:=r.kind==.Model && plugins_world(&l)!=nil + report,error=library_ingest(&l,origin,{},settings,documents if model else "",allocator) + if error.code!=.None { + error.message=strings.clone(error.message,caller) + return + } + entry.id,entry.manifest=report.package_id,report.manifest + list,_,_:=asset.packages_load(top,context.temp_allocator) + for &other in list.packages { + if other.id==entry.id {entry.assets=other.assets} + } + if pin_error:=project_pin(top,entry);pin_error!=.None { + report_line(&added.failures,"%s: %s could not be written (%v): a folder is relative to the assets, with forward slashes, and never leaves them",top,asset.PACKAGES_FILE,pin_error) + return + } + added.packages=1 + if !model {package_refresh(&l,top,entry,&added,true)} + return +} + +// Brings the packages the project at root pins to builds this engine reads, +// and its documents along. The report, in allocator, says what was ingested +// or fetched, and by package name what no source here and no peer gave. +project_refresh :: proc(root:string,options:Library_Options,allocator:=context.allocator) -> (report:Library_Report) { + report_begin(&report,allocator) + l:Library_State + library_begin(&l,options,allocator) + defer library_end(&l) + context.temp_allocator=l.temp + top:=project.path_absolute(root,context.temp_allocator) + list,_,error:=asset.packages_load(top,context.temp_allocator) + if error!=.None { + report_line(&report.failures,"%s/%s does not read (%v): fix it from its history; nothing was refreshed",top,asset.PACKAGES_FILE,error) + return + } + report.packages=len(list.packages) + if len(list.packages)==0 {return} + project_follow_begin(&l,top) + for &entry,at in list.packages { + if options.progress.procedure!=nil {fmt.wprintfln(options.progress,"package %d of %d",at+1,len(list.packages))} + package_refresh(&l,top,entry,&report,false) + } + return +} + +// Pins entry in the packages.mirai of the project at root, in place of the +// entry of the same package: the only writer of that file. +project_pin :: proc(root:string,entry:asset.Project_Package) -> asset.Error { + list,_,error:=asset.packages_load(root,context.temp_allocator) + if error!=.None {return error} + for &other in list.packages { + if other.id==entry.id { + other=entry + return asset.packages_save(root,&list) + } + } + if _,no_room:=append(&list.packages,entry);no_room!=nil {return .Out_Of_Memory} + return asset.packages_save(root,&list) +} + +// What ingest, project_add and project_refresh share for one call. +@(private="file") +Library_State :: struct { + options:Library_Options, + scratch:mem.Dynamic_Arena, + temp:mem.Allocator, + plugins:Library_Imports, + plugins_read:bool, + formats:[]importer.Format, // FORMATS and those of the plugins, once read + assets:string, // of the project followed + registry:^asset.Registry, // its assets, scanned once +} + +@(private="file") +library_begin :: proc(l:^Library_State,options:Library_Options,allocator:mem.Allocator) { + l.options=options + mem.dynamic_arena_init(&l.scratch,allocator,allocator,block_size=1<<20,out_band_size=1<<20) + l.temp=base.dynamic_arena_allocator(&l.scratch) + l.formats=FORMATS[:] +} + +@(private="file") +library_end :: proc(l:^Library_State) { + if l.plugins.release!=nil {l.plugins.release(l.plugins.state)} + mem.dynamic_arena_destroy(&l.scratch) +} + +// The formats, the whole importers and the game types of the plugins of the +// root of the library, read the first time a source needs them. +@(private="file") +plugins_read :: proc(l:^Library_State) { + if l.plugins_read {return} + l.plugins_read=true + if l.options.plugins==nil || l.options.root=="" {return} + l.plugins=l.options.plugins(l.options.root,l.temp) + all:=make([dynamic]importer.Format,0,len(FORMATS)+len(l.plugins.formats),l.temp) + append(&all,..FORMATS[:]) + append(&all,..l.plugins.formats) + l.formats=all[:] +} + +@(private="file") +project_follow_begin :: proc(l:^Library_State,top:string) { + if l.options.root=="" {l.options.root=top} + l.assets=project.folder_path(top,.Assets,l.temp) + l.registry=new(asset.Registry,l.temp) + asset.registry_init(l.registry,l.assets,"",l.temp) + asset.registry_scan(l.registry) +} + +// What reads the source at origin: kind (None for one a plugin imports +// whole), the formats it is read with and the whole importer that claims it. +@(private="file") +Source_Reader :: struct { + kind:Source_Kind, + whole:Whole_Source, + claimed:bool, +} + +@(private="file") +reader_of :: proc(l:^Library_State,origin:string) -> (r:Source_Reader,ok:bool) { + r.kind=source_kind(origin,l.formats) + if r.kind==.None || r.kind==.Model { + plugins_read(l) + if r.whole,r.claimed=whole_find(l.plugins.wholes,origin);r.claimed {return {kind=.None,whole=r.whole,claimed=true},true} + r.kind=source_kind(origin,l.formats) + } + return r,r.kind!=.None +} + +// The key of the build of p this engine reads (asset.import_key): the +// versions of the engine with that of what reads its origin, an engine +// importer or a format of the engine or of the plugins. False for a source a +// plugin imports whole, whose version only its importer knows. +@(private="file") +package_key :: proc(l:^Library_State,p:^asset.Package) -> (key:store.Hash,ok:bool) { + version:u32 + switch p.origin.importer { + case "image","sound","font","prompts": + version=importer.import_version(0) + case: + found:=false + for format in FORMATS { + if format.name==p.origin.importer {version,found=importer.import_version(format.version),true} + } + if !found { + plugins_read(l) + for format in l.formats { + if format.name==p.origin.importer {version,found=importer.import_version(format.version),true} + } + } + if !found {return} + } + versions:=importer.import_versions(version) + return asset.import_key(p.identity,versions[:]) +} + +@(private="file") +Found :: struct { + build:asset.Build, + hash:store.Hash, + final:bool, + fast_here:bool, // a Fast build is here, which a Final one was required over +} + +// The build of p under key this store has whole: Final, or Fast when Fast +// textures are allowed. +@(private="file") +build_here :: proc(l:^Library_State,p:^asset.Package,key:store.Hash) -> (f:Found,ok:bool) { + s:=l.options.store + digits:[64]u8 + name:=store.hex(key,&digits) + for namespace,at in ([2]string{asset.REF_IMPORTS,asset.REF_IMPORTS_FAST}) { + hash,ref_error:=store.ref_get(s,namespace,name) + if ref_error!=.None {continue} + b,error:=asset.build_get(s,hash,l.temp) + if error!=.None || b.package_id!=p.id || b.key!=key {continue} + whole:=true + for o in b.outputs {whole&&=store.has(s,o.blob)} + if !whole {continue} + if at==1 && !l.options.fast_textures { + f.fast_here=true + continue + } + return {build=b,hash=hash,final=at==0},true + } + return +} + +// The build of p this engine reads here: by its key, or for a source a plugin +// imports whole, whose key only its importer knows, the one this machine +// chose for its manifest (asset.REF_BUILDS) when it is whole. +@(private="file") +build_found :: proc(l:^Library_State,p:^asset.Package,manifest,key:store.Hash,keyed:bool) -> (f:Found,ok:bool) { + if keyed {return build_here(l,p,key)} + s:=l.options.store + digits:[64]u8 + hash,ref_error:=store.ref_get(s,asset.REF_BUILDS,store.hex(asset.build_select(manifest,p),&digits)) + if ref_error!=.None {return} + b,error:=asset.build_get(s,hash,l.temp) + if error!=.None || b.package_id!=p.id {return} + for o in b.outputs { + if !store.has(s,o.blob) {return} + } + if !b.final && !l.options.fast_textures {return {fast_here=true},false} + return {build=b,hash=hash,final=b.final},true +} + +// Asks the source of the library for the ref (namespace, name). +@(private="file") +fetch :: proc(l:^Library_State,namespace,name:string) -> bool { + source:=l.options.source + return source.fetch!=nil && source.fetch(source.user,l.options.store,namespace,name) +} + +// The import of the source at origin into the store, with its lock: what +// ingest does. manifest: the package it was ingested as (zero: the one this +// machine names for origin). documents: the folder of the project the +// documents and the prefab of the import are written or followed in ("": +// none), with the registry and the game types of the library. +@(private="file") +library_ingest :: proc(l:^Library_State,path:string,manifest:store.Hash,settings:^importer.Import_Settings,documents:string,allocator:mem.Allocator) -> (report:importer.Import_Report,error:importer.Error) { + caller:=context.temp_allocator + context.temp_allocator=l.temp + s:=l.options.store + if s==nil {return {},importer.error_make(.Write,"%s: the library keeps packages in a store: give it one",path,allocator=caller)} + if importer.unsafe_path(path) || !os.exists(path) {return {},importer.error_make(.Not_Found,"%s does not exist",path,allocator=caller)} + origin:=importer.path_key(path) + r,known:=reader_of(l,origin) + if !known { + return {},importer.error_make(.Unsupported,"%s is not a source Mirai imports: a model (%s), an image, a sound (WAV, Ogg Vorbis), a font (TrueType, OpenType) or a manifest of prompts (%s); a format a plugin adds imports with the plugin in the project%s", + origin,importer.format_names(l.formats),importer.PROMPTS_MANIFEST,l.plugins.problem!=""?fmt.tprintf(" (%s)",l.plugins.problem):"",allocator=caller) + } + name:[64]u8 + lock,locked:=workspace.lock_acquire(fmt.tprintf("%s/locks/%s",s.root,asset.origin_name(origin,&name)),fmt.tprintf("mirai ingest %s",origin)) + defer if locked {workspace.lock_release(&lock)} + hashes:=importer.source_hashes_load(s,origin,l.temp) + defer importer.source_hashes_save(s,origin,&hashes) + code:=r.whole.code if r.claimed else 0 + // Current: the work is done, here or by another process that held the lock. + if !l.options.rebuild && settings==nil && documents=="" { + if p,hash,found,_:=importer.package_previous(s,manifest,origin,l.temp);found { + kept,valid:=importer.settings_parse(p.settings) + if valid && importer.identity_of(p.id,origin,p.inputs[:],kept,code,&hashes)==p.identity { + key,keyed:=package_key(l,&p) + if f,here:=build_found(l,&p,hash,key,keyed);here { + { + importer.report_init(&report,allocator) + report.package_id,report.manifest,report.build,report.key,report.final=p.id,hash,f.hash,key,f.final + report.settings=kept + return + } + } + } + } + } + options:=importer.Import_Options{ + store=s,cache_source=l.options.source,manifest=manifest,documents=documents,root=l.options.root, + rebuild=l.options.rebuild,fast_textures=l.options.fast_textures,worker=l.options.worker,hashes=&hashes, + } + if documents!="" {options.registry,options.world=l.registry,l.plugins.world} + if options.root=="" {options.root,_=source_root(origin,l.temp)} + if settings!=nil {options.settings,options.ignore_stored=settings^,true} + progress:importer.Progress + if l.options.progress.procedure!=nil { + importer.progress_begin(&progress,l.options.progress,filepath.base(origin)) + options.progress=&progress + } + wholes:[]Whole_Source + if r.claimed {wholes=l.plugins.wholes} + report,error=import_source(origin,options,l.formats,wholes,allocator) + if error.code!=.None {error.message=strings.clone(error.message,caller)} + return +} + +// One package a project pins, brought to a build this engine reads and +// followed into the project: when its build changed, or always with follow. +@(private="file") +package_refresh :: proc(l:^Library_State,top:string,entry:asset.Project_Package,report:^Library_Report,follow:bool) { + s:=l.options.store + entry:=entry + digits,id_text:[64]u8 + id_name:=base.asset_id_format(entry.id,id_text[:32]) + p,error:=asset.package_get(s,entry.manifest,l.temp) + if error==.Not_Found && fetch(l,asset.REF_PACKAGES,id_name) {p,error=asset.package_get(s,entry.manifest,l.temp)} + if error==.None && p.id!=entry.id {error=.Corrupt} + #partial switch error { + case .None: + case .Not_Found: + report_line(&report.missing,"package %s: the store at %s has not its manifest %s, and no peer gave it: fetch it from a machine that ingested it",id_name,s.root,store.hex(entry.manifest,&digits)) + return + case: + report_line(&report.failures,"package %s: its manifest %s in the store at %s is not a package of this version (%v): ingest its source again",id_name,store.hex(entry.manifest,&digits),s.root,error) + return + } + origin_here:=p.origin.path!="" && !importer.unsafe_path(p.origin.path) && os.exists(p.origin.path) + documents:=l.assets if entry.folder=="" else fmt.tprintf("%s/%s",l.assets,entry.folder) + prefab:=model_prefab_of(l,&p) + // A source here that changed is ingested again, keeping its ids, and the project follows it. + if origin_here && source_changed(l,&p) { + reingest(l,top,&entry,&p,documents,prefab,"its source changed",report) + return + } + key,keyed:=package_key(l,&p) + f:Found + found:bool + f,found=build_found(l,&p,entry.manifest,key,keyed) + if !found && keyed && fetch(l,asset.REF_IMPORTS,store.hex(key,&digits)) { + if f,found=build_here(l,&p,key);found {report.fetched+=1} + } + if !found { + if origin_here { + reingest(l,top,&entry,&p,documents,prefab,"this engine reads no build of it here",report) + return + } + switch { + case !keyed: + report_line(&report.missing,"package %s: a plugin imports it whole, and only a machine with its source (%s) can build it: ingest it there with this engine",p.name,p.origin.path) + case f.fast_here: + report_line(&report.missing,"package %s: only a build with Fast textures is here, and Final is required: a machine with its source (%s) must ingest it with this engine, or a peer that has its Final build (import key %s) must give it",p.name,p.origin.path,store.hex(key,&digits)) + case: + report_line(&report.missing,"package %s: no build of it for this engine (import key %s) is here or at a peer: a machine with its source (%s) must ingest it with this engine",p.name,store.hex(key,&digits),p.origin.path) + } + return + } + select:=store.hex(asset.build_select(entry.manifest,&p),&digits) + previous,_:=store.ref_get(s,asset.REF_BUILDS,select) + if previous!=f.hash { + if ref_error:=store.ref_set(s,asset.REF_BUILDS,select,f.hash);ref_error!=.None { + report_line(&report.failures,"package %s: the ref %s/%s could not be set in the store at %s: %v",p.name,asset.REF_BUILDS,select,s.root,ref_error) + return + } + } + if !follow && previous==f.hash {return} + // A model's prefab is written from its source in the game's types: the import does it, with the documents. + if prefab && origin_here && plugins_world(l)!=nil { + reingest(l,top,&entry,&p,documents,prefab,"it is followed into the project",report) + return + } + documents_write(l,&p,&f.build,documents,report) + if prefab && follow { + if _,has:=asset.registry_find(l.registry,prefab_id(&p));!has { + report_line(&report.warnings,"package %s: its prefab is written from its source (%s) in the types of the game: add it on a machine that has the source, with the project's editor built",p.name,p.origin.path) + } + } +} + +// The types of the game of the library's root, read once. +@(private="file") +plugins_world :: proc(l:^Library_State) -> ^core.World_Info { + plugins_read(l) + return l.plugins.world +} + +@(private="file") +documents_write :: proc(l:^Library_State,p:^asset.Package,b:^asset.Build,documents:string,report:^Library_Report) { + written,error:=importer.documents_follow(l.options.store,p,b,documents,l.registry,l.temp) + if error.code!=.None { + report_line(&report.failures,"package %s: %s",p.name,error.message) + return + } + for w in written { + if w.warning!="" {report_line(&report.warnings,"package %s: %s",p.name,w.warning)} + } +} + +// Whether what the source of p is made from now is not what p was ingested +// from: its bytes, those of its inputs, or the code of the plugin that +// imports it whole. +@(private="file") +source_changed :: proc(l:^Library_State,p:^asset.Package) -> bool { + settings,valid:=importer.settings_parse(p.settings) + if !valid {return false} + code:u64 + if r,known:=reader_of(l,p.origin.path);known && r.claimed {code=r.whole.code} + hashes:=importer.source_hashes_load(l.options.store,p.origin.path,l.temp) + defer importer.source_hashes_save(l.options.store,p.origin.path,&hashes) + return importer.identity_of(p.id,p.origin.path,p.inputs[:],settings,code,&hashes)!=p.identity +} + +// Ingests the source of p again, keeping its ids, and pins the manifest it +// gives; a model is imported with the project's folder, which follows its +// documents and writes its prefab, and any other follows from its build. +@(private="file") +reingest :: proc(l:^Library_State,top:string,entry:^asset.Project_Package,p:^asset.Package,documents:string,prefab:bool,why:string,report:^Library_Report) { + if l.options.progress.procedure!=nil {fmt.wprintfln(l.options.progress,"importing %s: %s",p.origin.path,why)} + started:=time.tick_now() + model:=prefab && plugins_world(l)!=nil + written,error:=library_ingest(l,p.origin.path,entry.manifest,nil,documents if model else "",l.temp) + if error.code!=.None { + report_line(&report.failures,"package %s: %s",p.name,error.message) + return + } + defer importer.report_destroy(&written) + report.ingested+=1 + for warning in written.warnings {report_line(&report.warnings,"package %s: %s",p.name,warning)} + if l.options.progress.procedure!=nil {fmt.wprintfln(l.options.progress," %s: %d assets, %.1f s",p.name,len(written.assets),time.duration_seconds(time.tick_since(started)))} + if written.manifest!=entry.manifest { + entry.manifest=written.manifest + if pin_error:=project_pin(top,entry^);pin_error!=.None { + report_line(&report.failures,"package %s: %s/%s could not be written: %v",p.name,top,asset.PACKAGES_FILE,pin_error) + return + } + } + if model {return} + again,read_error:=asset.package_get(l.options.store,written.manifest,l.temp) + b,build_error:=asset.build_get(l.options.store,written.build,l.temp) + if read_error!=.None || build_error!=.None { + report_line(&report.failures,"package %s: the store at %s does not read what its import put (%v, %v)",p.name,l.options.store.root,read_error,build_error) + return + } + documents_write(l,&again,&b,documents,report) + if prefab { + if _,has:=asset.registry_find(l.registry,prefab_id(&again));!has { + report_line(&report.warnings,"package %s: its prefab is written in the types of the game, which the library of the project's editor has: %s",p.name,"build the project's editor, then refresh") + } + } +} + +// Whether p is a model whose prefab its import writes from the source: one a +// format reads, with a scene its build has no baseline of. +@(private="file") +model_prefab_of :: proc(l:^Library_State,p:^asset.Package) -> bool { + if prefab_id(p)=={} {return false} + for format in FORMATS { + if format.name==p.origin.importer {return true} + } + plugins_read(l) + for format in l.formats { + if format.name==p.origin.importer {return true} + } + return false +} + +@(private="file") +prefab_id :: proc(p:^asset.Package) -> base.Asset_Id { + for &a in p.assets { + if a.kind==asset.KIND_SCENE {return a.id} + } + return {} +} + +@(private="file") +report_begin :: proc(report:^Library_Report,allocator:mem.Allocator) { + report.allocator=allocator + report.missing.allocator,report.failures.allocator,report.warnings.allocator=allocator,allocator,allocator +} + +@(private="file") +report_line :: proc(lines:^[dynamic]string,format:string,args:..any) { + append(lines,fmt.aprintf(format,..args,allocator=lines.allocator)) +} diff --git a/cook/import/sources/library_test.odin b/cook/import/sources/library_test.odin new file mode 100644 index 000000000..4839dbebb --- /dev/null +++ b/cook/import/sources/library_test.odin @@ -0,0 +1,343 @@ +package sources + +import "core:fmt" +import "core:os" +import "core:path/filepath" +import "core:strings" +import "core:sync" +import "core:testing" +import "core:thread" +import "mirai:asset" +import "mirai:cook/prefab" +import importer "mirai:cook/import" +import "mirai:std/base" +import "mirai:store" + +// The files under folder, by their paths under it, in the temp allocator. +@(private="file") +files_of :: proc(folder:string) -> map[string]bool { + files:=make(map[string]bool,context.temp_allocator) + todo:=make([dynamic]string,context.temp_allocator) + append(&todo,folder) + for next:=0;next (listed:map[string]asset.Package_Listed) { + listed=make(map[string]asset.Package_Listed,context.temp_allocator) + list,_,error:=asset.packages_load(root,context.temp_allocator) + if !testing.expect_value(t,error,asset.Error.None,loc=loc) {return} + resolved,resolve_error:=asset.packages_resolve(s,&list,context.temp_allocator) + if !testing.expect_value(t,resolve_error,asset.Error.None,loc=loc) {return} + testing.expectf(t,len(resolved.problems)==0,"%v",resolved.problems[:],loc=loc) + for l in resolved.listed {listed[l.path]=l} + return +} + +// Every ref of the namespaces of builds of s, removed: what an engine whose +// versions changed finds of them. +@(private="file") +builds_forget :: proc(s:^store.Store) { + for namespace in ([3]string{asset.REF_IMPORTS,asset.REF_IMPORTS_FAST,asset.REF_BUILDS}) { + refs,_:=store.ref_list(s,namespace,context.temp_allocator) + for ref in refs {store.ref_remove(s,namespace,ref.name)} + } +} + +@(private="file") +report_clean :: proc(t:^testing.T,report:Library_Report,loc:=#caller_location) -> bool { + return testing.expectf(t,len(report.failures)==0 && len(report.missing)==0,"failures %v, missing %v",report.failures[:],report.missing[:],loc=loc) +} + +@(private="file") +ingested :: proc(t:^testing.T,source:string,options:Library_Options,loc:=#caller_location) -> (report:importer.Import_Report,ok:bool) { + error:importer.Error + report,error=ingest(source,options,allocator=context.temp_allocator) + ok=testing.expectf(t,error.code==.None,"%s: %s",source,error.message,loc=loc) + return +} + +// A project pins its packages by manifest: a peer with the project and none +// of its sources fetches their builds from a store that has them, lists and +// loads them, and imports nothing. Without such a store it says, for each +// package, what gives it, and neither removes a file nor builds anything. +@(test) +a_peer_without_sources_lists_and_loads_and_nothing_is_removed_or_built :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-peer") + if top=="" {return} + context.user_ptr=prefab.import_world(t) + downloads,game:=fmt.tprintf("%s/downloads",top),fmt.tprintf("%s/game",top) + os.make_directory_all(downloads) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + box:=box_source(downloads,"box.gltf","Wood","Paint") + sign:=fmt.tprintf("%s/sign.png",downloads) + _=os.write_entire_file(sign,png_bytes(8,8,{200,40,40})) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + options:=Library_Options{store=here,root=game,plugins=test_library} + for source in ([2]string{box,sign}) { + report,ok:=ingested(t,source,options) + if !ok {return} + added:=project_add(game,report.manifest,"props",nil,options,context.temp_allocator) + if !report_clean(t,added) {return} + } + listed:=listed_of(t,here,game) + _,has_mesh:=listed["props/Box.mesh"] + _,has_sign:=listed["props/sign.tex"] + testing.expectf(t,has_mesh && has_sign,"the cache of both is listed under their folder: %v",listed) + + // A peer: the project, not its sources, and a store of its own. + testing.expect(t,os.rename(downloads,fmt.tprintf("%s/moved",top))==nil) + peer:=fmt.tprintf("%s/peer",top) + folder_copy(game,peer) + there:=test_store(t,fmt.tprintf("%s/there",top)) + if there==nil {return} + alone:=options + alone.store,alone.root=there,peer + before:=files_of(peer) + lacking:=project_refresh(peer,alone,context.temp_allocator) + testing.expectf(t,len(lacking.missing)==2 && len(lacking.failures)==0 && lacking.ingested==0 && lacking.fetched==0,"%v",lacking) + list,_,_:=asset.packages_load(peer,context.temp_allocator) + for entry in list.packages { + id:[base.ASSET_ID_TEXT]u8 + named:=false + for line in lacking.missing {named||=strings.contains(line,base.asset_id_format(entry.id,id[:]))} + testing.expectf(t,named,"the package %v is named: %v",entry.id,lacking.missing[:]) + } + builds,_:=store.ref_list(there,asset.REF_BUILDS,context.temp_allocator) + testing.expect(t,len(builds)==0,"nothing is built") + after:=files_of(peer) + testing.expect_value(t,len(after),len(before)) + for file in before {testing.expectf(t,after[file],"%s stays",file)} + + // With a store that has them, the builds are fetched: listed where the project lists them, loaded. + fetching:=alone + fetching.source=copy_source(here) + fetched:=project_refresh(peer,fetching,context.temp_allocator) + testing.expectf(t,fetched.fetched==2 && fetched.ingested==0,"%v",fetched) + if !report_clean(t,fetched) {return} + seen:=listed_of(t,there,peer) + testing.expect_value(t,len(seen),len(listed)) + for path,l in listed { + other,found:=seen[path] + if !testing.expectf(t,found && other.id==l.id && other.state==.Built,"%s is listed with its id",path) {continue} + a:asset.Asset + if testing.expectf(t,asset.asset_load(&a,other.file,context.temp_allocator)==.None,"%s loads",path) { + testing.expect_value(t,a.blob.header.id,l.id) + asset.asset_close(&a) + } + } + for file in before {testing.expectf(t,os.exists(fmt.tprintf("%s%s",peer,file)),"%s stays",file)} + again:=project_refresh(peer,alone,context.temp_allocator) + testing.expectf(t,again.fetched==0 && again.ingested==0,"then nothing is asked for: %v",again) + report_clean(t,again) +} + +// A source outside any project is ingested where it is: nothing is written +// beside it, two projects that add it list the same blobs, ingesting it again +// imports nothing, and once it changed it is another manifest of the same +// package with the same ids, which a project follows on its refresh. +@(test) +a_source_outside_any_project_ingests_and_two_projects_use_the_same_blobs :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-outside") + if top=="" {return} + context.user_ptr=prefab.import_world(t) + downloads:=fmt.tprintf("%s/downloads",top) + os.make_directory_all(downloads) + box:=box_source(downloads,"box.gltf","Wood","Paint") + s:=test_store(t,fmt.tprintf("%s/store",top)) + if s==nil {return} + options:=Library_Options{store=s,plugins=test_library} + first,ok:=ingested(t,box,options) + if !ok {return} + testing.expect(t,len(first.assets)>0 && first.prefab_path=="","its cache and documents are blobs; the prefab is a project's") + testing.expect_value(t,len(files_of(downloads)),2) + again,again_ok:=ingested(t,box,options) + testing.expect(t,again_ok && len(again.assets)==0 && again.manifest==first.manifest && again.package_id==first.package_id,"current: nothing is imported again") + + projects:=[2]string{fmt.tprintf("%s/one",top),fmt.tprintf("%s/two",top)} + lists:[2]map[string]asset.Package_Listed + for project,at in projects { + os.make_directory_all(fmt.tprintf("%s/assets",project)) + o:=options + o.root=project + if !report_clean(t,project_add(project,first.manifest,"props",nil,o,context.temp_allocator)) {return} + lists[at]=listed_of(t,s,project) + p,_:=asset.package_get(s,first.manifest,context.temp_allocator) + r:asset.Registry + asset.registry_init(&r,fmt.tprintf("%s/assets",project),"",context.temp_allocator) + asset.registry_scan(&r) + for a in p.assets { + if asset.import_cached(a.kind) {continue} + _,found:=asset.registry_find(&r,a.id) + testing.expectf(t,found,"%s: the document %s, the prefab among them, is a file of the project with its id",project,a.name) + } + } + testing.expect(t,len(lists[0])>0 && len(lists[0])==len(lists[1])) + for path,l in lists[0] {testing.expectf(t,lists[1][path].file==l.file,"%s is the same blob for both",path)} + + box_source(downloads,"box.gltf","Wood","Paint","0.9") + changed,changed_ok:=ingested(t,box,options) + if !changed_ok {return} + testing.expect(t,changed.manifest!=first.manifest && changed.package_id==first.package_id,"a changed source is another manifest of its package") + was,_:=asset.package_get(s,first.manifest,context.temp_allocator) + now,_:=asset.package_get(s,changed.manifest,context.temp_allocator) + testing.expect_value(t,len(now.assets),len(was.assets)) + for a in was.assets { + kept:=false + for b in now.assets {kept||=b.id==a.id && b.kind==a.kind && b.name==a.name} + testing.expectf(t,kept,"%s keeps its id",a.name) + } + o:=options + o.root=projects[0] + followed:=project_refresh(projects[0],o,context.temp_allocator) + report_clean(t,followed) + list,_,_:=asset.packages_load(projects[0],context.temp_allocator) + testing.expect(t,len(list.packages)==1 && list.packages[0].manifest==changed.manifest,"the project pins what its source is now") +} + +// When the versions of the engine change, no build of a package is current: +// a machine with its source ingests it again, and one without it says which +// package needs it, by name. A build with Fast textures is not one a cook +// takes: where Final is required and only Fast is here, that is said too. +@(test) +an_engine_change_builds_again_where_the_source_is_and_is_reported_by_name_elsewhere :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-engine") + if top=="" {return} + downloads,game:=fmt.tprintf("%s/downloads",top),fmt.tprintf("%s/game",top) + os.make_directory_all(downloads) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + sign:=fmt.tprintf("%s/billboard.png",downloads) + _=os.write_entire_file(sign,png_bytes(8,8,{200,40,40})) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + options:=Library_Options{store=here,root=game} + first,ok:=ingested(t,sign,options) + if !ok || !report_clean(t,project_add(game,first.manifest,"",nil,options,context.temp_allocator)) {return} + current:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,current.ingested==0 && current.fetched==0,"a current package is left alone: %v",current) + + builds_forget(here) + rebuilt:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,rebuilt.ingested==1,"%v",rebuilt) + report_clean(t,rebuilt) + _,has:=listed_of(t,here,game)["billboard.tex"] + testing.expect(t,has) + + testing.expect(t,os.rename(downloads,fmt.tprintf("%s/moved",top))==nil) + builds_forget(here) + lacking:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,lacking.ingested==0 && len(lacking.missing)==1 && strings.contains(lacking.missing[0],"package billboard") && strings.contains(lacking.missing[0],"must ingest it"),"%v",lacking.missing[:]) + _,listed:=listed_of(t,here,game)["billboard.tex"] + testing.expect(t,listed,"what is not built is still listed, and nothing is removed") + + // Fast only, where Final is required: an image no Final of which was ever built. + poster:=fmt.tprintf("%s/poster.png",top) + _=os.write_entire_file(poster,png_bytes(8,8,{30,60,220})) + fast:=options + fast.fast_textures=true + quick,quick_ok:=ingested(t,poster,fast) + if !quick_ok || !report_clean(t,project_add(game,quick.manifest,"",nil,fast,context.temp_allocator)) {return} + testing.expect(t,!quick.final,"a refresh that takes Fast builds Fast") + testing.expect(t,os.remove(poster)==nil) + final:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,len(final.missing)==2 && strings.contains(strings.join(final.missing[:],"\n",context.temp_allocator),"package poster: only a build with Fast textures is here, and Final is required"),"%v",final.missing[:]) + testing.expectf(t,len(project_refresh(game,fast,context.temp_allocator).missing)==1,"with Fast taken only the billboard, whose source is gone, is missing") +} + +// What a project's packages.mirai or the store says that is not a record of +// this version is reported, never a crash nor a default. +@(test) +hostile_packages_and_manifests_are_reported_without_a_crash :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-hostile") + if top=="" {return} + game:=fmt.tprintf("%s/game",top) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + s:=test_store(t,fmt.tprintf("%s/store",top)) + if s==nil {return} + options:=Library_Options{store=s,root=game} + garbage,_:=store.put_bytes(s,transmute([]u8)string(`{"version": 1, "id": "nonsense"}`)) + other:=asset.Package{id={9,9},name="other",origin={path="/nowhere/other.png",importer="image"}} + other_manifest,_:=asset.package_put(s,&other) + lost:=asset.Package{id={7,7},name="lost",origin={path="/nowhere/lost.png",importer="image"}} + lost_manifest,_:=asset.package_put(s,&lost) + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + append(&list.packages,asset.Project_Package{id={1,1},manifest=garbage},asset.Project_Package{id={2,2},manifest=other_manifest},asset.Project_Package{id={7,7},manifest=lost_manifest,folder="lost"}) + testing.expect_value(t,asset.packages_save(game,&list),asset.Error.None) + report:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,report.packages==3 && len(report.failures)==2 && len(report.missing)==1 && strings.contains(report.missing[0],"package lost"),"%v",report) + + text,_:=os.read_entire_file(fmt.tprintf("%s/%s",game,asset.PACKAGES_FILE),context.temp_allocator) + work:=make([]u8,len(text),context.temp_allocator) + path:=fmt.tprintf("%s/%s",game,asset.PACKAGES_FILE) + refused:=0 + for length:=0;length0) + } + marks:="{}[]\":,0-9ef \x00\xff" + state:=u64(0x9e3779b97f4a7c15) + for _ in 0..<300 { + copy(work,text) + state=state*6364136223846793005+1442695040888963407 + work[int((state>>33)%u64(len(work)))]=marks[int((state>>20)%u64(len(marks)))] + _=os.write_entire_file(path,work) + _=project_refresh(game,options,context.temp_allocator) + } + testing.expectf(t,refused>0,"cut texts are refused: %d",refused) + testing.expect(t,!os.exists(fmt.tprintf("%s/assets/lost",game)),"nothing is written for what is not here") +} + +// Two that ingest one source at once take turns, and the second finds the +// work done. +@(test) +two_ingests_of_one_source_at_once_import_it_once :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-lock") + if top=="" {return} + sign:=fmt.tprintf("%s/sign.png",top) + _=os.write_entire_file(sign,png_bytes(64,64,{20,140,40})) + Run :: struct {root,source:string,assets:int,manifest:store.Hash,failed:bool,ready:^sync.Barrier} + ready:sync.Barrier + sync.barrier_init(&ready,2) + runs:[2]Run + for &run in runs {run={root=fmt.tprintf("%s/store",top),source=sign,ready=&ready}} + work :: proc(run:^Run) { + s,error:=store.open(run.root) + if error!=.None {run.failed=true;return} + defer store.close(&s) + sync.barrier_wait(run.ready) + report,ingest_error:=ingest(run.source,{store=&s}) + defer importer.report_destroy(&report) + run.failed=ingest_error.code!=.None + run.assets,run.manifest=len(report.assets),report.manifest + } + threads:[2]^thread.Thread + for &run,at in runs {threads[at]=thread.create_and_start_with_poly_data(&run,work)} + for th in threads { + thread.join(th) + thread.destroy(th) + } + testing.expect(t,!runs[0].failed && !runs[1].failed && runs[0].manifest==runs[1].manifest && runs[0].manifest!={}) + testing.expectf(t,(runs[0].assets==0)!=(runs[1].assets==0),"one imports and the other finds it done: %d and %d assets",runs[0].assets,runs[1].assets) +} diff --git a/cook/import/sources/reimport_test.odin b/cook/import/sources/reimport_test.odin index e3c0ceaac..2c664c852 100644 --- a/cook/import/sources/reimport_test.odin +++ b/cook/import/sources/reimport_test.odin @@ -20,9 +20,9 @@ import "mirai:cook/import/gltf" import "mirai:cook/textures" import "mirai:project" import "mirai:std/base" +import "mirai:store" // A box with two materials, the first with the image wood.png next to it. -@(private="file") box_source :: proc(folder,file:string,first,second:string,roughness:="0.5") -> string { vertices:=[15]f32{0,0,0,1,0,0,0,1,0,0,0,1,0,0,1} encoded,_:=base64.encode(slice.to_bytes(vertices[:]),allocator=context.temp_allocator) @@ -39,7 +39,7 @@ box_source :: proc(folder,file:string,first,second:string,roughness:="0.5") -> s @(private="file") box_import :: proc(t:^testing.T,source,root:string) -> (report:importer.Import_Report,ok:bool) { error:importer.Error - report,error=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) + report,error=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) ok=testing.expectf(t,error.code==.None,"%s: %s",source,error.message) return } @@ -64,10 +64,9 @@ slot_materials :: proc(t:^testing.T,report:^importer.Import_Report,types:^asset. return {view.slots[0].material,view.slots[1].material} } -// What an import of a project builds that nobody edits is of its imported -// folder, named by its id; what the editor edits is of its assets, in the -// folder of the import beside the source, found by its id wherever it was -// moved. +// What an import builds that nobody edits is a blob of the store; what the +// editor edits is also a file of the project, in the folder of the import, +// found by its id wherever it was moved; the package lists them all. @(test) the_parts_of_an_import_name_its_prefab_and_keep_their_ids_where_they_were_moved :: proc(t:^testing.T) { root,assets:=test_project(t,"reimport-moved") @@ -88,18 +87,19 @@ the_parts_of_an_import_name_its_prefab_and_keep_their_ids_where_they_were_moved testing.expect(t,strings.has_prefix(wood.detail,"of its own: an island"),wood.detail) for written in first.assets { cached:=asset.import_cached(written.kind) - testing.expectf(t,strings.contains(written.path,cached?"/.mirai/imported/":"/assets/models/box/"),"%s: of the %s",written.path,cached?"imported folder":"assets") + testing.expectf(t,strings.contains(written.path,cached?"/store/blobs/":"/assets/models/box/"),"%s: of the %s",written.path,cached?"store":"assets") } - sidecar,found,_:=importer.sidecar_load(source,context.temp_allocator) - testing.expect(t,found && sidecar.folder=="box" && len(sidecar.outputs)==len(first.assets),"the sidecar next to the source lists what it gives and where") - for output in sidecar.outputs { - written,listed:=written_of(&first,output.kind,output.kind==asset.KIND_SCENE?"box":output.name) - testing.expectf(t,listed && written.id==output.id && output.part==(output.kind==asset.KIND_MESH),"%s is listed with its id: %v",output.name,output) + s:=test_store(t,fmt.tprintf("%s/store",root)) + p,read_error:=asset.package_get(s,first.manifest,context.temp_allocator) + testing.expect(t,read_error==.None && len(p.assets)==len(first.assets),"the package lists what the source gives") + for a in p.assets { + written,listed:=written_of(&first,a.kind,a.kind==asset.KIND_SCENE?"box":a.name) + testing.expectf(t,listed && written.id==a.id && a.part==(a.kind==asset.KIND_MESH),"%s is listed with its id: %v",a.name,a) } a,loaded:=read_asset(t,prefab,&types) if loaded { info,error:=asset.import_info(&a,&types) - testing.expect(t,error==.None && info!=nil && string_of(info.source[:])=="assets/models/box.gltf" && .Cooked in a.blob.header.flags,"the prefab records its import") + testing.expect(t,error==.None && info!=nil && string_of(info.source[:])=="box.gltf" && .Cooked in a.blob.header.flags,"the prefab records its import") asset.asset_close(&a) } @@ -120,25 +120,26 @@ the_parts_of_an_import_name_its_prefab_and_keep_their_ids_where_they_were_moved testing.expect_value(t,len(again.warnings),0) } -// The ids of an import are those its sidecar lists: a source moved, renamed -// or changed keeps them as long as its sidecar goes with it, and a copy of -// the source alone is another import. +// The ids of an import are those of its package: a source moved, renamed or +// changed keeps them with the manifest its project pins, and a copy of the +// source is another package. @(test) -a_source_keeps_the_ids_of_its_import_with_its_sidecar_and_a_copy_without_it_is_another_import :: proc(t:^testing.T) { +a_source_keeps_the_ids_of_its_package_and_a_copy_is_another_package :: proc(t:^testing.T) { root,assets:=test_project(t,"reimport-renamed") if root=="" {return} os.make_directory_all(fmt.tprintf("%s/props",assets)) source:=box_source(assets,"box.gltf","Wood","Paint") first,ok:=box_import(t,source,root);if !ok {return};defer importer.report_destroy(&first) - // Moved, renamed and changed, with its sidecar. + // Moved, renamed and changed, imported with the manifest the project pins. crate:=box_source(fmt.tprintf("%s/props",assets),"crate.gltf","Wood","Paint","0.8") - testing.expect(t,os.rename(fmt.tprintf("%s.import",source),fmt.tprintf("%s.import",crate))==nil && os.remove(source)==nil) - moved,moved_ok:=box_import(t,crate,root);if !moved_ok {return};defer importer.report_destroy(&moved) - testing.expect_value(t,moved.prefab,first.prefab) + testing.expect(t,os.remove(source)==nil) + moved,moved_error:=import_test(t,crate,{world=prefab.import_world(t),root=root,textures=.Uncompressed,manifest=first.manifest}) + if !testing.expectf(t,moved_error.code==.None,"%s",moved_error.message) {return};defer importer.report_destroy(&moved) + testing.expect(t,moved.prefab==first.prefab && moved.package_id==first.package_id) for written in first.assets { same,found:=written_of(&moved,written.kind,written.kind==asset.KIND_SCENE?"crate":written.name) - // A file of the imported folder is named by what it is built from, which changed. + // A document is followed in its file; a blob is another one, made from what changed. testing.expectf(t,found && same.id==written.id && (same.path==written.path || asset.import_cached(written.kind)),"%s keeps its id and its file",written.name) } testing.expect_value(t,len(moved.warnings),0) @@ -146,15 +147,15 @@ a_source_keeps_the_ids_of_its_import_with_its_sidecar_and_a_copy_without_it_is_a scene,_:=written_of(&moved,asset.KIND_SCENE,"crate") if a,loaded:=read_asset(t,scene,&types);loaded { info,_:=asset.import_info(&a,&types) - testing.expect(t,info!=nil && string_of(info.source[:])=="assets/props/crate.gltf","the import names the source where it is now") + testing.expect(t,info!=nil && string_of(info.source[:])=="crate.gltf","the import names the source by its name in its package") asset.asset_close(&a) } - // The same bytes without the sidecar: an import of its own, which takes nothing of the other. + // The same bytes elsewhere: a package of its own, which takes nothing of the other. copy:=box_source(assets,"copy.gltf","Wood","Paint","0.8") - second,second_error:=import_source(copy,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) + second,second_error:=import_test(t,copy,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) if !testing.expectf(t,second_error.code==.None,"%s",second_error.message) {return};defer importer.report_destroy(&second) - testing.expect(t,second.prefab!={} && second.prefab!=first.prefab,"the copy writes a prefab of its own") + testing.expect(t,second.prefab!={} && second.prefab!=first.prefab && second.package_id!=first.package_id,"the copy writes a prefab of its own") for written in second.assets { for other in first.assets {testing.expectf(t,other.id!=written.id,"%s has an id of its own",written.path)} } @@ -182,21 +183,26 @@ a_material_renamed_in_the_source_is_a_new_one_and_a_part_taken_out_is_left_alone for warning in renamed.warnings {left||=strings.contains(warning,"Wood.mat is no longer in the source")} testing.expectf(t,left,"the import says Wood left: %v",renamed.warnings) - // Whether a mesh is a part is decided once and kept in the sidecar: taken out there, it stays out. + // Whether a mesh is a part is decided once and kept in the package: taken out there, it stays out. mesh,_:=written_of(&renamed,asset.KIND_MESH,"Box") testing.expect(t,header_of(mesh.path).owner==first.prefab) - file:=fmt.tprintf("%s.import",source) - listed,_:=os.read_entire_file(file,context.temp_allocator) - taken_out,replaced:=strings.replace(string(listed),", \"part\": true","",1,context.temp_allocator) - testing.expect(t,replaced && os.write_entire_file(file,taken_out)==nil) - after,after_ok:=box_import(t,source,root);if !after_ok {return};defer importer.report_destroy(&after) + s:=test_store(t,fmt.tprintf("%s/store",root)) + p,_:=asset.package_get(s,renamed.manifest,context.temp_allocator) + taken_out:=false + for &a in p.assets { + if a.kind==asset.KIND_MESH && a.part {a.part,taken_out=false,true} + } + manifest,put_error:=asset.package_put(s,&p) + testing.expect(t,taken_out && put_error==.None) + after,after_error:=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,manifest=manifest}) + if !testing.expectf(t,after_error.code==.None,"%s",after_error.message) {return};defer importer.report_destroy(&after) box,has_box:=written_of(&after,asset.KIND_MESH,"Box") testing.expect(t,has_box && box.id==mesh.id) header:=header_of(box.path) testing.expect(t,header.owner=={} && .Cooked in header.flags,"the part taken out keeps its id and its import, and stays out of its owner") // Forced, every material and texture is a part. - forced,forced_ok:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,parts=.All,ignore_stored=true}) + forced,forced_ok:=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,parts=.All,ignore_stored=true}) if !testing.expectf(t,forced_ok.code==.None,"%s",forced_ok.message) {return};defer importer.report_destroy(&forced) for written in forced.assets { if written.kind==asset.KIND_MATERIAL || written.kind==asset.KIND_TEXTURE {testing.expectf(t,header_of(written.path).owner==first.prefab,"%s is forced into the prefab",written.name)} @@ -271,7 +277,7 @@ a_material_is_a_part_only_when_its_texture_is_an_atlas_of_one_model :: proc(t:^t if root=="" {return} atlas_images(assets) source:=atlas_source(assets,"props.gltf",{"atlas","waves","across","shared"}) - report,error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) + report,error:=import_test(t,source,{world=prefab.import_world(t),root=root,textures=.Uncompressed}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return};defer importer.report_destroy(&report) reason :: proc(material:string) -> string { switch material { @@ -294,7 +300,7 @@ a_material_is_a_part_only_when_its_texture_is_an_atlas_of_one_model :: proc(t:^t // Two models that embed the same image each get a texture and a material of // their own, encoded once in the content cache: neither import changes a -// file or a sidecar of the other, and each keeps its atlas as a part. +// file or the package of the other, and each keeps its atlas as a part. @(test) two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other :: proc(t:^testing.T) { root,assets:=test_project(t,"parts-two-models") @@ -304,16 +310,14 @@ two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other cache:=test_store(t,fmt.tprintf("%s/store",root)) if cache==nil {return} first_source:=atlas_source(fmt.tprintf("%s/first",assets),"crate.gltf",{"atlas","waves","across","shared"}) - first,error:=import_source(first_source,{world=prefab.import_world(t),root=root,store=cache}) + first,error:=import_test(t,first_source,{world=prefab.import_world(t),root=root,store=cache}) + first_package,_:=asset.package_get(cache,first.manifest,context.temp_allocator) if !testing.expectf(t,error.code==.None,"%s",error.message) {return};defer importer.report_destroy(&first) atlas,_:=written_of(&first,asset.KIND_MATERIAL,"Atlas") texture,_:=written_of(&first,asset.KIND_TEXTURE,"atlas") testing.expect(t,header_of(atlas.path).owner==first.prefab && header_of(texture.path).owner==first.prefab) Kept :: struct {path:string,bytes:[]u8} kept:=make([dynamic]Kept,context.temp_allocator) - sidecar:=fmt.tprintf("%s.import",first_source) - sidecar_bytes,_:=os.read_entire_file(sidecar,context.temp_allocator) - append(&kept,Kept{sidecar,sidecar_bytes}) for written in first.assets { bytes,_:=os.read_entire_file(written.path,context.temp_allocator) append(&kept,Kept{written.path,bytes}) @@ -321,7 +325,7 @@ two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other cached:=cached_refs(cache,textures.CACHE) second_source:=atlas_source(fmt.tprintf("%s/second",assets),"barrel.gltf",{"atlas","waves","across","shared"}) - second,second_error:=import_source(second_source,{world=prefab.import_world(t),root=root,store=cache}) + second,second_error:=import_test(t,second_source,{world=prefab.import_world(t),root=root,store=cache}) if !testing.expectf(t,second_error.code==.None,"%s",second_error.message) {return};defer importer.report_destroy(&second) second_atlas,wrote_material:=written_of(&second,asset.KIND_MATERIAL,"Atlas") second_texture,wrote_texture:=written_of(&second,asset.KIND_TEXTURE,"atlas") @@ -331,149 +335,12 @@ two_models_with_the_same_image_get_a_texture_each_and_neither_touches_the_other bytes,_:=os.read_entire_file(file.path,context.temp_allocator) testing.expectf(t,slice.equal(bytes,file.bytes),"%s is as the first import left it",file.path) } + again_package,_:=asset.package_get(cache,first.manifest,context.temp_allocator) + testing.expect(t,len(again_package.assets)==len(first_package.assets),"the package of the first is as it was") again:=cached_refs(cache,textures.CACHE) testing.expectf(t,len(again)==len(cached),"the images are encoded once: %d entries of the cache, then %d",len(cached),len(again)) } -// A checkout has the sources and their sidecars, and not what they build that -// nobody edits: sync builds that again, the same assets, whatever the order -// its sources come in, and builds nothing when nothing changed. -@(test) -a_checkout_without_its_imported_folder_gets_the_same_assets_from_its_sources :: proc(t:^testing.T) { - root,assets:=test_project(t,"sync") - if root=="" {return} - atlas_images(fmt.tprintf("%s/set/crate",assets)) - atlas_images(fmt.tprintf("%s/set/barrel",assets)) - crate:=atlas_source(fmt.tprintf("%s/set/crate",assets),"crate.gltf",{"atlas","waves","across","shared"}) - barrel:=atlas_source(fmt.tprintf("%s/set/barrel",assets),"barrel.gltf",{"atlas","waves","across","shared"}) - _=os.write_entire_file(fmt.tprintf("%s/sign.png",assets),png_bytes(8,8,{200,40,40})) - sources:=[3]string{crate,barrel,fmt.tprintf("%s/sign.png",assets)} - Built :: struct {path:string,id:base.Asset_Id,bytes:[]u8} - built:=make([dynamic]Built,context.temp_allocator) - caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",root)),library=test_library} - context.user_ptr=prefab.import_world(t) - for source in sources { - report,error:=import_source(source,{world=prefab.import_world(t),root=root,textures=.Uncompressed,store=caches.store}) - if !testing.expectf(t,error.code==.None,"%s: %s",source,error.message) {return} - for written in report.assets {append(&built,Built{path=strings.clone(written.path,context.temp_allocator),id=written.id})} - importer.report_destroy(&report) - } - for &file in built {file.bytes,_=os.read_entire_file(file.path,context.temp_allocator)} - sidecars:[3]string - for source,at in sources { - text,_:=os.read_entire_file(fmt.tprintf("%s.import",source),context.temp_allocator) - sidecars[at]=string(text) - } - imported:=fmt.tprintf("%s/.mirai/imported",root) - cached:=0 - for file in built {cached+=int(strings.has_prefix(file.path,imported))} - testing.expect(t,cached>0 && cached (listed:[dynamic]Listed,missing:int) { - r:asset.Registry - asset.registry_init(&r,fmt.tprintf("%s/assets",root)) - defer asset.registry_destroy(&r) - testing.expect_value(t,asset.registry_scan(&r),asset.Error.None) - listed=make([dynamic]Listed,context.temp_allocator) - for file in r.files { - if file.listed {append(&listed,Listed{file.id,strings.clone(file.path,context.temp_allocator)})} - } - lost:=make([dynamic]asset.Registry_Missing,context.temp_allocator) - asset.registry_missing(&r,&lost) - return listed,len(lost) - } - path_of :: proc(listed:[]Listed,path:string) -> (id:base.Asset_Id,found:bool) { - for entry in listed { - if entry.path==path {return entry.id,true} - } - return - } - before,missing:=listing(t,root) - body,has_body:=path_of(before[:],"set/crate/crate/Body.mesh") - keg,has_keg:=path_of(before[:],"set/barrel/barrel/Body.mesh") - atlas,has_atlas:=path_of(before[:],"set/crate/crate/atlas.tex") - _,has_sign:=path_of(before[:],"sign.tex") - testing.expectf(t,has_body && has_keg && has_atlas && has_sign && missing==0 && len(before)==cached,"what an import gives with its documents, the texture of an image beside it: %v",before[:]) - - // A source that changed is imported again, and it alone; a file it reads counts as the source. - _=os.write_entire_file(fmt.tprintf("%s/sign.png",assets),png_bytes(8,8,{40,40,200})) - changed:=sync(root,caches,context.temp_allocator) - testing.expectf(t,changed.imported==1 && len(changed.failures)==0,"%v",changed) - _=os.write_entire_file(fmt.tprintf("%s/set/crate/across.png",assets),png_pixels(64,64,gradient_green)) - read:=sync(root,caches,context.temp_allocator) - testing.expectf(t,read.imported==1 && len(read.failures)==0,"an image a model reads changed: %v",read) - - // A folder with two sources moves, one of them is renamed and another source is deleted: at - // once, without an import, every asset is listed where its source is, with the id it had, - // what names them still finds them and nothing of the deleted source is listed. - testing.expect(t,os.rename(fmt.tprintf("%s/set",assets),fmt.tprintf("%s/props",assets))==nil) - for end in ([2]string{"",".import"}) {testing.expect(t,os.rename(fmt.tprintf("%s/props/barrel/barrel.gltf%s",assets,end),fmt.tprintf("%s/props/barrel/keg.gltf%s",assets,end))==nil)} - testing.expect(t,os.remove(fmt.tprintf("%s/sign.png",assets))==nil) - after,after_missing:=listing(t,root) - moved_body,_:=path_of(after[:],"props/crate/crate/Body.mesh") - moved_keg,_:=path_of(after[:],"props/barrel/barrel/Body.mesh") - moved_atlas,_:=path_of(after[:],"props/crate/crate/atlas.tex") - testing.expectf(t,moved_body==body && moved_keg==keg && moved_atlas==atlas && body!={} && keg!={} && after_missing==0 && len(after)==len(before)-1,"%v",after[:]) - for entry in after {testing.expectf(t,strings.has_prefix(entry.path,"props/"),"%s is listed where its source is",entry.path)} - // Nothing changed in what they are built from: the next build imports nothing, and removes what the deleted source gave. - again_synced:=sync(root,caches,context.temp_allocator) - testing.expectf(t,again_synced.sources==2 && again_synced.imported==0 && again_synced.removed==1 && len(again_synced.failures)==0,"%v",again_synced) -} - // A format of models a plugin adds, which no format of the engine reads: glTF after a magic of its own. @(private="file") PROBE_MAGIC :: "PROBE:" @(private="file") @@ -487,45 +354,49 @@ probe_format :: proc() -> importer.Format { return probe } -// Every import goes to the cache, whatever reads it: a format a plugin adds -// too, which sync reads again from the plugins of the project, and a source -// outside the assets is refused, saying where it goes. +// A format a plugin adds reads a source as the formats of the engine do: a +// refresh that must ingest it again reads it with the plugins of the project, +// and says so without them. @(test) -the_sources_of_a_plugin_format_build_in_the_cache_and_a_source_outside_the_assets_is_refused :: proc(t:^testing.T) { - root:=base.test_folder(t,"sync-plugin") +a_refresh_reads_a_source_again_with_the_formats_of_the_plugins :: proc(t:^testing.T) { + root:=base.test_folder(t,"refresh-plugin") if root=="" {return} - outside:=fmt.tprintf("%s/downloads",root) - project:=fmt.tprintf("%s/game",root) - os.make_directory_all(outside) - os.make_directory_all(fmt.tprintf("%s/assets",project)) - box:=box_source(outside,"box.gltf","Wood","Paint") - _,refused:=import_source(box,{world=prefab.import_world(t),root=project,textures=.Uncompressed}) - testing.expect(t,refused.code==.Not_Found && strings.contains(refused.message,"assets") && !os.exists(fmt.tprintf("%s.import",box)),refused.message) - - formats:=make([dynamic]importer.Format,context.temp_allocator) - append(&formats,..FORMATS[:]) - append(&formats,probe_format()) - crate:=box_source(fmt.tprintf("%s/assets",project),"crate.probe","Wood","Paint") + game:=fmt.tprintf("%s/game",root) + os.make_directory_all(fmt.tprintf("%s/assets",game)) + crate:=box_source(fmt.tprintf("%s/assets",game),"crate.probe","Wood","Paint") text,_:=os.read_entire_file(crate,context.temp_allocator) _=os.write_entire_file(crate,transmute([]u8)strings.concatenate({PROBE_MAGIC,string(text)},context.temp_allocator)) - first,error:=import_source(crate,{world=prefab.import_world(t),root=project,textures=.Uncompressed},formats[:]) - if !testing.expectf(t,error.code==.None,"%s",error.message) {return};defer importer.report_destroy(&first) - mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") - testing.expect(t,has_mesh && strings.contains(mesh.path,"/.mirai/imported/"),"what a plugin format gives that nobody edits is of the cache") - // A clone: sync builds it again with the formats of the plugins, and says so without them. - caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",root))} - testing.expect(t,base.remove_all(fmt.tprintf("%s/.mirai",project))==nil) - lacking:=sync(project,caches,context.temp_allocator) - testing.expectf(t,len(lacking.failures)==1 && strings.contains(lacking.failures[0],"no format"),"%v",lacking) + s:=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return} context.user_ptr=prefab.import_world(t) - caches.library=proc(root:string,allocator:mem.Allocator) -> Library_Imports { + probing :: proc(root:string,allocator:mem.Allocator) -> Library_Imports { listed:=make([]importer.Format,1,allocator) listed[0]=probe_format() return {formats=listed,world=(^core.World_Info)(context.user_ptr)} } - synced:=sync(project,caches,context.temp_allocator) - testing.expectf(t,synced.sources==1 && synced.imported==1 && len(synced.failures)==0,"%v",synced) - testing.expect(t,os.exists(mesh.path),"the mesh is built again where it was") + options:=Library_Options{store=s,root=game,plugins=probing} + first,error:=ingest(crate,options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=project_add(game,first.manifest,"",nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") + testing.expect(t,has_mesh && strings.contains(mesh.path,"/store/blobs/"),"what a plugin format gives that nobody edits is a blob") + + // An engine of other versions: without the plugins nothing reads it, with them it is ingested again. + forget :: proc(s:^store.Store) { + for namespace in ([2]string{asset.REF_IMPORTS,asset.REF_BUILDS}) { + refs,_:=store.ref_list(s,namespace,context.temp_allocator) + for ref in refs {store.ref_remove(s,namespace,ref.name)} + } + } + forget(s) + lacking:=options + lacking.plugins=nil + without:=project_refresh(game,lacking,context.temp_allocator) + testing.expectf(t,len(without.failures)==1 && strings.contains(without.failures[0],"not a source Mirai imports"),"%v",without) + with:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,with.ingested==1 && len(with.failures)==0 && len(with.missing)==0,"%v",with) + testing.expect(t,os.exists(mesh.path),"the mesh is the blob it was") } // A source a plugin imports whole: "LEVEL ", a level of the glTF at @@ -542,7 +413,7 @@ level_run :: proc(state:rawptr,source:string,options:importer.Import_Options,all bytes,_:=os.read_entire_file(source,context.temp_allocator) model_path:=fmt.tprintf("%s/%s",filepath.dir(source),string(bytes[len("LEVEL "):])) o:importer.Outputs - if error=importer.outputs_begin(&o,source,"",u64(len(bytes)),base.hash64(bytes),importer.import_version(0),options,allocator);error.code!=.None {return} + if error=importer.outputs_begin(&o,source,"",u64(len(bytes)),base.hash64(bytes),importer.import_version(0),options,allocator=allocator);error.code!=.None {return} model:importer.Imported_Model if error=importer.output_input(&o,model_path);error.code==.None {model,error=importer.import_model(model_path,FORMATS[:],options)} defer importer.model_destroy(&model) @@ -553,75 +424,65 @@ level_run :: proc(state:rawptr,source:string,options:importer.Import_Options,all asset.document_init(&kit,KIT,id,context.temp_allocator) kit.kind_version=version if asset.document_save(&kit,path)!=.None {error=importer.error_make(.Write,"%s could not be written",path)} - importer.output_written(&o,id,KIT,"hall",path,false) + if error.code==.None {error=importer.output_written(&o,id,"hall")} + } + if error.code!=.None { + _,_=importer.outputs_end(&o,true) + return {},"",error } - return importer.outputs_end(&o,error.code!=.None),"",error + report,error=importer.outputs_end(&o,false,allocator) + return report,"",error } -// Sync runs the importer of a plugin that claims a whole source when what it -// is built from changes: its bytes or those of a file it reads, the code of -// the plugin, or the version of a kind of document it wrote, of the engine or -// of a plugin; it leaves a current one alone. +// A refresh ingests again a whole source a plugin claims when what it is +// built from changes: its bytes or those of a file it reads, or the code of +// the plugin; it leaves a current one alone, and the ids stay. @(test) -sync_imports_again_a_whole_source_when_what_it_is_built_from_changes :: proc(t:^testing.T) { - root,assets:=test_project(t,"sync-whole") +a_refresh_ingests_again_a_whole_source_when_what_it_is_built_from_changes :: proc(t:^testing.T) { + root,assets:=test_project(t,"refresh-whole") if root=="" {return} - write_text(root,project.PROJECT_FILE,`{"name": "Hall", "version": "0.1.0"}`) - os.make_directory_all(fmt.tprintf("%s/plugins/kit",root)) - kit_manifest :: proc(root:string,version:int) {write_text(fmt.tprintf("%s/plugins/kit",root),project.PLUGIN_FILE,strings.concatenate({`{"name": "kit", "version": "1.0.0", "asset_kinds": [{"extension": ".kit", "kind": "KITS", "version": `,fmt.tprint(version),`}]}`},context.temp_allocator))} - kit_manifest(root,2) props:=fmt.tprintf("%s/props",assets) os.make_directory_all(props) box_source(props,"crate.gltf","Wood","Paint") level:=write_text(assets,"hall.level","LEVEL props/crate.gltf") state:=Level_Importer{kit=2} - wholes:=[1]Whole_Source{{claims=level_claims,run=level_run,state=&state}} - caches:=Sync_Options{store=test_store(t,fmt.tprintf("%s/store",root))} - first,error:=import_source(level,{root=root,textures=.Uncompressed,store=caches.store},wholes=wholes[:]) - if !testing.expectf(t,error.code==.None,"%s",error.message) {return} - defer importer.report_destroy(&first) - mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") - testing.expect(t,has_mesh && strings.contains(mesh.path,"/.mirai/imported/"),"the mesh of the level is of the cache") - sidecar,_,_:=importer.sidecar_load(level,context.temp_allocator) - testing.expectf(t,slice.contains(sidecar.inputs[:],"props/crate.gltf") && slice.contains(sidecar.inputs[:],"props/wood.png"),"the files it reads are its inputs: %v",sidecar.inputs) - - lacking:=sync(root,caches,context.temp_allocator) - testing.expectf(t,len(lacking.failures)==1 && strings.contains(lacking.failures[0],"no format or importer"),"%v",lacking) + s:=test_store(t,fmt.tprintf("%s/store",root)) + if s==nil {return} context.user_ptr=&state - caches.library=proc(root:string,allocator:mem.Allocator) -> Library_Imports { + leveling :: proc(root:string,allocator:mem.Allocator) -> Library_Imports { wholes:=make([]Whole_Source,1,allocator) wholes[0]={claims=level_claims,run=level_run,state=context.user_ptr,code=(^Level_Importer)(context.user_ptr).code} return {wholes=wholes} } - current:=sync(root,caches,context.temp_allocator) - testing.expectf(t,current.sources==1 && current.imported==0 && len(current.failures)==0 && state.runs==1,"a current source is left alone: %v",current) - Change :: enum {Input,Code,Engine_Kind,Plugin_Kind,Clone} - material,has_material:=written_of(&first,asset.KIND_MATERIAL,"Wood") - kit,has_kit:=written_of(&first,KIT,"hall") - if !testing.expect(t,has_material && has_kit,"the level wrote documents") {return} + options:=Library_Options{store=s,root=root,plugins=leveling,fast_textures=true} + first,error:=ingest(level,options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + mesh,has_mesh:=written_of(&first,asset.KIND_MESH,"Box") + testing.expect(t,has_mesh && strings.contains(mesh.path,"/store/blobs/"),"the mesh of the level is a blob") + p,_:=asset.package_get(s,first.manifest,context.temp_allocator) + testing.expectf(t,slice.contains(p.inputs[:],"props/crate.gltf") && slice.contains(p.inputs[:],"props/wood.png"),"the files it reads are its inputs: %v",p.inputs) + added:=project_add(root,first.manifest,"",nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + _,has_kit:=written_of(&first,KIT,"hall") + testing.expect(t,has_kit,"the level wrote a document") + + runs:=state.runs + current:=project_refresh(root,options,context.temp_allocator) + testing.expectf(t,current.ingested==0 && len(current.failures)==0 && len(current.missing)==0 && state.runs==runs,"a current source is left alone: %v",current) + Change :: enum {Input,Code} for change in Change { switch change { case .Input: _=os.write_entire_file(fmt.tprintf("%s/wood.png",props),png_bytes(4,4,{20,90,200})) case .Code: state.code=7 - case .Engine_Kind: - data,_:=os.read_entire_file(material.path,context.temp_allocator) - old:=asset.MATERIAL_VERSION-1 - data[10],data[11]=u8(old),u8(old>>8) - testing.expect(t,os.write_entire_file(material.path,data)==nil) - case .Plugin_Kind: - kit_manifest(root,3) - state.kit=3 - case .Clone: testing.expect(t,base.remove_all(fmt.tprintf("%s/.mirai",root))==nil) } - runs:=state.runs - changed:=sync(root,caches,context.temp_allocator) - testing.expectf(t,changed.imported==1 && len(changed.failures)==0 && state.runs==runs+1,"%v: %v",change,changed) - testing.expectf(t,sync(root,caches,context.temp_allocator).imported==0,"%v: then nothing",change) + runs=state.runs + changed:=project_refresh(root,options,context.temp_allocator) + testing.expectf(t,changed.ingested==1 && len(changed.failures)==0 && state.runs==runs+1,"%v: %v",change,changed) + testing.expectf(t,project_refresh(root,options,context.temp_allocator).ingested==0,"%v: then nothing",change) } - testing.expect(t,header_of(material.path).kind_version==asset.MATERIAL_VERSION && header_of(material.path).id==material.id,"the material is of this version again, with its id") - testing.expect(t,header_of(kit.path).kind_version==3 && header_of(kit.path).id==kit.id,"the document of the plugin is of its version again, with its id") - again,_,_:=importer.sidecar_load(level,context.temp_allocator) - testing.expect(t,len(again.outputs)==len(sidecar.outputs) && again.outputs[0].id==sidecar.outputs[0].id,"the level keeps its assets") + list,_,_:=asset.packages_load(root,context.temp_allocator) + again,_:=asset.package_get(s,list.packages[0].manifest,context.temp_allocator) + testing.expect(t,list.packages[0].manifest!=first.manifest && len(again.assets)==len(p.assets) && again.assets[0].id==p.assets[0].id,"the level keeps its assets, pinned anew") } // The code of a plugin is its Odin sources and those of the plugins it holds, @@ -646,34 +507,35 @@ the_code_of_a_plugin_changes_with_its_sources_alone :: proc(t:^testing.T) { // given once stays in its settings. @(test) an_image_on_its_own_holds_what_its_name_says_until_its_settings_say_otherwise :: proc(t:^testing.T) { - _,assets:=test_project(t,"image-use") + root,assets:=test_project(t,"image-use") if assets=="" {return} + kept:=fmt.tprintf("%s/store",root) run :: proc(args:..string) -> (code:int,err:string) { output,errors:=strings.builder_make(context.temp_allocator),strings.builder_make(context.temp_allocator) code=command(args,strings.to_writer(&output),strings.to_writer(&errors)) return code,strings.to_string(errors) } - stored :: proc(source:string) -> importer.Image_Use { - settings,_,_:=importer.settings_load(source) + stored :: proc(t:^testing.T,kept,source:string) -> importer.Image_Use { + settings,_,_:=importer.settings_load({store=test_store(t,kept)},source) return settings.image } masks:=fmt.tprintf("%s/T_Katana_SpecularMasks.png",assets) diffuse:=fmt.tprintf("%s/T_Katana_Diffuse.png",assets) for path in ([?]string{masks,diffuse}) {_=os.write_entire_file(path,png_bytes(4,4,{10,200,30}))} - code,err:=run(masks,"--uncompressed") + code,err:=run(masks,"--uncompressed","--store",kept) testing.expectf(t,code==0,"%s",err) - testing.expect_value(t,stored(masks),importer.Image_Use.Data) - code,err=run(diffuse,"--uncompressed") + testing.expect_value(t,stored(t,kept,masks),importer.Image_Use.Data) + code,err=run(diffuse,"--uncompressed","--store",kept) testing.expectf(t,code==0,"%s",err) - testing.expect_value(t,stored(diffuse),importer.Image_Use.Color) + testing.expect_value(t,stored(t,kept,diffuse),importer.Image_Use.Color) // Given once, the use stays: the name no longer decides. - code,err=run(masks,"--uncompressed","--srgb") + code,err=run(masks,"--uncompressed","--srgb","--store",kept) testing.expectf(t,code==0,"%s",err) - code,err=run(masks,"--uncompressed") + code,err=run(masks,"--uncompressed","--store",kept) testing.expectf(t,code==0,"%s",err) - testing.expect_value(t,stored(masks),importer.Image_Use.Color) + testing.expect_value(t,stored(t,kept,masks),importer.Image_Use.Color) Named :: struct {name:string,use:importer.Image_Use} for named in ([?]Named{{"data",.Data},{"Color",.Color},{"normal",.Normal},{"NORMAL_GREEN_DOWN",.Normal_Green_Down}}) { @@ -690,7 +552,7 @@ importing_over_a_skeleton_of_an_older_version_says_what_it_did_not_keep :: proc( root,folder:=test_project(t,"old-skeleton") if folder=="" {return} source:=copy_in(root,FIXTURES,"skeletal-analytic.glb") - first,error:=import_source(source,{world=prefab.import_world(t),textures=.Uncompressed}) + first,error:=import_test(t,source,{world=prefab.import_world(t),textures=.Uncompressed}) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} defer importer.report_destroy(&first) skeleton:importer.Written @@ -700,7 +562,7 @@ importing_over_a_skeleton_of_an_older_version_says_what_it_did_not_keep :: proc( old:=asset.SKELETON_VERSION-1 data[10],data[11]=u8(old),u8(old>>8) testing.expect(t,os.write_entire_file(skeleton.path,data)==nil) - again,failed:=import_source(source,{world=prefab.import_world(t),textures=.Uncompressed}) + again,failed:=import_test(t,source,{world=prefab.import_world(t),textures=.Uncompressed}) if !testing.expectf(t,failed.code==.None,"%s",failed.message) {return} defer importer.report_destroy(&again) told:=false diff --git a/cook/import/sources/settings_test.odin b/cook/import/sources/settings_test.odin index 04e0cb7fb..a0ac8da03 100644 --- a/cook/import/sources/settings_test.odin +++ b/cook/import/sources/settings_test.odin @@ -8,6 +8,8 @@ import "mirai:asset" import "mirai:std/base" import "mirai:std/geom" import importer "mirai:cook/import" +import "mirai:cook/prefab" +import "mirai:store" // An OBJ of a sphere, smooth, with texture coordinates. @(private="file") @@ -36,7 +38,7 @@ a_source_without_colliders_gives_a_mesh_of_levels_without_any :: proc(t:^testing if !ok {return} defer importer.model_destroy(&model) testing.expect(t,model.settings.front==.X && !model.stored) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{front=.X}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{front=.X}) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) if error.code!=.None || len(report.assets)==0 {return} @@ -58,68 +60,59 @@ a_source_without_colliders_gives_a_mesh_of_levels_without_any :: proc(t:^testing testing.expect(t,int(view.info.vertex_count)==len(view.vertices) && view.levels[0].triangle_count string {return fmt.tprintf("{{\"version\": 3, \"settings\": %s, \"outputs\": [%s]}}",settings,outputs)} - for broken in ([]string{ - whole("{\"lods\": \"Sometimes\"}",""),whole("{\"scale\": 2}",""),"{",whole("{\"rdo_lambda\": 1000}",""),whole("{\"alpha_coverage\": 1.5}",""),whole("{\"alpha_coverage\": \"nan\"}",""),whole("{\"rig\": \"Fox\", \"rig_crotch\": 0.9}",""), - "{}","{\"lods\": \"None\"}","{\"version\": 2, \"settings\": {}, \"outputs\": []}", - "{\"version\": 3, \"settings\": {}, \"folder\": \"../up\", \"outputs\": []}","{\"version\": 3, \"settings\": {}, \"folder\": \"/root\", \"outputs\": []}", - whole("{}","{\"kind\": \"MESH\", \"name\": \"Ball\", \"index\": 0}"), - whole("{}",fmt.tprintf("{{\"kind\": \"MESH\", \"name\": \"Ball\", \"index\": 0, %s}}, {{\"kind\": \"MAT\", \"name\": \"Skin\", \"index\": 0, %s}}",ID,ID)), - whole("{}",fmt.tprintf("{{\"kind\": \"MAT\", \"name\": \"Skin\", \"index\": 0, %s, \"part\": true}}",ID)), - whole("{}",fmt.tprintf("{{\"kind\": \"MESH\", \"name\": \"Ball\", \"index\": 0, %s, \"path\": \"Ball.mesh\"}}",ID)), - }) { - write_text(folder,"ball.obj.import",broken) - _,_,broken_error:=importer.settings_load(source) - testing.expectf(t,broken_error.code==.Malformed,"%s is refused",broken) - _,import_error:=importer.import_model(source,FORMATS[:]) - testing.expect_value(t,import_error.code,importer.Error_Code.Malformed) + // Settings nobody wrote are refused, not the defaults: a value an import does not know, + // out of its range, a field it does not have, heights without the rig they go with. + for broken in ([]string{"{\"lods\": \"Sometimes\"}","{\"scale\": 2}","{","{\"rdo_lambda\": 1000}","{\"alpha_coverage\": 1.5}","{\"alpha_coverage\": \"nan\"}","{\"rig\": \"Fox\", \"rig_crotch\": 0.9}"}) { + _,valid:=importer.settings_parse(broken) + testing.expectf(t,!valid,"%s is refused",broken) } - write_text(folder,"ball.obj.import",whole("{}","")) - stored,found,load_error=importer.settings_load(source) - testing.expect(t,found && load_error.code==.None && stored=={},"a setting that is not named has its default") + // A package that keeps them refuses its import. + kept:=p + kept.settings="{\"lods\": \"Sometimes\"}" + manifest,put_error:=asset.package_put(options.store,&kept) + testing.expect_value(t,put_error,asset.Error.None) + pinned:=options + pinned.manifest=manifest + _,import_error:=importer.import_model(source,FORMATS[:],pinned) + testing.expect_value(t,import_error.code,importer.Error_Code.Malformed) } @(test) @@ -138,26 +131,29 @@ a_standalone_image_takes_its_use_from_its_settings :: proc(t:^testing.T) { if folder=="" {return} path:=fmt.tprintf("%s/normal.png",folder) _=os.write_entire_file(path,png_bytes(8,8,{128,128,255})) - testing.expect(t,importer.sidecar_save(path,{settings={image=.Normal,textures=.Uncompressed}})) - report,error:=importer.import_texture(path,{root=root,image=.Color}) + first,first_error:=importer.import_texture(path,test_options(t,path,{root=root,image=.Normal,textures=.Uncompressed})) + if !testing.expectf(t,first_error.code==.None,"%s",first_error.message) {return} + importer.report_destroy(&first) + options:=test_options(t,path,{root=root,image=.Color}) + report,error:=importer.import_texture(path,options) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) if error.code!=.None {return} testing.expect(t,report.assets[0].id!={}) testing.expect(t,strings.contains(report.assets[0].detail,"8x8"),report.assets[0].detail) testing.expect_value(t,report.settings,importer.Import_Settings{image=.Normal,textures=.Uncompressed}) - text,_:=os.read_entire_file(fmt.tprintf("%s%s",path,importer.SIDECAR_EXTENSION),context.temp_allocator) - testing.expect(t,strings.contains(string(text),`"image": "Normal"`) && strings.contains(string(text),`"textures": "Uncompressed"`) && !strings.contains(string(text),"lods"),string(text)) + p,_:=asset.package_get(options.store,report.manifest,context.temp_allocator) + testing.expect(t,strings.contains(p.settings,`"image": "Normal"`) && strings.contains(p.settings,`"textures": "Uncompressed"`) && !strings.contains(p.settings,"lods"),p.settings) } @(test) -the_coverage_of_a_standalone_image_is_a_setting_its_sidecar_keeps :: proc(t:^testing.T) { +the_coverage_of_a_standalone_image_is_a_setting_its_package_keeps :: proc(t:^testing.T) { _,folder:=test_project(t,"image-coverage") if folder=="" {return} path:=fmt.tprintf("%s/leaves.png",folder) _=os.write_entire_file(path,png_bytes(16,16,{40,120,30},masked=true)) import_image :: proc(t:^testing.T,path:string,options:importer.Import_Options,loc:=#caller_location) -> (written:importer.Written,coverage:f32,ok:bool) { - report,error:=import_source(path,options) + report,error:=import_test(t,path,options) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None && len(report.assets)==1,"%v %s",error.code,error.message,loc=loc) {return} written=report.assets[0] @@ -172,7 +168,7 @@ the_coverage_of_a_standalone_image_is_a_setting_its_sidecar_keeps :: proc(t:^tes first,coverage,ok:=import_image(t,path,{image=.Color,alpha_coverage=0.5}) if !ok {return} testing.expect_value(t,coverage,f32(0.5)) - stored,found,_:=importer.settings_load(path) + stored,found,_:=importer.settings_load(test_options(t,path),path) testing.expect(t,found && stored==importer.Import_Settings{image=.Color,alpha_coverage=0.5},"the coverage is saved with the use of the image") // Whoever imports it again, without knowing the material that shows it, gets the same texture. again,kept_coverage,kept_ok:=import_image(t,path,{}) @@ -188,10 +184,11 @@ a_vertex_that_is_not_a_number_fails_its_model_with_a_message :: proc(t:^testing. defer importer.model_destroy(&model) model.meshes[0].vertices[5].position.y=transmute(f32)u32(0x7fc00000) out:=fmt.tprintf("%s/out",folder) - report,error:=write_into(&model,out) + report,error:=write_into(t,&model,out) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.Malformed) testing.expectf(t,strings.contains(error.message,"not a number") && strings.contains(error.message,model.meshes[0].name),"%q",error.message) - entries,_:=os.read_all_directory_by_path(fmt.tprintf("%s/.mirai/imported",out),context.temp_allocator) - for entry in entries {testing.expectf(t,!strings.has_suffix(entry.name,".mesh"),"%s stays after the failure",entry.name)} + s:=test_store(t,fmt.tprintf("%s/store",out)) + refs,_:=store.ref_list(s,asset.REF_PACKAGES,context.temp_allocator) + testing.expect(t,len(refs)==0,"a failed import names no package") } diff --git a/cook/import/sources/support_test.odin b/cook/import/sources/support_test.odin index 5e08d81c8..2d2e27d16 100644 --- a/cook/import/sources/support_test.odin +++ b/cook/import/sources/support_test.odin @@ -71,9 +71,9 @@ string_of :: proc(field:[]u8) -> string { } // Writes model as the import of a copy of its source in the assets of the -// test project at root, as import_write takes it: a source in no assets -// folder imports nowhere. -write_into :: proc(model:^importer.Imported_Model,root:string,options:importer.Import_Options={}) -> (report:importer.Import_Report,error:importer.Error) { +// test project at root, as import_write takes it, into the store of the test +// (test_options). +write_into :: proc(t:^testing.T,model:^importer.Imported_Model,root:string,options:importer.Import_Options={}) -> (report:importer.Import_Report,error:importer.Error) { assets:=fmt.tprintf("%s/assets",root) copied:=fmt.tprintf("%s/%s",assets,filepath.base(model.source)) if model.source!=copied { @@ -84,7 +84,7 @@ write_into :: proc(model:^importer.Imported_Model,root:string,options:importer.I } options:=options options.root=root - return importer.import_write(model,options) + return importer.import_write(model,test_options(t,model.source,options)) } // Copies files of a folder (FIXTURES, DEMO), by their paths under it, into @@ -102,8 +102,32 @@ copy_in :: proc(root,from:string,files:..string) -> string { return first } -// A folder of a test as the root of a project, with its assets folder, where -// the sources of its imports are. "" when there is none. +// options of an import of the source at path as a project that adds it +// imports it: in the store of the test, /store, with its documents in +// the folder of the source (root: options.root, else the project whose assets +// hold the source, else its folder), unless options give them. +test_options :: proc(t:^testing.T,path:string,options:importer.Import_Options={},loc:=#caller_location) -> importer.Import_Options { + options:=options + root:=options.root + if root=="" {root,_=source_root(path,context.temp_allocator)} + if root=="" {root=filepath.dir(path)} + os.make_directory_all(root) + // A root that cannot hold a store leaves none: the import says so. + if options.store==nil { + if s,error:=store.open(fmt.tprintf("%s/store",root),context.temp_allocator);error==.None { + options.store=new(store.Store,context.temp_allocator) + options.store^=s + } + } + if options.documents=="" {options.documents=filepath.dir(importer.path_key(path))} + return options +} + +// import_source with test_options. +import_test :: proc(t:^testing.T,path:string,options:importer.Import_Options={},formats:[]importer.Format=nil,wholes:[]Whole_Source=nil,allocator:=context.allocator,loc:=#caller_location) -> (report:importer.Import_Report,error:importer.Error) { + return import_source(path,test_options(t,path,options,loc),formats,wholes,allocator) +} + // A store at folder, in the temp allocator; nil when it cannot be opened. test_store :: proc(t:^testing.T,folder:string,loc:=#caller_location) -> ^store.Store { s,error:=store.open(folder,context.temp_allocator) @@ -113,6 +137,21 @@ test_store :: proc(t:^testing.T,folder:string,loc:=#caller_location) -> ^store.S return kept } +// The registry of the assets of the project at root with the cache of the +// package the import of report wrote listed at their top, as a project that +// pins it lists them. +registry_with :: proc(t:^testing.T,root:string,report:^importer.Import_Report,r:^asset.Registry,loc:=#caller_location) -> bool { + s:=test_store(t,fmt.tprintf("%s/store",root),loc) + if s==nil {return false} + list:asset.Packages + list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) + append(&list.packages,asset.Project_Package{id=report.package_id,manifest=report.manifest}) + resolved,error:=asset.packages_resolve(s,&list,context.temp_allocator) + if !testing.expect_value(t,error,asset.Error.None,loc=loc) {return false} + asset.registry_init(r,fmt.tprintf("%s/assets",root),"",context.allocator,resolved.listed[:]) + return testing.expect_value(t,asset.registry_scan(r),asset.Error.None,loc=loc) +} + // The results a cache of s keeps (cook/cache), in the temp allocator. cached_refs :: proc(s:^store.Store,name:string) -> []store.Ref { buffer:[store.NAME_MAX]u8 @@ -126,20 +165,35 @@ cached_key :: proc(ref:store.Ref) -> u64 { return key } -// A source that copies what the store at user keeps, as a peer would give it. +// A source that copies what the store at user keeps, as a peer would give +// it: the blob of the ref and, for a build, every blob it lists. copy_source :: proc(from:^store.Store) -> cache.Source { fetch :: proc(user:rawptr,into:^store.Store,namespace,name:string) -> bool { from:=(^store.Store)(user) hash,error:=store.ref_get(from,namespace,name) - if error!=.None {return false} - path,path_error:=store.blob_path(from,hash,context.temp_allocator) - if path_error!=.None {return false} - if put,put_error:=store.put_file(into,path);put_error!=.None || put!=hash {return false} + if error!=.None || !blob_copy(from,into,hash) {return false} + if namespace==asset.REF_IMPORTS { + b,build_error:=asset.build_get(from,hash,context.temp_allocator) + if build_error!=.None {return false} + for o in b.outputs { + if !blob_copy(from,into,o.blob) {return false} + } + } return store.ref_set(into,namespace,name,hash)==.None } return {fetch=fetch,user=from} } +// Copies the blob hash of from into into. +blob_copy :: proc(from,into:^store.Store,hash:store.Hash) -> bool { + path,path_error:=store.blob_path(from,hash,context.temp_allocator) + if path_error!=.None {return false} + put,put_error:=store.put_file(into,path) + return put_error==.None && put==hash +} + +// A folder of a test as the root of a project, with its assets folder, where +// the sources of its imports are. "" when there is none. test_project :: proc(t:^testing.T,name:string) -> (root,assets:string) { root=base.test_folder(t,name) if root=="" {return} diff --git a/cook/import/sources/sync.odin b/cook/import/sources/sync.odin deleted file mode 100644 index 50588ddcb..000000000 --- a/cook/import/sources/sync.odin +++ /dev/null @@ -1,247 +0,0 @@ -package sources - -import "core:fmt" -import "core:io" -import "core:mem" -import "core:os" -import "core:path/filepath" -import "core:strings" -import "core:time" -import "mirai:asset" -import "mirai:core" -import "mirai:cook/cache" -import importer "mirai:cook/import" -import "mirai:project" -import "mirai:std/base" -import "mirai:std/jobs" -import "mirai:std/serial" -import "mirai:store" - -// Builds again what the imports of a project or a plugin keep in its imported -// folder, which is not versioned, from the sources in its assets folder and -// their sidecars: what every build of a project runs. An output of the cache -// is current when its file is named by the key of what its import builds from -// now (importer.import_key): the bytes of the source and of the files it -// read, its settings and the importer (for a source a plugin imports whole, -// the code of that plugin), with textures built Final, or Fast under -// Sync_Options.fast_textures, which also takes them Final; a file whose size -// and modification time the cache of the root knows is not read -// (importer.Source_Hashes). An import of documents alone is current by its -// stamp, named the same way. A source with an output that is not current, or -// with a document of a version of its kind this build does not read, is -// imported, with the formats of the engine or, for one none of them reads, -// the importer of a plugin of the project that claims it whole or else the -// formats the plugins add; a file of the folder that no sidecar with its -// source lists is removed. - -// What the library of the editor of the project at root gives its imports: -// the formats of models and the importers of whole sources its plugins add, -// with what they read prepared for its assets, and the types of its game, -// which the prefab of a model is written in; release(state) lets them go. -// problem: why there are none. Only a program that links the editor reads it -// (editor.import_library). -Library :: #type proc(root:string,allocator:mem.Allocator) -> Library_Imports -Library_Imports :: struct { - formats:[]importer.Format, - wholes:[]Whole_Source, - world:^core.World_Info, - release:proc(state:rawptr), - state:rawptr, - problem:string, -} - -Sync_Options :: struct { - store:^store.Store, // whose caches keep what imports build (importer.Import_Options); nil: none - cache_source:cache.Source, - rebuild:bool, // imports every source again - // Textures built Fast, as the editor imports them, unless the texture - // cache keeps them Final: what a build that does not cook needs. Outputs - // built Final are current too. False: Final, what a cooked game ships. - fast_textures:bool, - worker:jobs.Worker_Ref, - library:Library, // nil: the formats of the engine alone, and no model imports - // Where it says, as it goes, each source it imports, why and how far it - // is (importer.Progress); nil: nowhere. - progress:io.Writer, -} - -Sync_Report :: struct { - sources:int, // with outputs - imported:int, // imported now - removed:int, // files of the imported folder that no sidecar lists, removed - failures:[dynamic]string, // a source that did not import, with the reason - warnings:[dynamic]string, - allocator:mem.Allocator, -} - -sync_report_destroy :: proc(report:^Sync_Report) { - for text in report.failures {delete(text,report.allocator)} - for text in report.warnings {delete(text,report.allocator)} - delete(report.failures) - delete(report.warnings) - report^={} -} - -// Brings the imported folder of the project or the plugin at root up to date -// with the sources of its assets folder. The report, in allocator, counts -// what was imported and lists the sources that did not import: with one, the -// folder is as up to date as the others allow, and nothing is removed from it. -sync :: proc(root:string,options:Sync_Options={},allocator:=context.allocator) -> (report:Sync_Report) { - report.allocator=allocator - report.failures.allocator,report.warnings.allocator=allocator,allocator - scratch:mem.Dynamic_Arena - mem.dynamic_arena_init(&scratch,allocator,allocator,block_size=1<<20,out_band_size=1<<20) - defer mem.dynamic_arena_destroy(&scratch) - context.temp_allocator=base.dynamic_arena_allocator(&scratch) - - top:=project.path_absolute(root,context.temp_allocator) - imported:=project.imported_path(top,context.temp_allocator) - assets:=project.folder_path(top,.Assets,context.temp_allocator) - registry:asset.Registry - asset.registry_init(®istry,assets,"",context.temp_allocator) - asset.registry_scan(®istry) - import_options:=importer.Import_Options{ - root=top,registry=®istry,rebuild=options.rebuild,worker=options.worker,fast_textures=options.fast_textures, - store=options.store,cache_source=options.cache_source, - } - // The library, read once a source needs a format of its plugins or a model imports. - library:Library_Read - defer if library.release!=nil {library.release(library.state)} - versions:=kind_versions(top,context.temp_allocator) - complete:=true - // The sources to import, found first so that each line says how many there are. - Pending :: struct {source:string,formats:[]importer.Format,wholes:[]Whole_Source,kind:Source_Kind,version:u32,code:u64,why:string} - pending:=make([dynamic]Pending,context.temp_allocator) - todo:=make([dynamic]string,context.temp_allocator) - append(&todo,assets) - for next:=0;next bool { - for output in sidecar.outputs { - if asset.import_cached(output.kind) {continue} - file,found:=asset.registry_find(registry,output.id) - if !found {continue} - if version,known:=versions[file.kind];known && file.kind_version!=version {return false} - } - return true -} - -// The version of each kind of asset this build reads: those of the engine -// and those the plugins of the engine and of the project at or above root -// declare. -@(private="file") -kind_versions :: proc(root:string,allocator:=context.allocator) -> map[asset.Fourcc]u16 { - versions:=make(map[asset.Fourcc]u16,allocator) - for info in asset.KINDS {versions[info.kind]=info.version} - names:=project.FOLDER_NAMES - plugins:=make([dynamic]project.Plugin,context.temp_allocator) - if engine:=project.engine_root_find("",context.temp_allocator);engine!="" { - _=project.plugins_discover(project.path(engine,names[.Plugins],allocator=context.temp_allocator),.Engine,&plugins,context.temp_allocator) - } - for folder:=root;folder!="" && folder!=".";folder=filepath.dir(folder) { - if os.is_file(project.path(folder,project.PROJECT_FILE,allocator=context.temp_allocator)) { - _=project.plugins_discover(project.path(folder,names[.Plugins],allocator=context.temp_allocator),.Project,&plugins,context.temp_allocator) - break - } - if filepath.dir(folder)==folder {break} - } - for &plugin in plugins { - for &kind in plugin.manifest.asset_kinds { - if letters:=base.text_get(kind.kind[:]);len(letters)==4 {versions[asset.fourcc(letters)]=kind.version} - } - } - return versions -} - -@(private="file") -Library_Read :: struct {using imports:Library_Imports,read:bool} - -@(private="file") -library_read :: proc(library:^Library_Read,options:Sync_Options,top:string,import_options:^importer.Import_Options) { - if library.read || options.library==nil {return} - library.imports=options.library(top,context.temp_allocator) - import_options.world=library.world - library.read=true -} diff --git a/cook/import/sources/write_test.odin b/cook/import/sources/write_test.odin index 844a45586..72d834b92 100644 --- a/cook/import/sources/write_test.odin +++ b/cook/import/sources/write_test.odin @@ -14,6 +14,7 @@ import stbi "vendor:stb/image" import "mirai:asset" import anim "mirai:std/anim" import "mirai:std/base" +import "mirai:store" import mm "mirai:std/math" import "mirai:std/serial" import importer "mirai:cook/import" @@ -69,14 +70,14 @@ a_written_obj_is_read_back_with_its_material_and_its_dependencies :: proc(t:^tes model,ok:=import_ok(t,FIXTURES+"concave.obj") if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder)) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder)) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.assets),2) types:asset.Types;asset.types_init(&types);defer asset.types_destroy(&types) mesh_written,has_mesh:=written_of(&report,asset.KIND_MESH,"ConcaveL") material_written,has_material:=written_of(&report,asset.KIND_MATERIAL,"red") - testing.expect(t,has_mesh && has_material && strings.contains(mesh_written.path,"/.mirai/imported/") && strings.has_suffix(material_written.path,"/assets/concave/red.mat") && mesh_written.bytes>0) + testing.expect(t,has_mesh && has_material && strings.contains(mesh_written.path,"/store/blobs/") && strings.has_suffix(material_written.path,"/assets/concave/red.mat") && mesh_written.bytes>0) if !has_mesh || !has_material {return} a,loaded:=read_asset(t,mesh_written,&types) @@ -108,7 +109,7 @@ a_written_model_with_textures_links_materials_to_them :: proc(t:^testing.T) { model,ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder) + report,error:=write_into(t,&model,folder) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),0) @@ -129,7 +130,7 @@ a_written_model_with_textures_links_materials_to_them :: proc(t:^testing.T) { plain_folder:=base.test_folder(t,"write-plant-rgba8") if plain_folder=="" {return} - plain_report,plain_error:=write_into(&model,plain_folder,{textures=.Uncompressed,ignore_stored=true}) + plain_report,plain_error:=write_into(t,&model,plain_folder,{textures=.Uncompressed,ignore_stored=true}) defer importer.report_destroy(&plain_report) testing.expect_value(t,plain_error.code,importer.Error_Code.None) if plain_written,found:=written_of(&plain_report,asset.KIND_TEXTURE,"leaves_color");testing.expect(t,found) { @@ -162,7 +163,7 @@ a_skinned_mesh_is_written_with_its_rig :: proc(t:^testing.T) { tracks:=[1]asset.Animation_Track_Desc{{target=base.text_get(model.skeletons[0].bones[len(model.skeletons[0].bones)-1].path[:]),property=.Rotation,keys=keys[:]}} bend:=[1]asset.Animation_Clip_Desc{{name="bend",tracks=tracks[:]}} brought:=model.animations;model.animations=bend[:] - report,error:=write_into(&model,folder) + report,error:=write_into(t,&model,folder) model.animations=brought defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) @@ -215,7 +216,7 @@ the_prefab_of_an_import_is_written_in_the_layout_of_the_game :: proc(t:^testing. if root=="" {return} world:=prefab.import_world(t,c.animated) if world==nil {return} - report,error:=import_source(copy_in(root,DEMO,..c.files),{world=world,root=root,textures=.Uncompressed}) + report,error:=import_test(t,copy_in(root,DEMO,..c.files),{world=world,root=root,textures=.Uncompressed}) if !testing.expectf(t,error.code==.None,"%s: %s",c.files[0],error.message) {return} defer importer.report_destroy(&report) p:=author.Project{name="layout",assets_root=assets,info=world} @@ -261,7 +262,7 @@ clips_onto_a_skeleton_write_only_the_anim :: proc(t:^testing.T) { bend:=[2]asset.Animation_Clip_Desc{{name="bend",tracks=tracks[:]},{name="cue",markers=cue[:]}} brought:=model.animations;model.animations=bend[:] player:=base.Asset_Id{0x1234,0x5678} - report,error:=write_into(&model,folder,{settings={skeleton=player}}) + report,error:=write_into(t,&model,folder,{settings={skeleton=player}}) model.animations=brought defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) @@ -288,7 +289,7 @@ a_humanoid_skeleton_is_written_with_its_body_and_keeps_what_a_person_set :: proc model,ok:=import_ok(t,DEMO+"characters/mirai/mirai.glb",options) if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder,options) + report,error:=write_into(t,&model,folder,options) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) written,found:=written_of(&report,asset.KIND_SKELETON,model.skeletons[0].name) @@ -372,7 +373,7 @@ imported_clips_that_move_the_bone_above_the_pelvis_carry_it_as_their_root :: pro {name="mixamo",tracks={{target=hips,property=.Position,keys=bob[:]}}}, } brought:=model.animations;model.animations=clips[:] - report,error:=write_into(&model,folder,options) + report,error:=write_into(t,&model,folder,options) model.animations=brought defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) @@ -430,7 +431,7 @@ importing_again_keeps_what_was_changed_in_a_material_and_takes_the_rest_from_the import_once :: proc(t:^testing.T,source,folder:string,options:importer.Import_Options,types:^asset.Types) -> string { model,ok:=import_ok(t,source,options);if !ok {return ""} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder,options);defer importer.report_destroy(&report) + report,error:=write_into(t,&model,folder,options);defer importer.report_destroy(&report) if !testing.expect_value(t,error.code,importer.Error_Code.None) {return ""} written,found:=written_of(&report,asset.KIND_MATERIAL,"Terracotta") if !testing.expect(t,found) {return ""} @@ -467,14 +468,14 @@ importing_again_keeps_the_identity_of_every_asset :: proc(t:^testing.T) { model,ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !ok {return} defer importer.model_destroy(&model) - first,first_error:=write_into(&model,fmt.tprintf("%s/one",folder)) + first,first_error:=write_into(t,&model,fmt.tprintf("%s/one",folder)) defer importer.report_destroy(&first) testing.expect_value(t,first_error.code,importer.Error_Code.None) again,again_ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !again_ok {return} defer importer.model_destroy(&again) - second,second_error:=write_into(&again,fmt.tprintf("%s/one",folder)) + second,second_error:=write_into(t,&again,fmt.tprintf("%s/one",folder)) defer importer.report_destroy(&second) testing.expect_value(t,second_error.code,importer.Error_Code.None) testing.expect_value(t,len(first.assets),len(second.assets)) @@ -482,23 +483,34 @@ importing_again_keeps_the_identity_of_every_asset :: proc(t:^testing.T) { testing.expectf(t,item.id==second.assets[i].id && item.path==second.assets[i].path,"%s keeps its id and its file",item.name) testing.expect(t,item.id!=base.Asset_Id{}) } + testing.expect(t,first.manifest==second.manifest,"and its package is the same manifest") ids:=make(map[base.Asset_Id]bool,context.temp_allocator) for item in first.assets { testing.expect(t,!ids[item.id],"ids are distinct") ids[item.id]=true } - // The documents beside the source and what nobody edits in the imported folder, with the sidecar that lists them. + // The documents in the folder of the import, and every asset a blob its package lists, with + // the prefab import_write leaves to its caller. documents,_:=os.read_all_directory_by_path(fmt.tprintf("%s/one/assets/plant",folder),context.temp_allocator) - cached,_:=os.read_all_directory_by_path(fmt.tprintf("%s/one/.mirai/imported",folder),context.temp_allocator) - testing.expect(t,len(documents)+len(cached)==len(first.assets) && os.exists(fmt.tprintf("%s/one/assets/plant.gltf.import",folder))) - - // A copy of a project, its sources and their sidecars with its documents, imports again with the ids of its assets. + s:=test_store(t,fmt.tprintf("%s/one/store",folder)) + p,_:=asset.package_get(s,first.manifest,context.temp_allocator) + cached:=0 + for a in p.assets {cached+=int(asset.import_cached(a.kind))} + testing.expectf(t,len(documents)+cached==len(first.assets) && len(p.assets)==len(first.assets)+int(first.prefab!={}) && !os.exists(fmt.tprintf("%s/one/assets/plant.gltf.import",folder)),"%d documents, %d of the cache, %d listed, %d written",len(documents),cached,len(p.assets),len(first.assets)) + + // A copy of a project, its sources and documents, imported with the manifest the project pins, keeps the ids of its assets. + shared:=test_store(t,fmt.tprintf("%s/shared",folder)) copy_in(fmt.tprintf("%s/a",folder),DEMO,"gallery/vase.obj","gallery/vase.mtl","gallery/vase_color.png") ids_by_root:[2][dynamic]base.Asset_Id + pinned:store.Hash for root,i in ([]string{"a","b"}) { - if i==1 {testing.expect(t,os.copy_directory_all(fmt.tprintf("%s/b/assets",folder),fmt.tprintf("%s/a/assets",folder))==nil)} - report,vase_error:=import_source(fmt.tprintf("%s/%s/assets/gallery/vase.obj",folder,root),{world=prefab.import_world(t)}) + if i==1 { + os.make_directory_all(fmt.tprintf("%s/b",folder)) + testing.expect(t,os.copy_directory_all(fmt.tprintf("%s/b/assets",folder),fmt.tprintf("%s/a/assets",folder))==nil) + } + report,vase_error:=import_test(t,fmt.tprintf("%s/%s/assets/gallery/vase.obj",folder,root),{world=prefab.import_world(t),store=shared,manifest=pinned}) testing.expect_value(t,vase_error.code,importer.Error_Code.None) + pinned=report.manifest ids_by_root[i].allocator=context.temp_allocator for item in report.assets {append(&ids_by_root[i],item.id)} importer.report_destroy(&report) @@ -518,7 +530,7 @@ images_that_are_missing_or_not_images_leave_a_warning_and_no_reference :: proc(t if !ok {return} defer importer.model_destroy(&model) testing.expect_value(t,len(model.warnings),0) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder)) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder)) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),2) @@ -556,7 +568,7 @@ an_embedded_image_is_a_texture_of_its_whole_size :: proc(t:^testing.T) { defer importer.model_destroy(&model) testing.expect_value(t,len(model.images),1) testing.expect(t,model.images[0].path=="" && len(model.images[0].data)>0 && model.images[0].name=="wood") - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),0) @@ -639,7 +651,7 @@ gltf_and_glb_write_pbr_maps_with_color_spaces_channels_and_independent_uvs :: pr testing.expect(t,(image.path=="")==binary && (len(image.data)>0)==binary) } options:=importer.Import_Options{root=folder,textures=.Uncompressed} - report,error:=write_into(&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),options) + report,error:=write_into(t,&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),options) if !testing.expect_value(t,error.code,importer.Error_Code.None) {continue} defer importer.report_destroy(&report) testing.expect_value(t,len(report.warnings),0) @@ -697,7 +709,7 @@ gltf_and_glb_write_pbr_maps_with_color_spaces_channels_and_independent_uvs :: pr if testing.expect(t,asset.mesh_level_decode(mesh.geometry,0,&vertices,&indices)) {testing.expect_value(t,vertices[0].uv1,[2]f32{0.25,0.75})} } } - blocks,blocks_error:=write_into(&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),{textures=.Blocks,ignore_stored=true}) + blocks,blocks_error:=write_into(t,&model,fmt.tprintf("%s/%s",folder,"glb-out" if binary else "gltf-out"),{textures=.Blocks,ignore_stored=true}) if !testing.expect_value(t,blocks_error.code,importer.Error_Code.None) {continue} defer importer.report_destroy(&blocks) testing.expect_value(t,len(blocks.warnings),0) @@ -721,24 +733,31 @@ a_standalone_image_becomes_a_texture :: proc(t:^testing.T) { if folder=="" {return} path:=fmt.tprintf("%s/Bricks Color.png",folder) _=os.write_entire_file(path,png_bytes(16,16,{150,60,40})) - report,error:=importer.import_texture(path,{root=root,image=.Color}) + options:=test_options(t,path,{root=root,image=.Color}) + report,error:=importer.import_texture(path,options) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) - testing.expect_value(t,len(report.assets),1) - testing.expect(t,report.assets[0].id!={} && strings.contains(report.assets[0].path,"/.mirai/imported/")) - again,again_error:=importer.import_texture(path,{root=root,image=.Normal,ignore_stored=true}) + if !testing.expect_value(t,len(report.assets),1) {return} + testing.expect(t,report.assets[0].id!={} && strings.contains(report.assets[0].path,"/store/blobs/")) + normal:=options + normal.image,normal.ignore_stored=.Normal,true + again,again_error:=importer.import_texture(path,normal) defer importer.report_destroy(&again) testing.expect_value(t,again_error.code,importer.Error_Code.None) - testing.expect(t,again.assets[0].id==report.assets[0].id && again.assets[0].path!=report.assets[0].path && !os.exists(report.assets[0].path),"imported again as a normal map, the image is the same texture, and its file of before goes") + if !testing.expect_value(t,len(again.assets),1) {return} + testing.expect(t,again.assets[0].id==report.assets[0].id && again.assets[0].path!=report.assets[0].path && os.exists(report.assets[0].path),"imported again as a normal map, the image is the same texture, another blob, and the store keeps the one of before") _=os.write_entire_file(path,[]u8{1,2,3}) - _,bad:=importer.import_texture(path,{root=root,image=.Color}) + _,bad:=importer.import_texture(path,options) testing.expect_value(t,bad.code,importer.Error_Code.Unsupported) - testing.expect(t,os.exists(again.assets[0].path),"a failed import leaves the files it had") - _,missing:=importer.import_texture(fmt.tprintf("%s/none.png",folder),{root=root,image=.Color}) + testing.expect(t,os.exists(again.assets[0].path),"a failed import leaves the blobs it had") + _,missing:=importer.import_texture(fmt.tprintf("%s/none.png",folder),options) testing.expect_value(t,missing.code,importer.Error_Code.Not_Found) - _,outside:=importer.import_texture(fmt.tprintf("%s/outside.png",root),{root=root,image=.Color}) - testing.expect(t,outside.code==.Not_Found && strings.contains(outside.message,"is not in the assets folder"),outside.message) + outside_path:=fmt.tprintf("%s/outside.png",root) + _=os.write_entire_file(outside_path,png_bytes(4,4,{1,2,3})) + outside,outside_error:=importer.import_texture(outside_path,options) + defer importer.report_destroy(&outside) + testing.expectf(t,outside_error.code==.None && len(outside.assets)==1,"an image outside the assets imports too: %s",outside_error.message) } // A Radiance image is an image whose light above 1 the texture keeps, in BC6H. @@ -752,7 +771,7 @@ a_radiance_image_becomes_an_hdr_texture :: proc(t:^testing.T) { for _ in 0..<8 {append(&hdr,200,200,200,135)} // 100 _=os.write_entire_file(path,hdr[:]) testing.expect_value(t,source_kind(path),Source_Kind.Image) - report,error:=import_source(path,{world=prefab.import_world(t)}) + report,error:=import_test(t,path,{world=prefab.import_world(t)}) defer importer.report_destroy(&report) if !testing.expectf(t,error.code==.None,"%s",error.message) {return} a:asset.Asset @@ -783,7 +802,7 @@ embedded_images_with_the_same_name_keep_distinct_texture_ids :: proc(t:^testing. "accessors":[{"bufferView":0,"componentType":5126,"count":3,"type":"VEC3","min":[0,0,0],"max":[1,1,0]},{"bufferView":1,"componentType":5126,"count":3,"type":"VEC2"}]}`},context.temp_allocator) path:=write_text(folder,"names.gltf",source) model,ok:=import_ok(t,path);if !ok {return};defer importer.model_destroy(&model) - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{textures=.Uncompressed}) if !testing.expect_value(t,error.code,importer.Error_Code.None) {return};defer importer.report_destroy(&report) testing.expect(t,len(report.image_ids)==2 && report.image_ids[0]!=report.image_ids[1]) testing.expect_value(t,len(report.warnings),0) @@ -815,7 +834,7 @@ names_are_made_safe_and_unique_as_file_names :: proc(t:^testing.T) { defer importer.model_destroy(&model) testing.expect_value(t,len(model.meshes),3) testing.expect(t,model.meshes[0].name!=model.meshes[2].name,"a repeated name is made unique in the model") - report,error:=write_into(&model,fmt.tprintf("%s/out",folder)) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder)) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) seen:=make(map[string]bool,context.temp_allocator) @@ -834,9 +853,9 @@ a_destination_that_cannot_be_a_folder_fails_without_leaving_files :: proc(t:^tes if !ok {return} defer importer.model_destroy(&model) blocker:=write_text(folder,"blocker","this is a file") - _,error:=write_into(&model,blocker) + _,error:=write_into(t,&model,blocker) testing.expect_value(t,error.code,importer.Error_Code.Write) - _,error=write_into(&model,fmt.tprintf("%s/inside",blocker)) + _,error=write_into(t,&model,fmt.tprintf("%s/inside",blocker)) testing.expect_value(t,error.code,importer.Error_Code.Write) } @@ -856,7 +875,7 @@ every_fixture_imports_writes_and_reads_back_with_its_dependencies_resolved :: pr model,ok:=import_ok(t,source,formats=text[:]) if !ok {continue} defer importer.model_destroy(&model) - report,error:=write_into(&model,out) + report,error:=write_into(t,&model,out) defer importer.report_destroy(&report) testing.expectf(t,error.code==.None && len(report.warnings)==0,"%s: %v %s",source,error.code,error.message) counts:[3]int @@ -881,8 +900,7 @@ every_fixture_imports_writes_and_reads_back_with_its_dependencies_resolved :: pr } testing.expectf(t,counts[0]==len(model.meshes) && counts[1]==len(model.materials),"%s: every mesh and material is an asset",source) registry:asset.Registry - asset.registry_init(®istry,fmt.tprintf("%s/assets",out)) - testing.expect_value(t,asset.registry_scan(®istry),asset.Error.None) + if !registry_with(t,out,&report,®istry) {continue} missing:[dynamic]asset.Registry_Missing missing.allocator=context.temp_allocator asset.registry_missing(®istry,&missing) @@ -898,7 +916,7 @@ the_report_names_the_asset_of_every_row_of_the_model :: proc(t:^testing.T) { model,ok:=import_ok(t,DEMO+"gallery/plant.gltf") if !ok {return} defer importer.model_destroy(&model) - report,error:=write_into(&model,folder) + report,error:=write_into(t,&model,folder) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.mesh_ids),1) @@ -951,7 +969,7 @@ gltf_anisotropy_with_a_roughness_texture_is_converted_texel_by_texel :: proc(t:^ base:=f32(128)/255 testing.expectf(t,abs(along-(base*base+(1-base*base)*0.36))<0.01 && abs(across-base*base)<0.01,"alphas %v and %v",along,across) testing.expect(t,model.images[rough].made.texels[0]==255 && model.images[rough].made.texels[2]==64,"occlusion and metallic stay") - report,error:=write_into(&model,fmt.tprintf("%s/out",folder),{}) + report,error:=write_into(t,&model,fmt.tprintf("%s/out",folder),{}) defer importer.report_destroy(&report) testing.expect_value(t,error.code,importer.Error_Code.None) testing.expect_value(t,len(report.warnings),0) diff --git a/cook/textures/cache.odin b/cook/textures/cache.odin index 877163337..1bf2e3902 100644 --- a/cook/textures/cache.odin +++ b/cook/textures/cache.odin @@ -1,6 +1,5 @@ package textures -import "core:hash/xxhash" import "core:mem" import "core:slice" import "mirai:asset" @@ -17,7 +16,7 @@ CACHE :: "textures" // An image built with options: its bytes, encoded, or the texels of made // (whose width is 0 for encoded bytes). cache_key_build :: proc(bytes:[]u8,made:Source,options:Build_Options) -> u64 { - key:=base.hash_combine(xxhash.XXH3_64_default(bytes),u64(u32(made.width))<<32|u64(u32(made.height))) + key:=base.hash_combine(base.hash64(bytes),u64(u32(made.width))<<32|u64(u32(made.height))) key=base.hash_combine(key,u64(u32(made.layers))|u64(made.format)<<32|u64(made.cube)<<40) return base.hash_combine(key,settings_hash(options))|1 } diff --git a/tools/demo_content/main.odin b/tools/demo_content/main.odin index eac6e56aa..f2101fce7 100644 --- a/tools/demo_content/main.odin +++ b/tools/demo_content/main.odin @@ -11,7 +11,7 @@ import "mirai:std/base" // on every run. What is authored (scenes, prefabs, interfaces, materials, // particles, tables of game data, clips of a few keys) is changed through the // API or the editor, and this names what it uses by its id; the sources it -// writes beside their assets are imported by the build (sources.sync). +// writes beside their assets are ingested and added with mirai import. // // mirai content demo [-- --physics-only|--features-only|--animation-only|--interface-only|--dig-only] diff --git a/tools/mirai/main.odin b/tools/mirai/main.odin index 6acc17b6f..ed156d0ee 100644 --- a/tools/mirai/main.odin +++ b/tools/mirai/main.odin @@ -49,7 +49,7 @@ COMMANDS :: [?]Command{ {"ui",UI_USAGE,"Reads, writes and checks the interfaces of a project as the documents ui.inspect gives and ui.write takes, with no editor",cmd_ui}, {"scene",SCENE_USAGE,"Reads, writes and checks the scenes of a project as the documents scene.inspect gives with root and scene.write takes, with no editor",cmd_scene}, {"content",CONTENT_USAGE,"Runs the content tool of a project, which writes the assets it makes from code",cmd_content}, - {"import","import [--max-texture N]","Imports a model, an image, a sound, a font, CSV texts or a prompts manifest in the assets of a project or a plugin as .mesh, .mat, .tex, .sound, .font, .strings and .prompts assets",cmd_import}, + {"import","import [--project P] [--folder F] [settings] | import --refresh ... [--fast]","Ingests a model, an image, a sound, a font or a prompts manifest from anywhere into the library of the machine store and adds it to a project; --refresh brings the packages a project or plugin pins to builds this engine reads",cmd_import}, } main :: proc() { diff --git a/tools/mirai/package.odin b/tools/mirai/package.odin index 76401594c..abbb7f082 100644 --- a/tools/mirai/package.odin +++ b/tools/mirai/package.odin @@ -117,7 +117,7 @@ cmd_package :: proc(root:string,args:[]string) -> int { for plugin in plugins { if os.is_dir(join(out,"plugins",plugin.name,"assets")) {append(&imported,join(out,"plugins",plugin.name))} } - if ok,message:=build.imports_sync(build.engine_paths(out,.Release,context.temp_allocator).import_program,imported[:],fast=false);!ok { + if ok,message:=build.imports_refresh(build.engine_paths(out,.Release,context.temp_allocator).import_program,imported[:],fast=false);!ok { fmt.eprintfln("mirai package: could not import the sources of the plugins: %s",message) return 1 } From fed408bbae3fb1bb61a49c17f5c7d33a7966c672 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 19:32:46 -0300 Subject: [PATCH 017/123] exchange: blobs and refs of the store between peers over net A new layer-3 package moves blobs and refs between peers on the connections of net hosts its owner runs: stateless requests by address and runs of pieces, one answer with size, table and first pieces, receiver-owned priority with two requests in flight per peer, pieces spread over several sources, liars in the table, a piece or the size marked and routed around, refs asked and listed under an owner's expose procedure, a blocking fetch for jobs, and every peer message validated. The store serves the table of a blob it is still receiving, and a last piece that proves the begun size wrong returns Wrong_Size. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-networking/SKILL.md | 2 + .agents/skills/mirai-store/SKILL.md | 19 +- .../skills/mirai-store/references/exchange.md | 64 ++ asset/packages.odin | 4 +- cook/build/engine.odin | 2 +- docs/ARCHITECTURE.md | 2 + exchange/ask.odin | 435 +++++++++++++ exchange/exchange.odin | 597 ++++++++++++++++++ exchange/exchange_test.odin | 326 ++++++++++ exchange/harness_test.odin | 244 +++++++ exchange/hostile_test.odin | 140 ++++ exchange/serve.odin | 204 ++++++ exchange/wire.odin | 188 ++++++ store/blobs.odin | 23 +- store/receive.odin | 12 +- store/store.odin | 1 + store/store_test.odin | 15 +- tools/mirai/package.odin | 2 +- 18 files changed, 2264 insertions(+), 16 deletions(-) create mode 100644 .agents/skills/mirai-store/references/exchange.md create mode 100644 exchange/ask.odin create mode 100644 exchange/exchange.odin create mode 100644 exchange/exchange_test.odin create mode 100644 exchange/harness_test.odin create mode 100644 exchange/hostile_test.odin create mode 100644 exchange/serve.odin create mode 100644 exchange/wire.odin diff --git a/.agents/skills/mirai-networking/SKILL.md b/.agents/skills/mirai-networking/SKILL.md index a2498377c..ea2d82a07 100644 --- a/.agents/skills/mirai-networking/SKILL.md +++ b/.agents/skills/mirai-networking/SKILL.md @@ -41,6 +41,8 @@ neither the data world nor physics. A replicated object is an opaque game id plu [clock and inputs](references/networking.md#time-and-inputs), [interpolation](references/networking.md#smoothing-and-interpolation-interpolationodin) and [known limits](references/networking.md#known-limits) +- Blobs and refs of the store between peers on a host's connections → + [exchange](../mirai-store/references/exchange.md#what-the-owner-calls) - Opening a host's UDP port (`mapping_start`, `mapping_poll`, `mapping_stop`, `mapping_close`; PCP, NAT-PMP, UPnP) → [API and lifecycle](references/port-mapping.md#api-and-lifecycle), [protocols](references/port-mapping.md#protocols) and diff --git a/.agents/skills/mirai-store/SKILL.md b/.agents/skills/mirai-store/SKILL.md index 989c54613..ac2600518 100644 --- a/.agents/skills/mirai-store/SKILL.md +++ b/.agents/skills/mirai-store/SKILL.md @@ -2,9 +2,11 @@ name: mirai-store description: Mirai machine-wide content-addressed store (the `store` package) - blobs named by their BLAKE3 address and cut in pieces, receiving a blob piece by piece with resume, range reads of - partial blobs, refs that name an address, usage and sweep, and the default root. Use when - storing or sharing heavy content between projects or peers, building a blob transport, an asset - library or a derived data cache on top of it, or diagnosing a corrupt or missing blob. + partial blobs, refs that name an address, usage and sweep, and the default root - and the + `exchange` package that moves blobs and refs between peers over `net` (wants, priorities, + several sources, lying peers, blocking fetch). Use when storing or sharing heavy content between + projects or peers, building an asset library or a derived data cache on top of it, fetching a + blob from another machine, or diagnosing a corrupt, missing or slow blob. --- # Mirai store @@ -21,11 +23,15 @@ derived data cache that use it decide what a ref means and what is live. - Receiving a blob piece by piece and range reads of a partial one: `store/receive.odin`. - Refs and their names: `store/refs.odin`. - Usage and sweep: `store/space.odin`. +- Moving blobs and refs between peers (`exchange`): what its owner calls → + [exchange](references/exchange.md#what-the-owner-calls), its rules → + [invariants](references/exchange.md#invariants), its tests → + [verification](references/exchange.md#verification). ## Workflow -1. A sender serves the piece table of a complete blob and its pieces; it reads ranges by path or - with the range read. +1. A sender serves the piece table of a complete blob, or of one it is still receiving, and its + pieces; it reads ranges by path or with the range read. 2. A receiver begins from the address and the table, gives each piece as it arrives in any order, and asks which pieces are missing to request them again, also after a restart. 3. A layer that names content sets a ref in its own namespace; it sweeps with the set of addresses @@ -40,7 +46,8 @@ derived data cache that use it decide what a ref means and what is live. bytes a piece, never its bytes. The root version carries the address rule: a root of another rule is refused, never migrated. - A table does not carry the length of its last piece: a size with the right piece count passes - the table check, and then no true last piece fits it, so nothing is published. + the table check, and then the true last piece returns `Wrong_Size` instead of being kept, so + nothing is published and the receiver begins again with the size the piece proves. - Only the two private procedures of `address.odin` (the value of a piece, the address of a table) touch the hash primitive; everything else, tests included, goes through them or the API, so the primitive can be replaced in one place. diff --git a/.agents/skills/mirai-store/references/exchange.md b/.agents/skills/mirai-store/references/exchange.md new file mode 100644 index 000000000..2f690c993 --- /dev/null +++ b/.agents/skills/mirai-store/references/exchange.md @@ -0,0 +1,64 @@ +# Exchange: blobs and refs between peers + +`exchange` (layer 3) moves the blobs and refs of a `store` between peers over the connections of +`net` hosts its owner already runs. It imports `store` and `net` and nothing engine-time, so a +headless peer and the editor link the same code. Its messages are in `exchange/wire.odin`. + +## What the owner calls + +- `channels()` gives the two reliable channels the exchange needs; the owner puts them in the + `Host_Desc.channels` of every host at the same index and passes that index as `Desc.channel`. +- `init` sizes every buffer from the `Desc`; `Desc.max_blob_bytes` decides the largest table, and a + table of that size is held per peer to send and to receive. +- Per host poll: `peer_add` on `Connected`, `peer_remove` on `Disconnected`, `receive` on every + `Message` (it says whether the message was its own). Then `update(now)` before `host_update`. +- A connection is given from its start: a message that began before `peer_add` is not read from + its middle, so a restarted exchange takes new connections. +- `want` (asking again raises the priority), `forget`, `progress`, `peer_progress`; `ref_ask`, + `names_ask`, `query_result`, `query_end`. +- `fetch` and `fetch_ref` block a job thread until the blob is complete in the store, or return + `Missing` once no connected peer has it; the owner keeps pumping meanwhile, since every procedure + takes the exchange's lock. + +## Invariants + +- Requests are stateless, by address and runs of pieces; there is no session or fetch id. Any peer + that has the pieces answers, also from a blob it is still receiving (`store.read` and + `store.blob_table` work on partial blobs). +- The answer to a blob carries its size, its table and its first pieces; adjacent pieces nobody + was asked for go as one run. A request is bounded by `REQUEST_ENTRIES` and `REQUEST_BYTES`. +- Priority belongs to the receiver: it asks for the most important wants first and nothing about + priority goes on the wire beyond the order of the entries. At most `BATCHES` requests are in + flight per peer, each sized to what the peer delivers in a round trip, so they keep twice that + outstanding without queueing a low priority want ahead of a later high one. +- Pieces are spread among the connected peers that do not lack the blob; a peer that lacks it is + not asked again for a while, and a want with no peer left to ask is `Missing`, asked again when + a peer joins. +- Nothing unverified reaches a reader: tables and pieces go through `receive_begin` and + `receive_piece`. A table that does not match its address, or a piece that does not match its + table, marks that peer wrong for that blob; a true last piece that proves the size wrong + (`Wrong_Size`) marks whoever gave the table, removes the partial blob and asks the table of + others. A peer stays wrong for a blob until it leaves. +- What a peer exposes is the owner's `Desc.expose`; without one no namespace is read. Names come + sorted, a page at a time after the last one seen. +- Hostile input: a malformed message, a request beyond the limits or an answer nobody could have + asked for in that form marks the peer bad, and it is neither asked nor served again; a well formed + message about something not asked for is ignored. A request that does not fit the room for + `BATCHES` requests is refused, and answers to questions find room only while the asker keeps a + few waiting, so a peer cannot make another send without bound. +- Serving and receiving do not allocate after `init`, except listing a namespace, for which the + store reads a folder. + +## Verification + +- `mirai check exchange` and `mirai test exchange` (no GPU): every test runs peers on `net`'s + `Sim_Network` in virtual time with temporary stores. +- Focal tests: `blobs_around_the_piece_boundaries_and_a_large_one_arrive_verified_under_loss`, + `a_high_priority_want_asked_later_completes_before_a_large_low_one`, + `a_transfer_resumes_after_the_connection_drops_and_after_the_store_reopens`, + `a_liar_in_the_table_a_piece_or_the_size_is_marked_and_the_blob_still_arrives`, + `a_peer_still_receiving_a_blob_serves_its_pieces_to_a_third`, + `refs_are_asked_and_listed_and_a_namespace_not_exposed_gives_nothing`, + `the_blocking_form_returns_for_a_hit_a_miss_and_a_disconnect`, + `hostile_messages_are_refused_without_a_crash` and + `a_game_channel_keeps_its_delay_small_during_a_transfer`. diff --git a/asset/packages.odin b/asset/packages.odin index 8fae37651..084446188 100644 --- a/asset/packages.odin +++ b/asset/packages.odin @@ -571,7 +571,7 @@ packages_resolve :: proc(s:^store.Store,list:^Packages,allocator:=context.alloca for o in b.outputs {outputs[o.id]=o} case .Missing: case .Out_Of_Memory: return {},.Out_Of_Memory - case .Not_Here_Yet,.Corrupt,.Invalid_Argument,.Io,.Unsupported_Layout: + case .Not_Here_Yet,.Corrupt,.Wrong_Size,.Invalid_Argument,.Io,.Unsupported_Layout: problem_add(&resolved,{entry.id,{},.Build_Unreadable}) or_return } folder:=entry.folder!=""?fmt.tprintf("%s/",entry.folder):"" @@ -945,7 +945,7 @@ store_error :: proc(error:store.Error) -> Error { switch error { case .None: return .None case .Missing,.Not_Here_Yet: return .Not_Found - case .Corrupt,.Unsupported_Layout: return .Corrupt + case .Corrupt,.Wrong_Size,.Unsupported_Layout: return .Corrupt case .Invalid_Argument: return .Invalid case .Out_Of_Memory: return .Out_Of_Memory case .Io: return .IO diff --git a/cook/build/engine.odin b/cook/build/engine.odin index b6823ae07..5aa1ee4e1 100644 --- a/cook/build/engine.odin +++ b/cook/build/engine.odin @@ -81,7 +81,7 @@ ENGINE_PACKAGES :: [?]Engine_Package{ {.Base,"audio","system"},{.Base,"platform","system"},{.Base,"rhi","system"}, {.Base,"project","system"},{.Base,"store","system"},{.Base,"core",""}, - {.Engine,"asset",""},{.Engine,"net",""},{.Engine,"render/abi","render"},{.Engine,"render","render"},{.Engine,"third_party/freetype","freetype"}, + {.Engine,"asset",""},{.Engine,"net",""},{.Engine,"exchange",""},{.Engine,"render/abi","render"},{.Engine,"render","render"},{.Engine,"third_party/freetype","freetype"}, {.Engine,"ui","ui"},{.Engine,"host",""},{.Engine,"engine/input","engine"},{.Engine,"engine/scene","engine"},{.Engine,"engine/volume","engine"}, {.Engine,"engine/render","engine"},{.Engine,"engine/audio","engine"},{.Engine,"engine/gui","engine"},{.Engine,"engine/camera","engine"},{.Engine,"engine/save","engine"}, diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 80935b647..3c8967f1b 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -336,6 +336,8 @@ Numbered layers; a package only imports lower layers. and keeps in `store` the records of imports (package and build manifests, `packages.mirai`). `rhi/shared_device` is the device the GPU tests share; only tests import it, also the GPU tests of `rhi`, in `rhi/tests`. +- **3** `exchange`: blobs and refs of `store` between peers over the connections of `net` hosts; + a headless peer and the editor link it. - **3** `render`, `platform`: hybrid renderer and SDL3. `render/abi` declares the constants the renderer shares with its shaders, on top of `asset` alone: `cook/shaders` imports it to write them for the shaders, since the tool cannot import `render`. diff --git a/exchange/ask.odin b/exchange/ask.odin new file mode 100644 index 000000000..397c85644 --- /dev/null +++ b/exchange/ask.odin @@ -0,0 +1,435 @@ +package exchange + +import "base:intrinsics" +import "core:mem" +import "core:slice" +import "mirai:store" +import wire "mirai:net" + +@(private) +want_find :: proc(x:^Exchange,address:store.Hash) -> int { + for k in 0.. (int,Error) { + if i:=want_find(x,address);i>=0 { + w:=&x.wants[i] + if priority>w.priority { + w.priority=priority + order_sort(x) + } + return i,.None + } + for &w,i in x.wants { + if w.used {continue} + generation:=w.generation+1 + have,asked:=w.have,w.asked + w={used=true,generation=generation,address=address,priority=priority,order=x.sequence,table_peer=-1,have=have,asked=asked} + x.sequence+=1 + x.order[x.want_count]=i + x.want_count+=1 + order_sort(x) + x.dirty=true + if store.has(x.desc.store,address) { + want_complete(x,i) + return i,.None + } + // A blob already being received resumes from the pieces it has. + size,size_error:=store.blob_size(x.desc.store,address) + if size_error==.None && size<=x.desc.max_blob_bytes { + arena:mem.Arena + if _,table_error:=store.blob_table(x.desc.store,address,scratch_arena(x,&arena));table_error==.None {table_known(x,i,-1,size)} + } + return i,.None + } + return -1,.Full +} + +@(private) +want_release :: proc(x:^Exchange,i:int) { + x.wants[i].used=false + for k in 0..0 && want_before(x,order[j],order[j-1]);j-=1 { + order[j],order[j-1]=order[j-1],order[j] + } + } +} + +@(private) +want_before :: proc(x:^Exchange,a,b:int) -> bool { + wa,wb:=&x.wants[a],&x.wants[b] + return wa.priority>wb.priority || wa.priority==wb.priority && wa.order bool {return bits[i>>6]&(u64(1)<>6]|=u64(1)<>6]&=~(u64(1)< int { + for word:=from>>6;word*64>6 {free&=~u64(0)<=w.count || !bit_has(w.have,m) {continue} + bit_off(w.have,m) + w.had-=1 + w.here-=i64(store.piece_size(size,m)) + } + if p>=0 { + // The first pieces asked with the table come in the same answer. + peer:=&x.peers[p] + for k in 0.. int { + p:=&x.peers[i] + rtt:=f64(0) + if stats,ok:=wire.connection_stats(p.host,p.connection);ok {rtt=f64(stats.rtt)} + return int(clamp(p.receive_rate*rtt,MIN_BATCH_BYTES,REQUEST_BYTES)) +} + +@(private) +asking :: proc(x:^Exchange) { + connected:=connected_mask(x) + for k in 0..=LACK_RETRY {w.lack=0} + if connected&~(w.wrong|w.lack)==0 { + if w.state==.Asking { + w.state=.Missing + w.misses+=1 + x.dirty=true + } + } else if w.state==.Missing { + w.state=.Asking + } + } + for &p,i in x.peers { + if !p.used || p.bad {continue} + if p.batches>0 && !p.stalled && x.now-p.last_heard>STALL_SECONDS { + p.stalled=true + flights_release(x,i) + } + if !p.stalled && p.batches=0 || budget0 { + start:=next_free(w,at) + if start>=w.count {break} + end:=start+1 + for endx.desc.max_blob_bytes || len(data)!=MSG_TABLE_HEAD+store.piece_count(size)*size_of(store.Hash) {peer_bad(x,i);return} + wi:=want_find(x,address) + if wi<0 {return} + w:=&x.wants[wi] + if w.state!=.Asking && w.state!=.Missing || w.known || w.wrong&(u64(1)<store.PIECE_BYTES {peer_bad(x,i);return} + wi:=want_find(x,address) + if wi<0 {return} + w:=&x.wants[wi] + if w.state!=.Asking && w.state!=.Missing || !w.known || index>=w.count || bit_has(w.have,index) || w.wrong&(u64(1)<1 {peer_bad(x,i);return} + bit:=u64(1)<1 || count>NAMES_PER_REPLY {peer_bad(x,i);return} + bit:=u64(1)< bool + +Desc :: struct { + store:^store.Store, + channel:int, // index of the first of the CHANNELS channels in the channels of every host + peers:int, // connections at once, default 8, at most MAX_PEERS + wants:int, // blobs asked at once, default 256 + queries:int, // ref questions at once, default 16 + max_blob_bytes:i64, // the largest blob taken and served, default 1 GiB: a table of its size is held per peer + expose:Expose, // nil exposes no namespace + expose_user:rawptr, + allocator:runtime.Allocator, +} + +Error :: enum u8 { + None, + Invalid, // bad argument, or a host without the channels of the exchange + Full, // a capacity of the Desc is exhausted + Missing, // no connected peer has it + Too_Large, // a blob larger than max_blob_bytes + Store, // the store refused it: Progress.store_error + Out_Of_Memory, // only from init +} + +State :: enum u8 { + Asking, // being received + Complete, // in the store + Missing, // no connected peer has it now; asked again when a peer joins or after a while + Failed, // error +} + +Progress :: struct { + state:State, + here:i64, // bytes in the store + total:i64, // -1 until the table is known + error:Error, + store_error:store.Error, +} + +Peer_Progress :: struct { + receive_rate,send_rate:f64, // piece bytes per second, smoothed + received,sent:u64, // piece bytes + lies:int, // blobs it gave a table, a piece or a size for that did not match + refused:u64, // its messages refused + bad:bool, // it sent what no peer sends: it is neither asked nor served +} + +Query :: struct {index,generation:u32} + +Query_State :: enum u8 {Waiting,Found,Not_Found} + +Name :: struct {name:string,address:store.Hash} + +Query_Result :: struct { + state:Query_State, + address:store.Hash, // of a ref + names:[]Name, // of a namespace, sorted, valid until query_end + more:bool, // the namespace has names after the last: ask again with it as after +} + +@(private) +Want :: struct { + address:store.Hash, + used,wanted:bool, + waiters:int, + generation:u32, + priority:int, + order:u64, + state:State, + error:Error, + store_error:store.Error, + known:bool, + size:i64, + count,had:int, + table_peer:int, // the peer asked for the table, or that gave it; -1 + wrong,lack:u64, // peers that lied about it, and that lack it + lack_since:f64, + misses:u32, // times it went Missing + have,asked:[]u64, + here:i64, +} + +@(private) +Flight :: struct {want:int,generation:u32,first,count:u32,blob:bool} + +@(private) +Serve_Kind :: enum u8 {Blob,Run,Done} + +@(private) +Serve :: struct {kind:Serve_Kind,address:store.Hash,first,count:u32} + +@(private) +Peer :: struct { + used,bad,stalled:bool, + host:^wire.Host, + connection:wire.Connection, + // Asking: the entries of the requests in flight, oldest first. + flights:[]Flight, + flight_head,flight_count:int, + batch_sizes:[BATCHES]int, + batch_head,batches:int, + last_heard:f64, + queries:int, + // Serving: the entries of the requests received, oldest first. + serve:[]Serve, + serve_head,serve_count:int, + started:bool, + cursor:u32, + entry_size:i64, + lack_first,lack_count:u32, + out:[]u8, + out_len,out_sent:int, + control:[]u8, + control_start,control_end:int, + // Receiving, per lane: a message that arrives in pieces. + assembly:[CHANNELS][]u8, + assembled:[CHANNELS]int, + overflow:[CHANNELS]bool, + received,sent,received_mark,sent_mark:u64, + receive_rate,send_rate:f64, + lies:int, + refused:u64, +} + +@(private) +Query_Kind :: enum u8 {Ref,Names} + +@(private) +Query_Slot :: struct { + used:bool, + generation:u32, + kind:Query_Kind, + state:Query_State, + waiters:int, + namespace,name:[store.NAME_MAX]u8, // name: the name of a ref, or after of a listing + namespace_len,name_len:int, + peer:int, + asked,answered:u64, + address:store.Hash, + more:bool, + names:[]Name, + name_count:int, + name_bytes:[]u8, +} + +Exchange :: struct { + desc:Desc, + allocator:runtime.Allocator, + lock:sync.Mutex, + changed:sync.Cond, + wants:[]Want, + order:[]int, // wants in use, most important first + want_count:int, + sequence:u64, + peers:[]Peer, + queries:[]Query_Slot, + scratch:[]u8, // what the store hands back: a table, the missing pieces + table_max:int, // pieces of the largest blob + now,rate_time:f64, + dirty:bool, +} + +channels :: proc() -> [CHANNELS]wire.Channel_Desc { + return {{kind=.Reliable,queue_bytes=CONTROL_BYTES},{kind=.Reliable,queue_bytes=DATA_QUEUE_BYTES}} +} + +init :: proc(x:^Exchange,desc:Desc) -> Error { + x^={desc=desc} + d:=&x.desc + x.allocator=d.allocator.procedure==nil?context.allocator:d.allocator + if d.store==nil || d.channel<0 {return .Invalid} + if d.peers<=0 {d.peers=8} + if d.wants<=0 {d.wants=256} + if d.queries<=0 {d.queries=16} + if d.max_blob_bytes<=0 {d.max_blob_bytes=1<<30} + if d.peers>MAX_PEERS || d.max_blob_bytes>1<<40 {return .Invalid} + x.table_max=store.piece_count(d.max_blob_bytes) + words:=(x.table_max+63)/64 + failed:=false + make_into :: proc(x:^Exchange,slot:^$T/[]$E,count:int,failed:^bool) { + made,error:=make(T,count,x.allocator) + if error!=nil {failed^=true;return} + slot^=made + } + make_into(x,&x.wants,d.wants,&failed) + make_into(x,&x.order,d.wants,&failed) + make_into(x,&x.peers,d.peers,&failed) + make_into(x,&x.queries,d.queries,&failed) + make_into(x,&x.scratch,x.table_max*size_of(store.Hash)+64,&failed) + if failed {destroy(x);return .Out_Of_Memory} + for &w in x.wants { + make_into(x,&w.have,words,&failed) + make_into(x,&w.asked,words,&failed) + } + data_bytes:=max(MSG_TABLE_HEAD+x.table_max*size_of(store.Hash),MSG_PIECE_HEAD+store.PIECE_BYTES) + for &p in x.peers { + make_into(x,&p.flights,BATCHES*REQUEST_ENTRIES,&failed) + make_into(x,&p.serve,BATCHES*(REQUEST_ENTRIES+1),&failed) + make_into(x,&p.out,data_bytes,&failed) + make_into(x,&p.control,CONTROL_BYTES,&failed) + make_into(x,&p.assembly[LANE_CONTROL],CONTROL_BYTES,&failed) + make_into(x,&p.assembly[LANE_DATA],data_bytes,&failed) + } + for &q in x.queries { + make_into(x,&q.names,NAMES_PER_REPLY,&failed) + make_into(x,&q.name_bytes,NAMES_PER_REPLY*store.NAME_MAX,&failed) + } + if failed {destroy(x);return .Out_Of_Memory} + return .None +} + +destroy :: proc(x:^Exchange) { + for w in x.wants {delete(w.have,x.allocator);delete(w.asked,x.allocator)} + for p in x.peers { + delete(p.flights,x.allocator);delete(p.serve,x.allocator);delete(p.out,x.allocator);delete(p.control,x.allocator) + for a in p.assembly {delete(a,x.allocator)} + } + for q in x.queries {delete(q.names,x.allocator);delete(q.name_bytes,x.allocator)} + delete(x.wants,x.allocator);delete(x.order,x.allocator);delete(x.peers,x.allocator) + delete(x.queries,x.allocator);delete(x.scratch,x.allocator) + x^={} +} + +// Starts exchanging with a connection of h from its start, when its Connected +// command comes: a message on it that began before is not read from its middle. +// The host must carry the channels of the exchange at Desc.channel. +peer_add :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) -> Error { + sync.mutex_guard(&x.lock) + if h==nil || connection==0 || x.desc.channel+CHANNELS>h.channel_count {return .Invalid} + for lane in 0..=0 {return .None} + for &p,i in x.peers { + if p.used {continue} + peer_reset(&p) + p.used,p.host,p.connection,p.last_heard=true,h,connection,x.now + clear_peer_bits(x,i) + x.dirty=true + return .None + } + return .Full +} + +// Stops exchanging with a connection: what it gave stays, what it was asked goes to others. +peer_remove :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) { + sync.mutex_guard(&x.lock) + if i:=peer_find(x,h,connection);i>=0 {peer_drop(x,i)} + signal(x) +} + +peer_progress :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) -> (Peer_Progress,bool) { + sync.mutex_guard(&x.lock) + i:=peer_find(x,h,connection) + if i<0 {return {},false} + p:=&x.peers[i] + return {receive_rate=p.receive_rate,send_rate=p.send_rate,received=p.received,sent=p.sent,lies=p.lies,refused=p.refused,bad=p.bad},true +} + +@(private) +peer_find :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) -> int { + for p,i in x.peers { + if p.used && p.host==h && p.connection==connection {return i} + } + return -1 +} + +@(private) +peer_reset :: proc(p:^Peer) { + keep:=Peer{flights=p.flights,serve=p.serve,out=p.out,control=p.control,assembly=p.assembly} + p^=keep +} + +@(private) +peer_drop :: proc(x:^Exchange,i:int) { + p:=&x.peers[i] + flights_release(x,i) + for &w in x.wants { + if w.used && w.table_peer==i && !w.known {w.table_peer=-1} + } + clear_peer_bits(x,i) + peer_reset(p) + x.dirty=true +} + +@(private) +clear_peer_bits :: proc(x:^Exchange,i:int) { + mask:=~(u64(1)< bool { + if command.kind!=.Message {return false} + lane:=int(command.channel)-x.desc.channel + if lane<0 || lane>=CHANNELS {return false} + sync.mutex_guard(&x.lock) + i:=peer_find(x,h,command.connection) + if i<0 {return true} + p:=&x.peers[i] + if p.bad {return true} + if p.assembled[lane]==0 && !p.overflow[lane] && !command.more { + message(x,i,lane,command.data) + } else { + buffer:=p.assembly[lane] + if !p.overflow[lane] && p.assembled[lane]+len(command.data)<=len(buffer) { + copy(buffer[p.assembled[lane]:],command.data) + p.assembled[lane]+=len(command.data) + } else { + p.overflow[lane]=true + } + if !command.more { + whole:=buffer[:p.assembled[lane]] + if p.overflow[lane] {oversized(x,i,lane,whole)} else {message(x,i,lane,whole)} + p.assembled[lane],p.overflow[lane]=0,false + } + } + signal(x) + return true +} + +// Sends what is due: answers, pieces served and the next requests. now is in +// seconds and must not go back. +update :: proc(x:^Exchange,now:f64) { + sync.mutex_guard(&x.lock) + x.now=max(x.now,now) + for &p,i in x.peers { + if !p.used {continue} + control_flush(x,i) + if !p.bad {serve_pump(x,i)} + } + asking(x) + queries_send(x) + rates(x) + signal(x) +} + +@(private) +rates :: proc(x:^Exchange) { + dt:=x.now-x.rate_time + if dt<0.1 {return} + x.rate_time=x.now + a:=min(dt/RATE_SECONDS,1) + for &p in x.peers { + if !p.used {continue} + p.receive_rate+=a*(f64(p.received-p.received_mark)/dt-p.receive_rate) + p.send_rate+=a*(f64(p.sent-p.sent_mark)/dt-p.send_rate) + p.received_mark,p.sent_mark=p.received,p.sent + } +} + +@(private) +signal :: proc(x:^Exchange) { + if !x.dirty {return} + x.dirty=false + sync.cond_broadcast(&x.changed) +} + +// Wants a blob: asks connected peers for it, most important first. Asking +// again raises its priority. A want stays until forget, also once complete. +want :: proc(x:^Exchange,address:store.Hash,priority:=0) -> Error { + sync.mutex_guard(&x.lock) + i,error:=want_add(x,address,priority) + if error==.None {x.wants[i].wanted=true} + return error +} + +forget :: proc(x:^Exchange,address:store.Hash) { + sync.mutex_guard(&x.lock) + if i:=want_find(x,address);i>=0 { + x.wants[i].wanted=false + if x.wants[i].waiters==0 {want_release(x,i)} + } +} + +progress :: proc(x:^Exchange,address:store.Hash) -> (Progress,bool) { + sync.mutex_guard(&x.lock) + i:=want_find(x,address) + if i<0 {return {},false} + w:=&x.wants[i] + return {state=w.state,here=w.here,total=w.known?w.size:-1,error=w.error,store_error=w.store_error},true +} + +// Returns once the blob is complete in the store, or once no connected peer +// has it (Missing). For job threads: the owner keeps calling update meanwhile. +fetch :: proc(x:^Exchange,address:store.Hash,priority:=0) -> Error { + sync.mutex_guard(&x.lock) + return fetch_locked(x,address,priority) +} + +// Returns the address peers name with namespace and name once its blob is +// complete in the store, or Missing once no connected peer names it or has it. +fetch_ref :: proc(x:^Exchange,namespace,name:string,priority:=0) -> (store.Hash,Error) { + sync.mutex_guard(&x.lock) + q,error:=query_add(x,.Ref,namespace,name,-1) + if error!=.None {return {},error} + slot:=&x.queries[q.index] + slot.waiters+=1 + for slot.state==.Waiting {sync.cond_wait(&x.changed,&x.lock)} + slot.waiters-=1 + found,address:=slot.state==.Found,slot.address + slot.used=false + if !found {return {},.Missing} + return address,fetch_locked(x,address,priority) +} + +@(private) +fetch_locked :: proc(x:^Exchange,address:store.Hash,priority:int) -> Error { + i,error:=want_add(x,address,priority) + if error!=.None {return error} + w:=&x.wants[i] + w.waiters+=1 + misses:=w.misses + result:Error + for { + if w.state==.Complete {break} + if w.state==.Failed {result=w.error;break} + if w.state==.Missing || w.misses!=misses {result=.Missing;break} + sync.cond_wait(&x.changed,&x.lock) + } + w.waiters-=1 + if !w.wanted && w.waiters==0 {want_release(x,i)} + return result +} + +// Asks every connected peer for the address namespace and name name. +ref_ask :: proc(x:^Exchange,namespace,name:string) -> (Query,Error) { + sync.mutex_guard(&x.lock) + return query_add(x,.Ref,namespace,name,-1) +} + +// Asks a peer for the names of a namespace after after, at most NAMES_PER_REPLY. +names_ask :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection,namespace:string,after:="") -> (Query,Error) { + sync.mutex_guard(&x.lock) + i:=peer_find(x,h,connection) + if i<0 {return {},.Invalid} + return query_add(x,.Names,namespace,after,i) +} + +query_result :: proc(x:^Exchange,q:Query) -> (Query_Result,bool) { + sync.mutex_guard(&x.lock) + slot:=query_slot(x,q) + if slot==nil {return {},false} + return {state=slot.state,address=slot.address,names=slot.names[:slot.name_count],more=slot.more},true +} + +query_end :: proc(x:^Exchange,q:Query) { + sync.mutex_guard(&x.lock) + if slot:=query_slot(x,q);slot!=nil && slot.waiters==0 {slot.used=false} +} + +@(private) +query_slot :: proc(x:^Exchange,q:Query) -> ^Query_Slot { + if int(q.index)>=len(x.queries) {return nil} + slot:=&x.queries[q.index] + if !slot.used || slot.generation!=q.generation {return nil} + return slot +} + +@(private) +query_add :: proc(x:^Exchange,kind:Query_Kind,namespace,name:string,peer:int) -> (Query,Error) { + if !store.name_valid(namespace) || !(store.name_valid(name) || kind==.Names && name=="") {return {},.Invalid} + for &slot,i in x.queries { + if slot.used {continue} + generation:=slot.generation+1 + names,bytes:=slot.names,slot.name_bytes + slot={used=true,generation=generation,kind=kind,peer=peer,names=names,name_bytes=bytes} + copy(slot.namespace[:],namespace) + copy(slot.name[:],name) + slot.namespace_len,slot.name_len=len(namespace),len(name) + x.dirty=true + return {u32(i),generation},.None + } + return {},.Full +} + +@(private) +connected_mask :: proc(x:^Exchange) -> (mask:u64) { + for p,i in x.peers { + if p.used && !p.bad {mask|=1<=QUERIES_PER_PEER {continue} + buffer:[MSG_NAMES_GET_MAX]u8 + message:=encode_question(buffer[:],q.kind==.Ref?MSG_REF_GET:MSG_NAMES_GET,namespace,name) + if !control_append(&p,message) {continue} + q.asked|=bit + p.queries+=1 + } + if targets&~q.answered==0 { + q.state=.Not_Found + x.dirty=true + } + } +} + +// Queues a message on the control channel of a peer; false when it has no room. +@(private) +control_append :: proc(p:^Peer,message:[]u8) -> bool { + need:=2+len(message) + if p.control_end+need>len(p.control) && p.control_start>0 { + copy(p.control,p.control[p.control_start:p.control_end]) + p.control_end-=p.control_start + p.control_start=0 + } + if p.control_end+need>len(p.control) {return false} + put_u16(p.control[p.control_end:],u16(len(message))) + copy(p.control[p.control_end+2:],message) + p.control_end+=need + return true +} + +@(private) +control_flush :: proc(x:^Exchange,i:int) { + p:=&x.peers[i] + for p.control_start runtime.Allocator { + mem.arena_init(arena,x.scratch) + return mem.arena_allocator(arena) +} diff --git a/exchange/exchange_test.odin b/exchange/exchange_test.odin new file mode 100644 index 000000000..29d3c4343 --- /dev/null +++ b/exchange/exchange_test.odin @@ -0,0 +1,326 @@ +package exchange + +import "core:fmt" +import "core:sync" +import "core:testing" +import "core:thread" +import "mirai:store" +import wire "mirai:net" + +// A server node 0 and a client node 1 connected; false if they are not. +@(private) +test_pair :: proc(t:^testing.T,w:^Test_World,name:string,game:=false) -> bool { + if !test_world(t,w,name,2,game) {return false} + server:=test_host(w,0,.Server) + client:=test_host(w,1,.Client) + return testing.expect(t,server!=nil && client!=nil && test_connect(w,client,w.nodes[0].endpoints[0]),"the peers connect") +} + +@(private) +test_put :: proc(t:^testing.T,s:^store.Store,size:int,seed:u64) -> store.Hash { + hash,error:=store.put_bytes(s,test_bytes(size,seed)) + testing.expect_value(t,error,store.Error.None) + return hash +} + +@(test) +blobs_around_the_piece_boundaries_and_a_large_one_arrive_verified_under_loss :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-loss") {return} + P:=store.PIECE_BYTES + sizes:=[]int{0,1,P-1,P,P+1,2*P,3*P+17,6<<20} + addresses:=make([]store.Hash,len(sizes),context.temp_allocator) + for size,k in sizes {addresses[k]=test_put(t,&w.nodes[0].s,size,u64(k)+1)} + for e in w.nodes[0].endpoints[:1] {wire.sim_endpoint_conditions(&w.sim,e,{latency=0.02,loss=0.05})} + wire.sim_endpoint_conditions(&w.sim,w.nodes[1].endpoints[0],{latency=0.02,loss=0.05}) + for a in addresses {testing.expect_value(t,want(&w.nodes[1].x,a),Error.None)} + testing.expect(t,test_until_complete(&w,1,addresses,60),"every blob arrives") + for a,k in addresses { + testing.expect_value(t,store.verify(&w.nodes[1].s,a),store.Error.None) + back:=make([]u8,sizes[k],context.temp_allocator) + testing.expect_value(t,store.read(&w.nodes[1].s,a,0,back),store.Error.None) + testing.expect(t,string(back)==string(test_bytes(sizes[k],u64(k)+1)),"the blob reads back its bytes") + p,_:=progress(&w.nodes[1].x,a) + testing.expect(t,p.state==.Complete && p.here==i64(sizes[k]) && p.total==i64(sizes[k]),"progress counts the whole blob") + } + rate,_:=peer_progress(&w.nodes[1].x,w.nodes[1].hosts[0],w.nodes[1].hosts[0].connection_handles[0]) + testing.expect(t,rate.received>=6<<20 && rate.receive_rate>0,"the peer's rate is measured") +} + +// What matters most is asked first: a small want asked after a large one and +// raised above it completes long before it. +@(test) +a_high_priority_want_asked_later_completes_before_a_large_low_one :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-priority") {return} + large:=test_put(t,&w.nodes[0].s,24<<20,1) + small:=test_put(t,&w.nodes[0].s,300<<10,2) + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=0.01,bandwidth=4e6}) + x:=&w.nodes[1].x + want(x,large) + test_step(&w,300) + want(x,small) + want(x,small,10) + small_done,large_done:f64 + for w.now<60 && large_done==0 { + test_step(&w) + if small_done==0 && test_state(x,small)==.Complete {small_done=w.now} + if test_state(x,large)==.Complete {large_done=w.now} + } + testing.expectf(t,small_done>0 && large_done>0,"both arrive: %.2f and %.2f s",small_done,large_done) + testing.expectf(t,small_done0 && before.here=before.here,"with no peer the want is missing and keeps what arrived") + testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0]),"the peers connect again") + testing.expect(t,test_until_complete(&w,1,{blob},30),"the blob completes after the connection returns") + testing.expect_value(t,store.verify(&w.nodes[1].s,blob),store.Error.None) + + second:=test_put(t,&w.nodes[0].s,8<<20,4) + want(x,second) + test_step(&w,600) + part,_:=progress(x,second) + testing.expect(t,part.here>0 && part.here=part.here,"the reopened store resumes from the pieces it has") + testing.expect(t,test_until_complete(&w,1,{second},30),"the second blob completes") + got,_:=peer_progress(&n.x,client,client.connection_handles[0]) + testing.expectf(t,got.received<=u64(8<<20)-u64(part.here)+u64(4*store.PIECE_BYTES),"after the reopen %d bytes arrive of the %d missing",got.received,(8<<20)-part.here) +} + +// Two sources, one of which lies; it is asked first, so its lie is met. +@(test) +a_liar_in_the_table_a_piece_or_the_size_is_marked_and_the_blob_still_arrives :: proc(t:^testing.T) { + for lie in ([]Test_Lie{.Table,.Piece,.Size}) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-liar",3) {return} + test_host(&w,0,.Server) + liar:=test_host(&w,1,.Client) + honest:=test_host(&w,2,.Client) + w.nodes[1].script,w.nodes[1].lie,w.nodes[1].source=test_liar,lie,&w.nodes[1].s + size:=5*store.PIECE_BYTES+1000 + blob:=test_put(t,&w.nodes[1].s,size,9) + test_put(t,&w.nodes[2].s,size,9) + if !testing.expect(t,test_connect(&w,liar,w.nodes[0].endpoints[0]) && test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + r:=&w.nodes[0] + want(&r.x,blob) + testing.expectf(t,test_until_complete(&w,0,{blob},20),"%v: the blob arrives",lie) + testing.expect_value(t,store.verify(&r.s,blob),store.Error.None) + server:=r.hosts[0] + lies:=[2]int{} + for c in server.connection_handles { + if c==0 {continue} + p,_:=peer_progress(&r.x,server,c) + lies[0 if p.lies>0 else 1]+=1 + } + testing.expectf(t,lies=={1,1},"%v: the liar is marked and the honest peer is not: %v",lie,lies) + } +} + +// A peer serves the pieces it has of a blob it is still receiving. +@(test) +a_peer_still_receiving_a_blob_serves_its_pieces_to_a_third :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-relay",3) {return} + test_host(&w,0,.Server) + b_client:=test_host(&w,1,.Client) + test_host(&w,1,.Server) + c_client:=test_host(&w,2,.Client) + if !testing.expect(t,test_connect(&w,b_client,w.nodes[0].endpoints[0]) && test_connect(&w,c_client,w.nodes[1].endpoints[1])) {return} + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=0.01,bandwidth=2e6}) + blob:=test_put(t,&w.nodes[0].s,6<<20,5) + want(&w.nodes[1].x,blob) + want(&w.nodes[2].x,blob) + relayed:=false + for w.now<60 && !store.has(&w.nodes[2].s,blob) { + test_step(&w) + c,_:=progress(&w.nodes[2].x,blob) + if c.here>0 && !store.has(&w.nodes[1].s,blob) {relayed=true} + } + testing.expect(t,relayed,"the third peer receives pieces before the relay has the whole blob") + testing.expect(t,store.has(&w.nodes[2].s,blob) && store.verify(&w.nodes[2].s,blob)==.None,"the third peer completes the blob") +} + +@(private) +test_expose_lib :: proc(user:rawptr,namespace:string) -> bool {return namespace=="lib"} + +@(test) +refs_are_asked_and_listed_and_a_namespace_not_exposed_gives_nothing :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-refs",2,desc={expose=test_expose_lib}) {return} + test_host(&w,0,.Server) + client:=test_host(&w,1,.Client) + if !testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0])) {return} + s:=&w.nodes[0].s + blob:=test_put(t,s,1000,1) + names:=make([]string,NAMES_PER_REPLY+5,context.temp_allocator) + for &name,k in names { + buffer:=make([]u8,4,context.temp_allocator) + buffer[0],buffer[1],buffer[2],buffer[3]='p',u8('a'+k/26/26%26),u8('a'+k/26%26),u8('a'+k%26) + name=string(buffer) + store.ref_set(s,"lib",name,blob) + } + store.ref_set(s,"secret","key",blob) + x:=&w.nodes[1].x + ask :: proc(w:^Test_World,x:^Exchange,q:Query) -> Query_Result { + for _ in 0..<2000 { + test_step(w) + if r,_:=query_result(x,q);r.state!=.Waiting {return r} + } + return {} + } + q,_:=ref_ask(x,"lib","pabc") + r:=ask(&w,x,q) + testing.expect(t,r.state==.Found && r.address==blob,"an exposed ref gives its address") + query_end(x,q) + q,_=ref_ask(x,"secret","key") + r=ask(&w,x,q) + testing.expect_value(t,r.state,Query_State.Not_Found) + query_end(x,q) + q,_=ref_ask(x,"lib","nothing") + testing.expect_value(t,ask(&w,x,q).state,Query_State.Not_Found) + query_end(x,q) + q,_=names_ask(x,client,client.connection_handles[0],"lib") + r=ask(&w,x,q) + testing.expect(t,r.state==.Found && len(r.names)==NAMES_PER_REPLY && r.more,"the first page of names") + last:="" + for n,k in r.names { + testing.expect(t,n.name==names[k] && n.address==blob,"names come sorted with their addresses") + last=n.name + } + after:=make([]u8,len(last),context.temp_allocator) + copy(after,last) + query_end(x,q) + q,_=names_ask(x,client,client.connection_handles[0],"lib",string(after)) + r=ask(&w,x,q) + testing.expect(t,r.state==.Found && len(r.names)==5 && !r.more && r.names[0].name==names[NAMES_PER_REPLY],"the page after the last name") + query_end(x,q) + q,_=names_ask(x,client,client.connection_handles[0],"secret") + r=ask(&w,x,q) + testing.expect(t,r.state==.Found && len(r.names)==0,"a namespace not exposed lists nothing") + query_end(x,q) +} + +@(private) +Test_Job :: struct {x:^Exchange,address:store.Hash,namespace,name:string,error:Error,found:store.Hash,done:bool} + +@(private) +test_job :: proc(j:^Test_Job) { + if j.namespace!="" { + j.found,j.error=fetch_ref(j.x,j.namespace,j.name) + } else { + j.error=fetch(j.x,j.address) + } + sync.atomic_store(&j.done,true) +} + +// Runs a fetch on a thread while the owner pumps; false if it does not return within seconds. +@(private) +test_fetch :: proc(w:^Test_World,j:^Test_Job,seconds:f64,during:proc(w:^Test_World)=nil) -> bool { + th:=thread.create_and_start_with_poly_data(j,test_job) + defer {thread.join(th);thread.destroy(th)} + end:=w.now+seconds + for w.now1<<20 {wire.host_disconnect(w.nodes[0].hosts[0])} + }) && j.error==.Missing,"a disconnect returns Missing") + testing.expect(t,!store.has(&w.nodes[1].s,big),"the disconnect came before the blob") +} + +// A 60 Hz unreliable message from the source keeps its delay small while blobs stream. +@(test) +a_game_channel_keeps_its_delay_small_during_a_transfer :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_pair(t,&w,"exchange-game",game=true) {return} + latency:=0.02 + wire.sim_endpoint_conditions(&w.sim,w.nodes[0].endpoints[0],{latency=latency,bandwidth=12.5e6}) + wire.sim_endpoint_conditions(&w.sim,w.nodes[1].endpoints[0],{latency=latency}) + blobs:=make([]store.Hash,6,context.temp_allocator) + for &b,k in blobs {b=test_put(t,&w.nodes[0].s,8<<20,u64(k)+1)} + for b in blobs {want(&w.nodes[1].x,b)} + g:=&w.nodes[1].game + g.latency,g.since=latency,w.now+1 + w.nodes[0].game.since=g.since + complete:=test_until_complete(&w,1,blobs,30) + testing.expect(t,complete,"the blobs arrive") + sent:=w.nodes[0].game.sent + testing.expectf(t,g.got>=sent*9/10 && sent>100,"%d of %d game messages arrive",g.got,sent) + mean:=g.mean/f64(max(g.got,1)) + testing.expectf(t,mean<0.03 && g.worst<0.08,"game messages take %.0f ms more than the latency on average, %.0f at worst",mean*1000,g.worst*1000) +} diff --git a/exchange/harness_test.odin b/exchange/harness_test.odin new file mode 100644 index 000000000..efc40a090 --- /dev/null +++ b/exchange/harness_test.odin @@ -0,0 +1,244 @@ +package exchange + +import "core:fmt" +import "core:testing" +import "mirai:std/base" +import "mirai:store" +import wire "mirai:net" + +// Peers of one simulated network, each with its store and exchange and up to +// two hosts, stepped together at the worker's cadence of 1 ms. +@(private) +Test_Node :: struct { + s:store.Store, + x:Exchange, + open:bool, // the exchange runs: a scripted peer has none + hosts:[2]^wire.Host, + endpoints:[2]int, + host_count:int, + script:proc(w:^Test_World,n:^Test_Node,h:^wire.Host,command:wire.Command), // a peer that answers by hand + lie:Test_Lie, + source:^store.Store, // what the scripted peer serves + game:Test_Game, +} + +@(private) +Test_Lie :: enum {None,Table,Piece,Size} + +// A 60 Hz unreliable message from the server to the client on channel 0. +@(private) +Test_Game :: struct {sent,got:int,mean,worst,latency,since:f64} + +@(private) +Test_World :: struct { + sim:wire.Sim_Network, + now:f64, + nodes:[4]Test_Node, + count,endpoints:int, + channels:[]wire.Channel_Desc, + channel:int, + folder:string, +} + +@(private) TEST_STEP :: 0.001 + +@(private) +test_world :: proc(t:^testing.T,w:^Test_World,name:string,nodes:int,game:=false,desc:=Desc{}) -> bool { + w.folder=base.test_folder(t,name) + if w.folder=="" {return false} + wire.sim_network_init(&w.sim,{seed=7,endpoints=8,packets=1<<16}) + ours:=channels() + if game { + w.channels=make([]wire.Channel_Desc,3,context.temp_allocator) + w.channels[0]={kind=.Unreliable} + w.channels[1],w.channels[2]=ours[0],ours[1] + w.channel=1 + } else { + w.channels=make([]wire.Channel_Desc,2,context.temp_allocator) + w.channels[0],w.channels[1]=ours[0],ours[1] + } + w.count=nodes + for k in 0.. bool { + n:=&w.nodes[k] + error:store.Error + n.s,error=store.open(fmt.tprintf("%s/node-%d",w.folder,k)) + if error!=.None {return false} + d:=desc + d.store,d.channel=&n.s,w.channel + if d.max_blob_bytes==0 {d.max_blob_bytes=64<<20} + if d.peers==0 {d.peers=4} + n.open=init(&n.x,d)==.None + return n.open +} + +@(private) +test_node_close :: proc(n:^Test_Node) { + if n.open {destroy(&n.x)} + store.close(&n.s) + n.open=false +} + +@(private) +test_world_destroy :: proc(w:^Test_World) { + for &n in w.nodes[:w.count] { + test_node_close(&n) + for h in n.hosts[:n.host_count] {wire.host_destroy(h);free(h)} + } + wire.sim_network_destroy(&w.sim) +} + +@(private) +test_host :: proc(w:^Test_World,k:int,mode:wire.Mode) -> ^wire.Host { + n:=&w.nodes[k] + h:=new(wire.Host) + endpoint:=w.endpoints + w.endpoints+=1 + if wire.host_init(h,{mode=mode,transport=wire.sim_endpoint_transport(&w.sim,endpoint),channels=w.channels,protocol_id=91,seed=u64(100+endpoint)})!=.None { + free(h) + return nil + } + n.hosts[n.host_count]=h + n.endpoints[n.host_count]=endpoint + n.host_count+=1 + return h +} + +// Connects the client host of node k to the server host at endpoint; false if it does not within 5 s. +@(private) +test_connect :: proc(w:^Test_World,client:^wire.Host,endpoint:int) -> bool { + wire.host_connect(client,wire.address_memory(endpoint)) + for _ in 0..<5000 { + test_step(w) + if stats,ok:=wire.connection_stats(client,client.connection_handles[0]);ok && client.connection_handles[0]!=0 { + _=stats + return true + } + } + return false +} + +@(private) +test_step :: proc(w:^Test_World,steps:=1) { + for _ in 0..=n.game.since {n.game.sent+=1} + } +} + +@(private) +test_bytes :: proc(size:int,seed:u64) -> []u8 { + data:=make([]u8,size,context.temp_allocator) + h:=base.Hostile{state=seed} + for &b in data {b=u8(base.hostile_next(&h))} + return data +} + +// Steps until every address is complete at node k or seconds pass. +@(private) +test_until_complete :: proc(w:^Test_World,k:int,addresses:[]store.Hash,seconds:f64) -> bool { + end:=w.now+seconds + for w.now State { + p,_:=progress(x,address) + return p.state +} + +// A scripted peer that answers requests from n.source and lies as n.lie says. +@(private) +test_liar :: proc(w:^Test_World,n:^Test_Node,h:^wire.Host,command:wire.Command) { + if command.kind!=.Message || int(command.channel)!=w.channel+LANE_CONTROL || len(command.data) []u8 { + n:=0 + for p in parts {n+=len(p)} + out:=make([]u8,n,context.temp_allocator) + at:=0 + for p in parts {copy(out[at:],p);at+=len(p)} + return out +} + +@(private) +test_le :: proc(v:u64,bytes:int) -> []u8 { + out:=make([]u8,bytes,context.temp_allocator) + for i in 0..>(8*uint(i)))} + return out +} + +@(private) +test_name :: proc(s:string) -> []u8 {return test_message({u8(len(s))},transmute([]u8)s)} + +@(private) +test_entry :: proc(kind:u8,address:store.Hash,first,count:u32) -> []u8 { + address:=address + return test_message({kind},address[:],test_le(u64(first),4),test_le(u64(count),4)) +} + +@(private) +test_request :: proc(entries:..[]u8) -> []u8 { + return test_message({MSG_REQUEST},test_le(u64(len(entries)),2),test_message(..entries)) +} + +// Every message a peer could send is validated: the malformed ones mark the +// peer bad, the well formed ones nobody asked for are ignored, and a peer +// cannot queue more than BATCHES requests. None crashes, and random bytes on +// both channels do not either. +@(test) +hostile_messages_are_refused_without_a_crash :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-hostile",2,desc={expose=test_expose_lib}) {return} + server:=test_host(&w,0,.Server) + client:=test_host(&w,1,.Client) + if !testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0])) {return} + x:=&w.nodes[0].x + c:=server.connection_handles[0] + if c==0 {c=server.connection_handles[1]} + blob:=test_put(t,&w.nodes[0].s,3*store.PIECE_BYTES+5,1) + other:=store.address_of_bytes(test_bytes(5,2)) + want(x,other) + zero:store.Hash + big:=make([]u8,store.PIECE_BYTES+1,context.temp_allocator) + Case :: struct {lane:int,data:[]u8,bad:bool} + cases:=[]Case{ + {LANE_CONTROL,nil,true}, + {LANE_CONTROL,{99},true}, + {LANE_DATA,{99},true}, + {LANE_CONTROL,{MSG_REQUEST},true}, + {LANE_CONTROL,test_request(),true}, + {LANE_CONTROL,test_message({MSG_REQUEST},test_le(65,2)),true}, + {LANE_CONTROL,test_message(test_request(test_entry(1,blob,0,1)),{0}),true}, + {LANE_CONTROL,test_request(test_entry(2,blob,0,1)),true}, + {LANE_CONTROL,test_request(test_entry(1,blob,0,0)),true}, + {LANE_CONTROL,test_request(test_entry(1,blob,max(u32),2)),true}, + {LANE_CONTROL,test_request(test_entry(1,blob,0,REQUEST_PIECES+1)),true}, + {LANE_CONTROL,test_request(test_entry(1,blob,0,REQUEST_PIECES),test_entry(1,blob,0,1)),true}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("lib")),true}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("Lib"),test_name("a")),true}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("lib"),{200},big[:10]),true}, + {LANE_CONTROL,test_message({MSG_NAMES_GET},test_name("lib"),test_name("../x")),true}, + {LANE_CONTROL,test_message({MSG_REF},test_name("lib"),test_name("a"),{2},zero[:]),true}, + {LANE_CONTROL,test_message({MSG_NAMES},test_name("lib"),{0},{0},test_le(3,2)),true}, + {LANE_CONTROL,test_message({MSG_NAMES},test_name("lib"),{0},{0},test_le(1,2),test_name("a")),true}, + {LANE_DATA,test_message({MSG_TABLE},blob[:],test_le(1<<62,8)),true}, + {LANE_DATA,test_message({MSG_TABLE},blob[:],test_le(2*store.PIECE_BYTES,8),zero[:]),true}, + {LANE_DATA,test_message({MSG_TABLE},blob[:]),true}, + {LANE_DATA,test_message({MSG_PIECE},blob[:],test_le(0,4)),true}, + {LANE_DATA,test_message({MSG_PIECE},blob[:],test_le(0,4),big),true}, + {LANE_DATA,test_message({MSG_PIECE},blob[:2]),true}, + {LANE_DATA,test_message({MSG_LACK},blob[:],test_le(0,4)),true}, + {LANE_DATA,{MSG_DONE},true}, + {LANE_DATA,{MSG_DONE,0},true}, + // Well formed, but nobody asked: ignored. + {LANE_CONTROL,test_message({MSG_REF},test_name("lib"),test_name("a"),{1},zero[:]),false}, + {LANE_CONTROL,test_message({MSG_NAMES},test_name("lib"),{0},{0},test_le(0,2)),false}, + {LANE_DATA,test_message({MSG_PIECE},blob[:],test_le(9,4),big[:100]),false}, + {LANE_DATA,test_message({MSG_PIECE},other[:],test_le(1<<31,4),big[:100]),false}, + {LANE_DATA,test_message({MSG_TABLE},zero[:],test_le(10,8),zero[:]),false}, + {LANE_DATA,test_message({MSG_LACK},zero[:],test_le(0,4),test_le(0,4)),false}, + {LANE_CONTROL,test_message({MSG_REF_GET},test_name("secret"),test_name("a")),false}, + {LANE_CONTROL,test_message({MSG_NAMES_GET},test_name("lib"),{0}),false}, + {LANE_CONTROL,test_request(test_entry(0,blob,0,2),test_entry(1,zero,0,3)),false}, + } + for c_,k in cases { + receive(x,server,{kind=.Message,channel=u8(w.channel+c_.lane),connection=c,data=c_.data}) + p,_:=peer_progress(x,server,c) + testing.expectf(t,p.bad==c_.bad,"case %d: bad %v",k,p.bad) + if p.bad { + peer_remove(x,server,c) + peer_add(x,server,c) + } + } + // A message that arrives in pieces past what this end holds. + piece:=make([]u8,wire.PIECE_BYTES,context.temp_allocator) + piece[0]=MSG_NAMES + for _ in 0..<8 {receive(x,server,{kind=.Message,channel=u8(w.channel+LANE_CONTROL),connection=c,data=piece,more=true})} + receive(x,server,{kind=.Message,channel=u8(w.channel+LANE_CONTROL),connection=c,data=piece[:1]}) + p,_:=peer_progress(x,server,c) + testing.expect(t,p.bad,"an oversized message marks the peer") + peer_remove(x,server,c) + peer_add(x,server,c) + // At most BATCHES requests queue; the next is refused, not served. + full:=make([][]u8,REQUEST_ENTRIES,context.temp_allocator) + for &e in full {e=test_entry(1,blob,0,1)} + request:=test_request(..full) + for _ in 0..0 && base.hostile_int(&h,2)==0 {data[0]=u8(1+base.hostile_int(&h,5))} + receive(x,server,{kind=.Message,channel=u8(w.channel+base.hostile_int(&h,CHANNELS)),connection=c,data=data[:n],more=base.hostile_int(&h,8)==0}) + if p,_=peer_progress(x,server,c);p.bad { + peer_remove(x,server,c) + peer_add(x,server,c) + } + } + test_step(&w,500) + testing.expect(t,!store.has(&w.nodes[0].s,other),"nothing unverified completes a blob") +} diff --git a/exchange/serve.odin b/exchange/serve.odin new file mode 100644 index 000000000..cf20b76e7 --- /dev/null +++ b/exchange/serve.odin @@ -0,0 +1,204 @@ +package exchange + +import "core:mem" +import "core:slice" +import "mirai:store" +import wire "mirai:net" + +// A request is taken whole or refused whole: a peer keeps at most BATCHES in +// flight, so one with more is not one this end needs to answer. +@(private) +on_request :: proc(x:^Exchange,i:int,data:[]u8) { + p:=&x.peers[i] + r:=Reader{data=data[1:],ok=true} + count:=int(read_u16(&r)) + if count==0 || count>REQUEST_ENTRIES || len(data)!=MSG_REQUEST_HEAD+count*ENTRY_BYTES {peer_bad(x,i);return} + pieces:=0 + for _ in 0..1 || kind==1 && n==0 || u64(first)+u64(n)>u64(x.table_max) || n>REQUEST_PIECES {peer_bad(x,i);return} + pieces+=int(n) + } + if pieces>REQUEST_PIECES {peer_bad(x,i);return} + if p.serve_count+count+1>len(p.serve) {p.refused+=1;return} + r=Reader{data=data[MSG_REQUEST_HEAD:],ok=true} + for _ in 0.. bool { + return x.desc.expose!=nil && x.desc.expose(x.desc.expose_user,namespace) +} + +// An answer that finds no room is dropped: a peer keeps at most +// QUERIES_PER_PEER questions waiting, which always fit. +@(private) +on_ref_get :: proc(x:^Exchange,i:int,data:[]u8) { + r:=Reader{data=data[1:],ok=true} + namespace:=read_name(&r) + name:=read_name(&r) + if !read_end(&r) {peer_bad(x,i);return} + address:store.Hash + found:=false + if exposed(x,namespace) { + hash,error:=store.ref_get(x.desc.store,namespace,name) + address,found=hash,error==.None + } + buffer:[MSG_REF_MAX]u8 + w:=Writer{data=buffer[:]} + write_u8(&w,MSG_REF) + write_name(&w,namespace) + write_name(&w,name) + write_u8(&w,found?1:0) + write_bytes(&w,address[:]) + if !control_append(&x.peers[i],buffer[:w.n]) {x.peers[i].refused+=1} +} + +// The names after after, sorted. Reading a namespace is the one answer the +// store allocates for: it reads a folder. +@(private) +on_names_get :: proc(x:^Exchange,i:int,data:[]u8) { + r:=Reader{data=data[1:],ok=true} + namespace:=read_name(&r) + after:=read_name(&r,empty=true) + if !read_end(&r) {peer_bad(x,i);return} + refs:[]store.Ref + if exposed(x,namespace) { + listed,error:=store.ref_list(x.desc.store,namespace,x.allocator) + if error==.None {refs=listed} + } + defer store.ref_list_delete(refs,x.allocator) + slice.sort_by(refs,proc(a,b:store.Ref) -> bool {return a.nameNAMES_REPLY_BYTES { + buffer[more_at]=1 + break + } + write_name(&w,ref.name) + write_bytes(&w,ref.hash[:]) + count+=1 + } + put_u16(buffer[count_at:],u16(count)) + if !control_append(&x.peers[i],buffer[:w.n]) {x.peers[i].refused+=1} +} + +// Streams the answers of peer i on its data channel while its queue has room. +@(private) +serve_pump :: proc(x:^Exchange,i:int) { + p:=&x.peers[i] + for { + if p.out_len>0 { + wire.message_send(p.host,p.connection,x.desc.channel+LANE_DATA,p.out[:p.out_len],&p.out_sent) + if p.out_sent bool { + p:=&x.peers[i] + s:=x.desc.store + for p.serve_count>0 { + e:=p.serve[p.serve_head] + if e.kind==.Done { + p.out[0]=MSG_DONE + p.out_len=1 + serve_pop(p) + return true + } + if !p.started { + p.started,p.cursor,p.lack_count=true,e.first,0 + size,error:=store.blob_size(s,e.address) + if error!=.None || size>x.desc.max_blob_bytes { + p.out_len=encode_lack(p.out,e.address,e.first,e.kind==.Blob?0:e.count) + serve_pop(p) + return true + } + p.entry_size=size + if e.kind==.Blob { + arena:mem.Arena + pieces,table_error:=store.blob_table(s,e.address,scratch_arena(x,&arena)) + if table_error!=.None || len(pieces)!=store.piece_count(size) { + p.out_len=encode_lack(p.out,e.address,0,0) + serve_pop(p) + return true + } + address:=e.address + w:=Writer{data=p.out} + write_u8(&w,MSG_TABLE) + write_bytes(&w,address[:]) + write_u64(&w,u64(size)) + write_bytes(&w,slice.to_bytes(pieces)) + p.out_len=w.n + return true + } + } + end:=u32(min(i64(e.first)+i64(e.count),i64(store.piece_count(p.entry_size)))) + for p.cursor0 { + // The piece is read again once the lack before it is out. + p.out_len=encode_lack(p.out,e.address,p.lack_first,p.lack_count) + p.lack_count=0 + return true + } + encode_piece_head(p.out,e.address,index) + p.out_len=MSG_PIECE_HEAD+bytes + p.cursor+=1 + p.sent+=u64(bytes) + return true + } + if p.lack_count==0 {p.lack_first=index} + p.lack_count+=1 + p.cursor+=1 + } + if p.lack_count>0 { + p.out_len=encode_lack(p.out,e.address,p.lack_first,p.lack_count) + p.lack_count=0 + return true + } + serve_pop(p) + } + return false +} diff --git a/exchange/wire.odin b/exchange/wire.odin new file mode 100644 index 000000000..2f51d4ad3 --- /dev/null +++ b/exchange/wire.odin @@ -0,0 +1,188 @@ +package exchange + +import "mirai:store" + +// Messages, little endian, a kind byte first. Names are a length byte and +// their bytes. +// +// Control channel: +// Request count u16, entries of kind u8 (0 blob, 1 run), address, first u32, count u32 +// Ref_Get namespace, name +// Ref namespace, name, found u8, address +// Names_Get namespace, after +// Names namespace, after, more u8, count u16, entries of name and address +// Data channel, the answers to requests in the order asked: +// Table address, size u64, piece hashes +// Piece address, index u32, bytes +// Lack address, first u32, count u32: pieces it does not have; count 0: the blob +// Done the end of the answer to a request + +@(private) MSG_REQUEST :: 1 +@(private) MSG_REF_GET :: 2 +@(private) MSG_REF :: 3 +@(private) MSG_NAMES_GET :: 4 +@(private) MSG_NAMES :: 5 + +@(private) MSG_TABLE :: 1 +@(private) MSG_PIECE :: 2 +@(private) MSG_LACK :: 3 +@(private) MSG_DONE :: 4 + +@(private) ENTRY_BYTES :: 1+32+4+4 +@(private) MSG_REQUEST_HEAD :: 1+2 +@(private) MSG_REQUEST_MAX :: MSG_REQUEST_HEAD+REQUEST_ENTRIES*ENTRY_BYTES +@(private) MSG_TABLE_HEAD :: 1+32+8 +@(private) MSG_PIECE_HEAD :: 1+32+4 +@(private) MSG_LACK_BYTES :: 1+32+4+4 +@(private) MSG_NAMES_GET_MAX :: 1+2*(1+store.NAME_MAX) +@(private) MSG_REF_MAX :: MSG_NAMES_GET_MAX+1+32 +@(private) MSG_NAMES_MAX :: MSG_NAMES_GET_MAX+3+NAMES_REPLY_BYTES + +@(private) +put_u16 :: proc(b:[]u8,v:u16) {b[0],b[1]=u8(v),u8(v>>8)} +@(private) +put_u32 :: proc(b:[]u8,v:u32) {for i in 0..<4 {b[i]=u8(v>>(8*uint(i)))}} +@(private) +put_u64 :: proc(b:[]u8,v:u64) {for i in 0..<8 {b[i]=u8(v>>(8*uint(i)))}} +@(private) +get_u16 :: proc(b:[]u8) -> u16 {return u16(b[0])|u16(b[1])<<8} + +@(private) +Writer :: struct {data:[]u8,n:int} + +@(private) +write_u8 :: proc(w:^Writer,v:u8) {w.data[w.n]=v;w.n+=1} +@(private) +write_u16 :: proc(w:^Writer,v:u16) {put_u16(w.data[w.n:],v);w.n+=2} +@(private) +write_u32 :: proc(w:^Writer,v:u32) {put_u32(w.data[w.n:],v);w.n+=4} +@(private) +write_u64 :: proc(w:^Writer,v:u64) {put_u64(w.data[w.n:],v);w.n+=8} +@(private) +write_bytes :: proc(w:^Writer,v:[]u8) {copy(w.data[w.n:],v);w.n+=len(v)} +@(private) +write_name :: proc(w:^Writer,v:string) {write_u8(w,u8(len(v)));write_bytes(w,transmute([]u8)v)} + +// Reads fields of a message; any read past its end leaves ok false and gives zeros. +@(private) +Reader :: struct {data:[]u8,at:int,ok:bool} + +@(private) +read_bytes :: proc(r:^Reader,n:int) -> []u8 { + if !r.ok || n<0 || n>len(r.data)-r.at {r.ok=false;return nil} + b:=r.data[r.at:][:n] + r.at+=n + return b +} +@(private) +read_u8 :: proc(r:^Reader) -> u8 { + b:=read_bytes(r,1) + return r.ok?b[0]:0 +} +@(private) +read_u16 :: proc(r:^Reader) -> u16 { + b:=read_bytes(r,2) + return r.ok?get_u16(b):0 +} +@(private) +read_u32 :: proc(r:^Reader) -> (v:u32) { + b:=read_bytes(r,4) + if r.ok {for i in 0..<4 {v|=u32(b[i])<<(8*uint(i))}} + return +} +@(private) +read_u64 :: proc(r:^Reader) -> (v:u64) { + b:=read_bytes(r,8) + if r.ok {for i in 0..<8 {v|=u64(b[i])<<(8*uint(i))}} + return +} +@(private) +read_hash :: proc(r:^Reader) -> (h:store.Hash) { + b:=read_bytes(r,32) + if r.ok {copy(h[:],b)} + return +} +// A ref name, or with empty an empty one too; anything else leaves ok false. +@(private) +read_name :: proc(r:^Reader,empty:=false) -> string { + n:=int(read_u8(r)) + b:=read_bytes(r,n) + if !r.ok {return ""} + s:=string(b) + if !(empty && n==0) && !store.name_valid(s) {r.ok=false;return ""} + return s +} +// Whether the whole message was read and nothing else is left. +@(private) +read_end :: proc(r:^Reader) -> bool {return r.ok && r.at==len(r.data)} + +@(private) +encode_question :: proc(buffer:[]u8,kind:u8,namespace,name:string) -> []u8 { + w:=Writer{data=buffer} + write_u8(&w,kind) + write_name(&w,namespace) + write_name(&w,name) + return buffer[:w.n] +} + +@(private) +encode_lack :: proc(buffer:[]u8,address:store.Hash,first,count:u32) -> int { + address:=address + w:=Writer{data=buffer} + write_u8(&w,MSG_LACK) + write_bytes(&w,address[:]) + write_u32(&w,first) + write_u32(&w,count) + return w.n +} + +@(private) +encode_piece_head :: proc(buffer:[]u8,address:store.Hash,index:u32) { + address:=address + w:=Writer{data=buffer} + write_u8(&w,MSG_PIECE) + write_bytes(&w,address[:]) + write_u32(&w,index) +} + +// One whole message from peer i on a lane. +@(private) +message :: proc(x:^Exchange,i:int,lane:int,data:[]u8) { + if len(data)==0 {peer_bad(x,i);return} + if lane==LANE_DATA { + x.peers[i].last_heard=x.now + x.peers[i].stalled=false + switch data[0] { + case MSG_TABLE: on_table(x,i,data) + case MSG_PIECE: on_piece(x,i,data) + case MSG_LACK: on_lack(x,i,data) + case MSG_DONE: on_done(x,i,data) + case: peer_bad(x,i) + } + return + } + switch data[0] { + case MSG_REQUEST: on_request(x,i,data) + case MSG_REF_GET: on_ref_get(x,i,data) + case MSG_NAMES_GET: on_names_get(x,i,data) + case MSG_REF: on_ref(x,i,data) + case MSG_NAMES: on_names(x,i,data) + case: peer_bad(x,i) + } +} + +// A message larger than this end holds: a table of a blob larger than +// max_blob_bytes fails the want; anything else is from a bad peer. +@(private) +oversized :: proc(x:^Exchange,i:int,lane:int,head:[]u8) { + if lane==LANE_DATA && len(head)>=MSG_TABLE_HEAD && head[0]==MSG_TABLE { + r:=Reader{data=head[1:],ok=true} + address:=read_hash(&r) + size:=i64(read_u64(&r)) + if size>x.desc.max_blob_bytes { + if w:=want_find(x,address);w>=0 && x.wants[w].state!=.Complete {want_fail(x,w,.Too_Large,.None)} + return + } + } + peer_bad(x,i) +} diff --git a/store/blobs.odin b/store/blobs.odin index ee5c49d51..93d8e1eab 100644 --- a/store/blobs.odin +++ b/store/blobs.odin @@ -107,9 +107,12 @@ read :: proc(s:^Store,hash:Hash,offset:i64,buffer:[]u8) -> Error { } // The piece hashes of a complete blob: its kept table when that one matches -// the address and the size, otherwise hashed from the file once and kept. +// the address and the size, otherwise hashed from the file once and kept. Of +// a blob being received, the table it began with, checked against the address. blob_table :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (pieces:[]Hash,error:Error) { - r:=reader_open(s,hash) or_return + r,open_error:=reader_open(s,hash) + if open_error==.Missing {return partial_table(s,hash,allocator)} + if open_error!=.None {return nil,open_error} defer reader_close(&r) make_error:runtime.Allocator_Error pieces,make_error=make([]Hash,piece_count(r.size),allocator) @@ -126,6 +129,22 @@ blob_table :: proc(s:^Store,hash:Hash,allocator:=context.allocator) -> (pieces:[ return pieces,.None } +@(private) +partial_table :: proc(s:^Store,hash:Hash,allocator:runtime.Allocator) -> (pieces:[]Hash,error:Error) { + p:=partial_open(s,hash,{}) or_return + defer partial_close(&p) + make_error:runtime.Allocator_Error + pieces,make_error=make([]Hash,p.count,allocator) + if make_error!=nil {return nil,.Out_Of_Memory} + error=read_exact(p.table,mem.slice_to_bytes(pieces),TABLE_HEADER) + if error==.None && address_value(p.size,pieces)!=hash {error=.Corrupt} + if error!=.None { + delete(pieces,allocator) + return nil,error + } + return pieces,.None +} + // A complete blob held open for many range reads; its owner closes it. Reader :: struct { file:^os.File, diff --git a/store/receive.odin b/store/receive.odin index a09fd1233..c5249df19 100644 --- a/store/receive.odin +++ b/store/receive.odin @@ -78,15 +78,23 @@ receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash) -> Error { // Keeps one received piece: checked against the table and written at its // offset; a duplicate is harmless. The piece that leaves none missing -// completes the blob. +// completes the blob. A last piece that matches its table entry at another +// length is Wrong_Size: the blob was begun with a wrong size, and is removed +// and begun again with the one the piece gives. receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8) -> Error { p,error:=partial_open(s,hash,{.Data,.Have}) if error==.Missing {return complete_piece(s,hash,index,piece)} if error!=.None {return error} defer partial_close(&p) - if index<0 || index>=p.count || len(piece)!=piece_size(p.size,index) {return .Invalid_Argument} + if index<0 || index>=p.count || len(piece)==0 || len(piece)>PIECE_BYTES {return .Invalid_Argument} expected:Hash read_exact(p.table,expected[:],TABLE_HEADER+i64(index)*size_of(Hash)) or_return + if len(piece)!=piece_size(p.size,index) { + if index" // in its binaries; their sources stay behind, as do the tests. @(private="file") PACKAGE_TRACKED :: [?]string{ - "asset","audio","core","editor","engine","host","net","platform","plugins","project","render","rhi","std","store","ui", + "asset","audio","core","editor","engine","exchange","host","net","platform","plugins","project","render","rhi","std","store","ui", "shaders","templates","recipes","assets/fonts",".agents/skills","docs/game","LICENSE-MIT","LICENSE-APACHE","THIRD_PARTY_NOTICES.md", } // Generated from those sources, and kept by git in no checkout. From b9a3cb0d8f55bbda59777279b4e2eae560710ca7 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 19:59:19 -0300 Subject: [PATCH 018/123] exchange: fix the review's defects of trust, reservations and shutdown A table of one piece begins nothing until its piece hashes to the address (the store refuses to begin one), and a failed piece never blames a peer for another's table. Wrong_Size discards only the partial blob. Tables and pieces are taken only from the peer asked; an oversized table fails that peer for the blob, never the want. Reservations live on progress of their want, a lapsed flight releases once, marks leave with a peer's slot, ended questions keep their credits until answered, success is what the store says now, destroy wakes waiters with Closed, and requests no longer depend on the server's cap. want_addresses gives the owner what its lease must hold. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-store/SKILL.md | 6 +- .../skills/mirai-store/references/exchange.md | 36 ++- exchange/ask.odin | 148 +++++++++--- exchange/exchange.odin | 78 ++++-- exchange/exchange_test.odin | 222 +++++++++++++++++- exchange/harness_test.odin | 65 ++++- exchange/hostile_test.odin | 16 +- exchange/serve.odin | 2 +- exchange/wire.odin | 14 +- store/receive.odin | 16 +- store/store_test.odin | 27 ++- 11 files changed, 539 insertions(+), 91 deletions(-) diff --git a/.agents/skills/mirai-store/SKILL.md b/.agents/skills/mirai-store/SKILL.md index 321d8640e..1c5cc730c 100644 --- a/.agents/skills/mirai-store/SKILL.md +++ b/.agents/skills/mirai-store/SKILL.md @@ -46,8 +46,10 @@ derived data cache that use it decide what a ref means and what is live. bytes a piece, never its bytes. The root version carries the address rule: a root of another rule is refused, never migrated. - A table does not carry the length of its last piece: a size with the right piece count passes - the table check, and then the true last piece returns `Wrong_Size` instead of being kept, so - nothing is published and the receiver begins again with the size the piece proves. + the table check, and then the true last piece returns `Wrong_Size` instead of being kept: the + partial folder is discarded (a complete blob never is) and the receiver begins again with the + size the piece proves. A table of one piece is its own address and proves nothing, so + `receive_begin` refuses it: such a blob is put whole once its bytes hash to the address. - Only the two private procedures of `address.odin` (the value of a piece, the address of a table) touch the hash primitive; everything else, tests included, goes through them or the API, so the primitive can be replaced in one place. diff --git a/.agents/skills/mirai-store/references/exchange.md b/.agents/skills/mirai-store/references/exchange.md index 2f690c993..b865727c3 100644 --- a/.agents/skills/mirai-store/references/exchange.md +++ b/.agents/skills/mirai-store/references/exchange.md @@ -16,9 +16,12 @@ headless peer and the editor link the same code. Its messages are in `exchange/w its middle, so a restarted exchange takes new connections. - `want` (asking again raises the priority), `forget`, `progress`, `peer_progress`; `ref_ask`, `names_ask`, `query_result`, `query_end`. +- `want_addresses` gives the addresses of every want, in progress or kept: the owner puts them in + its store lease so a sweep keeps them; the exchange holds no lease. - `fetch` and `fetch_ref` block a job thread until the blob is complete in the store, or return `Missing` once no connected peer has it; the owner keeps pumping meanwhile, since every procedure - takes the exchange's lock. + takes the exchange's lock. Success is what the store says now: a kept want whose blob was swept + is asked again. `destroy` wakes every waiter with `Closed` and frees once none is left. ## Invariants @@ -33,19 +36,28 @@ headless peer and the editor link the same code. Its messages are in `exchange/w outstanding without queueing a low priority want ahead of a later high one. - Pieces are spread among the connected peers that do not lack the blob; a peer that lacks it is not asked again for a while, and a want with no peer left to ask is `Missing`, asked again when - a peer joins. + a peer joins. What a peer was asked stays reserved to it only while it makes progress on that + want; otherwise it goes to others, that peer is not asked for the want for a while, and its late + answers release nothing a second time. +- A request addresses at most `MAX_BLOB_BYTES` and never depends on the server's own cap: a blob + above the server's `max_blob_bytes` is answered as lacking. - Nothing unverified reaches a reader: tables and pieces go through `receive_begin` and - `receive_piece`. A table that does not match its address, or a piece that does not match its - table, marks that peer wrong for that blob; a true last piece that proves the size wrong - (`Wrong_Size`) marks whoever gave the table, removes the partial blob and asks the table of - others. A peer stays wrong for a blob until it leaves. + `receive_piece`. A table or piece is taken only from the peer it was asked of; anything else + counts against that peer. A table that does not match its address, claims a size this end does + not hold, or a piece that does not match its table, marks that peer wrong for that blob, never + the want; a true last piece that proves the size wrong (`Wrong_Size`) marks whoever gave the + table, and the store drops only the partial blob. A table of one piece begins nothing: the piece + that came with it is put only if it hashes to the address, so a failed piece never blames a peer + for a table another gave. Marks are per peer and leave with it, never passing to the next peer + in its slot. - What a peer exposes is the owner's `Desc.expose`; without one no namespace is read. Names come sorted, a page at a time after the last one seen. - Hostile input: a malformed message, a request beyond the limits or an answer nobody could have asked for in that form marks the peer bad, and it is neither asked nor served again; a well formed message about something not asked for is ignored. A request that does not fit the room for `BATCHES` requests is refused, and answers to questions find room only while the asker keeps a - few waiting, so a peer cannot make another send without bound. + few waiting, so a peer cannot make another send without bound. A question its owner ended stays + until every peer asked answered or left, so late answers return their credits. - Serving and receiving do not allocate after `init`, except listing a namespace, for which the store reads a folder. @@ -60,5 +72,11 @@ headless peer and the editor link the same code. Its messages are in `exchange/w `a_peer_still_receiving_a_blob_serves_its_pieces_to_a_third`, `refs_are_asked_and_listed_and_a_namespace_not_exposed_gives_nothing`, `the_blocking_form_returns_for_a_hit_a_miss_and_a_disconnect`, - `hostile_messages_are_refused_without_a_crash` and - `a_game_channel_keeps_its_delay_small_during_a_transfer`. + `hostile_messages_are_refused_without_a_crash`, + `a_game_channel_keeps_its_delay_small_during_a_transfer`, + `a_wrong_size_found_after_another_process_published_the_blob_keeps_it`, + `a_peer_that_holds_a_want_without_progress_loses_it`, + `a_peer_in_a_freed_slot_is_not_blamed_for_its_predecessor`, + `late_ref_answers_return_their_credits`, + `a_late_done_does_not_release_what_another_peer_was_asked`, `destroy_wakes_a_waiting_fetch` and + `a_server_with_the_smallest_cap_serves_an_honest_request`. diff --git a/exchange/ask.odin b/exchange/ask.odin index 397c85644..94126ee78 100644 --- a/exchange/ask.odin +++ b/exchange/ask.odin @@ -22,6 +22,11 @@ want_add :: proc(x:^Exchange,address:store.Hash,priority:int) -> (int,Error) { w.priority=priority order_sort(x) } + // A blob kept complete may have been swept since: success is what the store says now. + if w.state==.Complete && !store.has(x.desc.store,w.address) { + want_begin(x,i) + x.dirty=true + } return i,.None } for &w,i in x.wants { @@ -34,21 +39,30 @@ want_add :: proc(x:^Exchange,address:store.Hash,priority:int) -> (int,Error) { x.want_count+=1 order_sort(x) x.dirty=true - if store.has(x.desc.store,address) { - want_complete(x,i) - return i,.None - } - // A blob already being received resumes from the pieces it has. - size,size_error:=store.blob_size(x.desc.store,address) - if size_error==.None && size<=x.desc.max_blob_bytes { - arena:mem.Arena - if _,table_error:=store.blob_table(x.desc.store,address,scratch_arena(x,&arena));table_error==.None {table_known(x,i,-1,size)} - } + want_begin(x,i) return i,.None } return -1,.Full } +// From what the store has now: complete, being received, or nothing. +@(private) +want_begin :: proc(x:^Exchange,i:int) { + w:=&x.wants[i] + if store.has(x.desc.store,w.address) { + want_complete(x,i) + return + } + want_unknown(x,i) + w.state=.Asking + // A blob already being received resumes from the pieces it has. + size,size_error:=store.blob_size(x.desc.store,w.address) + if size_error==.None && size<=x.desc.max_blob_bytes { + arena:mem.Arena + if _,table_error:=store.blob_table(x.desc.store,w.address,scratch_arena(x,&arena));table_error==.None {table_known(x,i,-1,size)} + } +} + @(private) want_release :: proc(x:^Exchange,i:int) { x.wants[i].used=false @@ -97,7 +111,7 @@ want_fail :: proc(x:^Exchange,i:int,error:Error,store_error:store.Error) { @(private) want_unknown :: proc(x:^Exchange,i:int) { w:=&x.wants[i] - w.known,w.size,w.count,w.had,w.here,w.table_peer=false,0,0,0,0,-1 + w.known,w.size,w.count,w.had,w.here,w.table_peer,w.claim=false,0,0,0,0,-1,false slice.zero(w.have) slice.zero(w.asked) } @@ -169,22 +183,25 @@ lie :: proc(x:^Exchange,i:int,p:int) { bit:=u64(1)< bool { + p:=&x.peers[i] + w:=&x.wants[wi] + for k in 0..=int(f.first) && index0 && !p.stalled && x.now-p.last_heard>STALL_SECONDS { - p.stalled=true - flights_release(x,i) + for k in 0..=0 || budget=w.count {break} end:=start+1 for endx.desc.max_blob_bytes || len(data)!=MSG_TABLE_HEAD+store.piece_count(size)*size_of(store.Hash) {peer_bad(x,i);return} + if !r.ok {peer_bad(x,i);return} wi:=want_find(x,address) - if wi<0 {return} + if wi<0 || !flight_of(x,i,wi,FLIGHT_TABLE) {x.peers[i].refused+=1;return} w:=&x.wants[wi] - if w.state!=.Asking && w.state!=.Missing || w.known || w.wrong&(u64(1)<x.desc.max_blob_bytes || len(data)!=MSG_TABLE_HEAD+store.piece_count(size)*size_of(store.Hash) {lie(x,wi,i);return} pieces:=slice.reinterpret([]store.Hash,data[MSG_TABLE_HEAD:]) + if len(pieces)==1 { + // A table of one piece is its own address: only its piece, in the same answer, proves it. + if pieces[0]!=address {lie(x,wi,i);return} + w.table_peer,w.claim=i,true + return + } switch error:=store.receive_begin(x.desc.store,address,size,pieces);error { case .None: table_known(x,wi,i,size) case .Corrupt,.Invalid_Argument: lie(x,wi,i) @@ -321,9 +382,20 @@ on_piece :: proc(x:^Exchange,i:int,data:[]u8) { piece:=data[min(MSG_PIECE_HEAD,len(data)):] if !r.ok || len(piece)==0 || len(piece)>store.PIECE_BYTES {peer_bad(x,i);return} wi:=want_find(x,address) - if wi<0 {return} + if wi<0 || !flight_of(x,i,wi,index) {x.peers[i].refused+=1;return} w:=&x.wants[wi] - if w.state!=.Asking && w.state!=.Missing || !w.known || index>=w.count || bit_has(w.have,index) || w.wrong&(u64(1)<=w.count || bit_has(w.have,index) {return} + flights_heard(x,i,wi) switch error:=store.receive_piece(x.desc.store,address,index,piece);error { case .None: bit_on(w.have,index) @@ -338,10 +410,10 @@ on_piece :: proc(x:^Exchange,i:int,data:[]u8) { case .Corrupt,.Invalid_Argument: lie(x,wi,i) case .Wrong_Size: - // The piece proves the size the table came with was wrong. + // The piece proves the size the table came with was wrong; the store + // dropped that partial blob, and keeps a complete one published meanwhile. lie(x,wi,w.table_peer) - store.remove(x.desc.store,address) - want_unknown(x,wi) + if store.has(x.desc.store,address) {want_complete(x,wi)} else {want_unknown(x,wi)} x.dirty=true case .Missing: if store.has(x.desc.store,address) {want_complete(x,wi)} else {want_unknown(x,wi)} @@ -358,11 +430,11 @@ on_lack :: proc(x:^Exchange,i:int,data:[]u8) { read_u32(&r) if !read_end(&r) {peer_bad(x,i);return} wi:=want_find(x,address) - if wi<0 {return} + if wi<0 || !flight_of(x,i,wi,FLIGHT_ANY) {x.peers[i].refused+=1;return} w:=&x.wants[wi] if w.lack==0 {w.lack_since=x.now} w.lack|=u64(1)<=0 && f.blob { + w:=&x.wants[f.want] + if w.used && w.generation==f.generation && w.claim && w.table_peer==i {lie(x,f.want,i)} + } + flight_release(x,i,f) p.flight_head=(p.flight_head+1)%len(p.flights) p.flight_count-=1 } diff --git a/exchange/exchange.odin b/exchange/exchange.odin index e0b5a1f43..190a5df1f 100644 --- a/exchange/exchange.odin +++ b/exchange/exchange.odin @@ -24,6 +24,7 @@ REQUEST_BYTES :: 16<<20 // piece bytes one request asks for BATCHES :: 2 // requests in flight to a peer FIRST_PIECES :: 2 // pieces asked with the table of a blob NAMES_PER_REPLY :: 128 +MAX_BLOB_BYTES :: 1<<40 // the largest max_blob_bytes, and what a request may address @(private) LANE_CONTROL :: 0 // requests, refs and their answers @(private) LANE_DATA :: 1 // tables, pieces and the end of each request @@ -33,9 +34,10 @@ NAMES_PER_REPLY :: 128 @(private) MIN_BATCH_BYTES :: 1<<20 @(private) QUERIES_PER_PEER :: 3 @(private) LACK_RETRY :: 2.0 // s a peer that lacked a blob is not asked for it again -@(private) STALL_SECONDS :: 5.0 // s without data from a peer with requests in flight: what it was asked goes to others +@(private) STALL_SECONDS :: 5.0 // s a peer keeps what it was asked of a want without progress on it, before others are asked @(private) RATE_SECONDS :: 1.0 // time constant of the smoothed rates @(private) REQUEST_PIECES :: REQUEST_BYTES/store.PIECE_BYTES +@(private) MAX_PIECES :: MAX_BLOB_BYTES/store.PIECE_BYTES // Whether peers may read the refs of namespace. Expose :: #type proc(user:rawptr,namespace:string) -> bool @@ -57,8 +59,8 @@ Error :: enum u8 { Invalid, // bad argument, or a host without the channels of the exchange Full, // a capacity of the Desc is exhausted Missing, // no connected peer has it - Too_Large, // a blob larger than max_blob_bytes Store, // the store refused it: Progress.store_error + Closed, // the exchange was destroyed while waiting Out_Of_Memory, // only from init } @@ -113,6 +115,7 @@ Want :: struct { size:i64, count,had:int, table_peer:int, // the peer asked for the table, or that gave it; -1 + claim:bool, // table_peer gave a table of one piece: nothing is begun until the piece proves it wrong,lack:u64, // peers that lied about it, and that lack it lack_since:f64, misses:u32, // times it went Missing @@ -121,7 +124,14 @@ Want :: struct { } @(private) -Flight :: struct {want:int,generation:u32,first,count:u32,blob:bool} +Flight :: struct { + want:int, + generation:u32, + first,count:u32, + blob:bool, + released:bool, // what it reserved went back to be asked of others + heard:f64, // when it was sent or the peer last made progress on its want +} @(private) Serve_Kind :: enum u8 {Blob,Run,Done} @@ -131,7 +141,7 @@ Serve :: struct {kind:Serve_Kind,address:store.Hash,first,count:u32} @(private) Peer :: struct { - used,bad,stalled:bool, + used,bad:bool, host:^wire.Host, connection:wire.Connection, // Asking: the entries of the requests in flight, oldest first. @@ -139,7 +149,6 @@ Peer :: struct { flight_head,flight_count:int, batch_sizes:[BATCHES]int, batch_head,batches:int, - last_heard:f64, queries:int, // Serving: the entries of the requests received, oldest first. serve:[]Serve, @@ -168,6 +177,7 @@ Query_Kind :: enum u8 {Ref,Names} @(private) Query_Slot :: struct { used:bool, + ended:bool, // its owner is done with it; it stays until every peer asked answers, which frees their credits generation:u32, kind:Query_Kind, state:Query_State, @@ -198,6 +208,8 @@ Exchange :: struct { table_max:int, // pieces of the largest blob now,rate_time:f64, dirty:bool, + closing:bool, // destroy is waiting for the waiters to leave + waiting:int, // fetch calls waiting } channels :: proc() -> [CHANNELS]wire.Channel_Desc { @@ -213,7 +225,7 @@ init :: proc(x:^Exchange,desc:Desc) -> Error { if d.wants<=0 {d.wants=256} if d.queries<=0 {d.queries=16} if d.max_blob_bytes<=0 {d.max_blob_bytes=1<<30} - if d.peers>MAX_PEERS || d.max_blob_bytes>1<<40 {return .Invalid} + if d.peers>MAX_PEERS || d.max_blob_bytes>MAX_BLOB_BYTES {return .Invalid} x.table_max=store.piece_count(d.max_blob_bytes) words:=(x.table_max+63)/64 failed:=false @@ -249,7 +261,14 @@ init :: proc(x:^Exchange,desc:Desc) -> Error { return .None } +// Wakes every fetch with Closed and frees once none waits; the owner stops +// calling the exchange first. destroy :: proc(x:^Exchange) { + sync.mutex_lock(&x.lock) + x.closing=true + sync.cond_broadcast(&x.changed) + for x.waiting>0 {sync.cond_wait(&x.changed,&x.lock)} + sync.mutex_unlock(&x.lock) for w in x.wants {delete(w.have,x.allocator);delete(w.asked,x.allocator)} for p in x.peers { delete(p.flights,x.allocator);delete(p.serve,x.allocator);delete(p.out,x.allocator);delete(p.control,x.allocator) @@ -274,7 +293,7 @@ peer_add :: proc(x:^Exchange,h:^wire.Host,connection:wire.Connection) -> Error { for &p,i in x.peers { if p.used {continue} peer_reset(&p) - p.used,p.host,p.connection,p.last_heard=true,h,connection,x.now + p.used,p.host,p.connection=true,h,connection clear_peer_bits(x,i) x.dirty=true return .None @@ -316,7 +335,10 @@ peer_drop :: proc(x:^Exchange,i:int) { p:=&x.peers[i] flights_release(x,i) for &w in x.wants { - if w.used && w.table_peer==i && !w.known {w.table_peer=-1} + if w.used && w.table_peer==i {w.table_peer,w.claim=-1,false} + } + for &q in x.queries { + if q.used && q.kind==.Names && q.peer==i {q.peer=-1} } clear_peer_bits(x,i) peer_reset(p) @@ -338,7 +360,7 @@ peer_bad :: proc(x:^Exchange,i:int) { if p.bad {return} flights_release(x,i) for &w in x.wants { - if w.used && w.table_peer==i && !w.known {w.table_peer=-1} + if w.used && w.table_peer==i && !w.known {w.table_peer,w.claim=-1,false} } p.bad=true p.serve_count,p.out_len,p.out_sent,p.started=0,0,0,false @@ -448,37 +470,61 @@ fetch :: proc(x:^Exchange,address:store.Hash,priority:=0) -> Error { // complete in the store, or Missing once no connected peer names it or has it. fetch_ref :: proc(x:^Exchange,namespace,name:string,priority:=0) -> (store.Hash,Error) { sync.mutex_guard(&x.lock) + if x.closing {return {},.Closed} q,error:=query_add(x,.Ref,namespace,name,-1) if error!=.None {return {},error} slot:=&x.queries[q.index] slot.waiters+=1 - for slot.state==.Waiting {sync.cond_wait(&x.changed,&x.lock)} + x.waiting+=1 + for slot.state==.Waiting && !x.closing {sync.cond_wait(&x.changed,&x.lock)} slot.waiters-=1 + x.waiting-=1 found,address:=slot.state==.Found,slot.address - slot.used=false + slot.ended=true + if x.closing { + sync.cond_broadcast(&x.changed) + return {},.Closed + } if !found {return {},.Missing} return address,fetch_locked(x,address,priority) } @(private) fetch_locked :: proc(x:^Exchange,address:store.Hash,priority:int) -> Error { + if x.closing {return .Closed} i,error:=want_add(x,address,priority) if error!=.None {return error} w:=&x.wants[i] w.waiters+=1 + x.waiting+=1 misses:=w.misses result:Error for { + if x.closing {result=.Closed;break} if w.state==.Complete {break} if w.state==.Failed {result=w.error;break} if w.state==.Missing || w.misses!=misses {result=.Missing;break} sync.cond_wait(&x.changed,&x.lock) } w.waiters-=1 + x.waiting-=1 + if x.closing { + sync.cond_broadcast(&x.changed) + return .Closed + } if !w.wanted && w.waiters==0 {want_release(x,i)} return result } +// The addresses of every want, in progress or kept, into buffer (Desc.wants +// long): what the owner's store lease holds alive against a sweep. +want_addresses :: proc(x:^Exchange,buffer:[]store.Hash) -> []store.Hash { + sync.mutex_guard(&x.lock) + n:=min(x.want_count,len(buffer)) + for k in 0.. (Query,Error) { sync.mutex_guard(&x.lock) @@ -502,14 +548,14 @@ query_result :: proc(x:^Exchange,q:Query) -> (Query_Result,bool) { query_end :: proc(x:^Exchange,q:Query) { sync.mutex_guard(&x.lock) - if slot:=query_slot(x,q);slot!=nil && slot.waiters==0 {slot.used=false} + if slot:=query_slot(x,q);slot!=nil && slot.waiters==0 {slot.ended=true} } @(private) query_slot :: proc(x:^Exchange,q:Query) -> ^Query_Slot { if int(q.index)>=len(x.queries) {return nil} slot:=&x.queries[q.index] - if !slot.used || slot.generation!=q.generation {return nil} + if !slot.used || slot.ended || slot.generation!=q.generation {return nil} return slot } @@ -542,9 +588,11 @@ connected_mask :: proc(x:^Exchange) -> (mask:u64) { queries_send :: proc(x:^Exchange) { connected:=connected_mask(x) for &q in x.queries { - if !q.used || q.state!=.Waiting {continue} + if q.used && q.ended && q.asked&~q.answered&connected==0 {q.used=false} + if !q.used || q.ended || q.state!=.Waiting {continue} namespace,name:=string(q.namespace[:q.namespace_len]),string(q.name[:q.name_len]) - targets:=connected if q.kind==.Ref else connected&(u64(1)<=QUERIES_PER_PEER {continue} diff --git a/exchange/exchange_test.odin b/exchange/exchange_test.odin index 29d3c4343..5e88c4a13 100644 --- a/exchange/exchange_test.odin +++ b/exchange/exchange_test.odin @@ -1,6 +1,6 @@ package exchange -import "core:fmt" +import "core:slice" import "core:sync" import "core:testing" import "core:thread" @@ -121,10 +121,12 @@ a_transfer_resumes_after_the_connection_drops_and_after_the_store_reopens :: pro testing.expectf(t,got.received<=u64(8<<20)-u64(part.here)+u64(4*store.PIECE_BYTES),"after the reopen %d bytes arrive of the %d missing",got.received,(8<<20)-part.here) } -// Two sources, one of which lies; it is asked first, so its lie is met. +// Two sources, one of which lies; it is asked first, so its lie is met. A +// table of one piece for a blob of several, or one larger than the receiver +// holds, fails the liar for the blob, never the honest peer or the want. @(test) a_liar_in_the_table_a_piece_or_the_size_is_marked_and_the_blob_still_arrives :: proc(t:^testing.T) { - for lie in ([]Test_Lie{.Table,.Piece,.Size}) { + for lie in ([]Test_Lie{.Table,.Piece,.Size,.Single,.Huge}) { w:Test_World defer test_world_destroy(&w) if !test_world(t,&w,"exchange-liar",3) {return} @@ -283,6 +285,12 @@ the_blocking_form_returns_for_a_hit_a_miss_and_a_disconnect :: proc(t:^testing.T testing.expect(t,test_fetch(&w,&j,10) && j.error==.None && store.has(&w.nodes[1].s,hit),"a hit returns once the blob is here") _,still:=progress(x,hit) testing.expect(t,!still,"a fetch leaves no want behind") + // A want kept complete whose blob a sweep removed is fetched again: success is what the store says. + want(x,hit) + store.remove(&w.nodes[1].s,hit) + j={x=x,address=hit} + testing.expect(t,test_fetch(&w,&j,10) && j.error==.None && store.has(&w.nodes[1].s,hit),"a swept blob is fetched again") + forget(x,hit) keyed:=test_put(t,&w.nodes[0].s,3000,2) store.ref_set(&w.nodes[0].s,"lib","derived",keyed) j={x=x,namespace="lib",name="derived"} @@ -324,3 +332,211 @@ a_game_channel_keeps_its_delay_small_during_a_transfer :: proc(t:^testing.T) { mean:=g.mean/f64(max(g.got,1)) testing.expectf(t,mean<0.03 && g.worst<0.08,"game messages take %.0f ms more than the latency on average, %.0f at worst",mean*1000,g.worst*1000) } + +// A scripted peer at node k that answers as lie says, connected to the server at node 0. +@(private) +test_scripted :: proc(t:^testing.T,w:^Test_World,k:int,lie:Test_Lie) -> ^wire.Host { + h:=test_host(w,k,.Client) + w.nodes[k].script,w.nodes[k].lie,w.nodes[k].source=test_liar,lie,&w.nodes[k].s + return h +} + +@(private) +test_received_by :: proc(x:^Exchange,h:^wire.Host) -> (lies:[2]int) { + for c in h.connection_handles { + if c==0 {continue} + p,_:=peer_progress(x,h,c) + lies[0 if p.lies>0 else 1]+=1 + } + return +} + +// The true last piece proves a begun size wrong after another process +// published the blob: the published blob stays. +@(test) +a_wrong_size_found_after_another_process_published_the_blob_keeps_it :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-published",2) {return} + server:=test_host(&w,0,.Server) + silent:=test_scripted(t,&w,1,.Hold) + if !testing.expect(t,test_connect(&w,silent,w.nodes[0].endpoints[0])) {return} + size:=3*store.PIECE_BYTES+1000 + data:=test_bytes(size,4) + blob:=store.address_of_bytes(data) + r:=&w.nodes[0] + want(&r.x,blob) + test_step(&w,20) + c:=server.connection_handles[0] + pieces:=make([]store.Hash,4,context.temp_allocator) + { + other,_:=store.open(w.folder) + defer store.close(&other) + store.put_bytes(&other,data) + table,_:=store.blob_table(&other,blob,context.temp_allocator) + copy(pieces,table) + store.remove(&other,blob) + } + told:=test_message({MSG_TABLE},blob[:],test_le(u64(size-100),8),slice.to_bytes(pieces)) + receive(&r.x,server,{kind=.Message,channel=u8(LANE_DATA),connection=c,data=told}) + test_step(&w,20) + p,_:=progress(&r.x,blob) + testing.expect(t,p.total==i64(size-100),"the told size is begun") + another,_:=store.open(r.s.root) + defer store.close(&another) + store.put_bytes(&another,data) + last:=test_message({MSG_PIECE},blob[:],test_le(3,4),data[3*store.PIECE_BYTES:]) + receive(&r.x,server,{kind=.Message,channel=u8(LANE_DATA),connection=c,data=last}) + testing.expect(t,store.has(&r.s,blob) && store.verify(&r.s,blob)==.None,"the published blob stays") + testing.expect_value(t,test_state(&r.x,blob),State.Complete) +} + +// A peer that reserved a want and then sends only unrelated data loses the +// reservation, and the honest peer serves the blob. +@(test) +a_peer_that_holds_a_want_without_progress_loses_it :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-hold",3) {return} + test_host(&w,0,.Server) + holder:=test_scripted(t,&w,1,.Hold) + honest:=test_host(&w,2,.Client) + blob:=test_put(t,&w.nodes[2].s,2*store.PIECE_BYTES+7,6) + if !testing.expect(t,test_connect(&w,holder,w.nodes[0].endpoints[0]) && test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + want(&w.nodes[0].x,blob) + testing.expect(t,test_until_complete(&w,0,{blob},15),"the honest peer serves the blob once the reservation lapses") +} + +// What a peer was marked for leaves with it: the next peer in its slot is not +// blamed for the size the first one gave. +@(test) +a_peer_in_a_freed_slot_is_not_blamed_for_its_predecessor :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-slot",3) {return} + server:=test_host(&w,0,.Server) + liar:=test_scripted(t,&w,1,.Size_First) + honest:=test_host(&w,2,.Client) + size:=4*store.PIECE_BYTES+1000 + blob:=test_put(t,&w.nodes[1].s,size,8) + test_put(t,&w.nodes[2].s,size,8) + if !testing.expect(t,test_connect(&w,liar,w.nodes[0].endpoints[0])) {return} + r:=&w.nodes[0] + want(&r.x,blob) + for _ in 0..<2000 { + test_step(&w) + if p,_:=progress(&r.x,blob);p.total>0 && p.here==i64(size-store.PIECE_BYTES) {break} + } + p,_:=progress(&r.x,blob) + testing.expect(t,p.total==i64(size-100),"the liar's size is begun with every piece but the last") + wire.host_disconnect(liar) + for _ in 0..<2000 { + test_step(&w) + if server.connection_handles[0]==0 && server.connection_handles[1]==0 {break} + } + if !testing.expect(t,test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + testing.expect(t,test_until_complete(&w,0,{blob},20),"the honest peer in the freed slot completes the blob") + testing.expect_value(t,test_received_by(&r.x,server),[2]int{0,1}) +} + +// A lookup answered by a fast peer leaves the slow peer's credits to return +// with its late answers, so a later lookup that needs the slow peer finds it. +@(test) +late_ref_answers_return_their_credits :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-credits",3,desc={expose=test_expose_lib}) {return} + test_host(&w,0,.Server) + fast:=test_host(&w,1,.Client) + slow:=test_host(&w,2,.Client) + if !testing.expect(t,test_connect(&w,fast,w.nodes[0].endpoints[0]) && test_connect(&w,slow,w.nodes[0].endpoints[0])) {return} + wire.sim_endpoint_conditions(&w.sim,w.nodes[2].endpoints[0],{latency=1}) + blob:=test_put(t,&w.nodes[2].s,100,1) + store.ref_set(&w.nodes[2].s,"lib","only",blob) + names:=[]string{"ka","kb","kc","kd"} + for name in names { + store.ref_set(&w.nodes[1].s,"lib",name,blob) + store.ref_set(&w.nodes[2].s,"lib",name,blob) + } + x:=&w.nodes[0].x + for name in names { + q,_:=ref_ask(x,"lib",name) + for _ in 0..<500 { + test_step(&w) + if r,_:=query_result(x,q);r.state!=.Waiting {break} + } + r,_:=query_result(x,q) + testing.expectf(t,r.state==.Found,"%s is found by the fast peer",name) + query_end(x,q) + } + q,_:=ref_ask(x,"lib","only") + found:=false + for _ in 0..<8000 { + test_step(&w) + if r,_:=query_result(x,q);r.state!=.Waiting {found=r.state==.Found;break} + } + testing.expect(t,found,"the slow peer is asked and names it") +} + +// After a peer's flight lapses and its want goes to another, its late Done +// does not free what the other was asked: no piece is served twice. +@(test) +a_late_done_does_not_release_what_another_peer_was_asked :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-late",3) {return} + test_host(&w,0,.Server) + late:=test_scripted(t,&w,1,.Late_Done) + honest:=test_host(&w,2,.Client) + // Two pieces, both asked with the table: the honest peer has a request free when the late Done comes. + size:=2*store.PIECE_BYTES + blob:=test_put(t,&w.nodes[2].s,size,3) + if !testing.expect(t,test_connect(&w,late,w.nodes[0].endpoints[0]) && test_connect(&w,honest,w.nodes[0].endpoints[0])) {return} + wire.sim_endpoint_conditions(&w.sim,w.nodes[2].endpoints[0],{latency=0.01,bandwidth=0.5e6}) + want(&w.nodes[0].x,blob) + testing.expect(t,test_until_complete(&w,0,{blob},30),"the honest peer serves the blob") + served,_:=peer_progress(&w.nodes[2].x,honest,honest.connection_handles[0]) + testing.expectf(t,served.sent==u64(size),"the honest peer serves %d bytes of %d",served.sent,size) + testing.expectf(t,w.nodes[1].requests==1,"the late peer is asked %d times",w.nodes[1].requests) +} + +// Destroying the exchange wakes a fetch waiting on a job with Closed, and +// frees only once it left. +@(test) +destroy_wakes_a_waiting_fetch :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-destroy",2) {return} + test_host(&w,0,.Server) + silent:=test_scripted(t,&w,1,.Hold) + if !testing.expect(t,test_connect(&w,silent,w.nodes[0].endpoints[0])) {return} + n:=&w.nodes[0] + j:=Test_Job{x=&n.x,address=store.address_of_bytes(test_bytes(10,5))} + th:=thread.create_and_start_with_poly_data(&j,test_job) + test_step(&w,200) + testing.expect(t,!sync.atomic_load(&j.done),"the fetch waits on the silent peer") + destroy(&n.x) + n.open=false + thread.join(th) + thread.destroy(th) + testing.expect_value(t,j.error,Error.Closed) +} + +// A server whose largest blob is one piece still answers a request for a +// blob's table and first pieces. +@(test) +a_server_with_the_smallest_cap_serves_an_honest_request :: proc(t:^testing.T) { + w:Test_World + defer test_world_destroy(&w) + if !test_world(t,&w,"exchange-cap",2) {return} + test_node_close(&w.nodes[0]) + if !testing.expect(t,test_node_open(&w,0,{max_blob_bytes=store.PIECE_BYTES})) {return} + test_host(&w,0,.Server) + client:=test_host(&w,1,.Client) + if !testing.expect(t,test_connect(&w,client,w.nodes[0].endpoints[0])) {return} + blob:=test_put(t,&w.nodes[0].s,1000,2) + want(&w.nodes[1].x,blob) + testing.expect(t,test_until_complete(&w,1,{blob},5),"the blob arrives") + p,_:=peer_progress(&w.nodes[0].x,w.nodes[0].hosts[0],w.nodes[0].hosts[0].connection_handles[0]) + testing.expect(t,!p.bad,"the server keeps the requester") +} diff --git a/exchange/harness_test.odin b/exchange/harness_test.odin index efc40a090..33e3fd29e 100644 --- a/exchange/harness_test.odin +++ b/exchange/harness_test.odin @@ -20,10 +20,26 @@ Test_Node :: struct { lie:Test_Lie, source:^store.Store, // what the scripted peer serves game:Test_Game, + requests:int, // requests a scripted peer got + held:int, // Late_Done: Dones owed + held_at:f64, // when they are sent + held_host:^wire.Host, + held_connection:wire.Connection, } +// What a scripted peer does with the requests it gets. @(private) -Test_Lie :: enum {None,Table,Piece,Size} +Test_Lie :: enum { + None, + Table, // a table that does not match the address + Piece, // pieces that do not match the table + Size, // a size the last piece contradicts + Single, // for a blob of several pieces, a size of half a piece and the address as its table + Huge, // a table larger than the receiver holds + Size_First, // Size, without the last piece + Hold, // nothing, and an unrelated Lack every 0.1 s + Late_Done, // nothing, and a Done for each request 5.1 s later +} // A 60 Hz unreliable message from the server to the client on channel 0. @(private) @@ -134,6 +150,7 @@ test_step :: proc(w:^Test_World,steps:=1) { for command in wire.host_poll(h) {test_command(w,&n,h,command)} if h.desc.mode==.Server && w.channel==1 {test_game_send(w,&n,h)} } + if n.script!=nil {test_script_tick(w,&n)} if n.open {update(&n.x,w.now)} for h in n.hosts[:n.host_count] {wire.host_update(h,w.now)} } @@ -202,13 +219,36 @@ test_state :: proc(x:^Exchange,address:store.Hash) -> State { return p.state } +@(private) +test_script_tick :: proc(w:^Test_World,n:^Test_Node) { + if n.lie==.Late_Done && n.held>0 && w.now>=n.held_at { + done:=[1]u8{MSG_DONE} + for _ in 0..1 || kind==1 && n==0 || u64(first)+u64(n)>u64(x.table_max) || n>REQUEST_PIECES {peer_bad(x,i);return} + if kind>1 || kind==1 && n==0 || u64(first)+u64(n)>MAX_PIECES || n>REQUEST_PIECES {peer_bad(x,i);return} pieces+=int(n) } if pieces>REQUEST_PIECES {peer_bad(x,i);return} diff --git a/exchange/wire.odin b/exchange/wire.odin index 2f51d4ad3..265fa76a6 100644 --- a/exchange/wire.odin +++ b/exchange/wire.odin @@ -150,8 +150,6 @@ encode_piece_head :: proc(buffer:[]u8,address:store.Hash,index:u32) { message :: proc(x:^Exchange,i:int,lane:int,data:[]u8) { if len(data)==0 {peer_bad(x,i);return} if lane==LANE_DATA { - x.peers[i].last_heard=x.now - x.peers[i].stalled=false switch data[0] { case MSG_TABLE: on_table(x,i,data) case MSG_PIECE: on_piece(x,i,data) @@ -171,18 +169,16 @@ message :: proc(x:^Exchange,i:int,lane:int,data:[]u8) { } } -// A message larger than this end holds: a table of a blob larger than -// max_blob_bytes fails the want; anything else is from a bad peer. +// A message larger than this end holds: a table asked of that peer fails the +// peer for that blob, never the want; one nobody asked for counts against the +// peer; anything else is from a bad peer. @(private) oversized :: proc(x:^Exchange,i:int,lane:int,head:[]u8) { if lane==LANE_DATA && len(head)>=MSG_TABLE_HEAD && head[0]==MSG_TABLE { r:=Reader{data=head[1:],ok=true} address:=read_hash(&r) - size:=i64(read_u64(&r)) - if size>x.desc.max_blob_bytes { - if w:=want_find(x,address);w>=0 && x.wants[w].state!=.Complete {want_fail(x,w,.Too_Large,.None)} - return - } + if w:=want_find(x,address);w>=0 && flight_of(x,i,w,FLIGHT_TABLE) {lie(x,w,i)} else {x.peers[i].refused+=1} + return } peer_bad(x,i) } diff --git a/store/receive.odin b/store/receive.odin index 14a5149d5..76a9701f6 100644 --- a/store/receive.odin +++ b/store/receive.odin @@ -41,10 +41,12 @@ Claim :: struct { CLAIMED :: ".claimed" // Starts receiving a blob from its address and piece table; the table is -// checked against the address before anything is kept. +// checked against the address before anything is kept. A table of one piece +// is its own address and proves nothing: such a blob is put whole, once its +// bytes are at hand, and receive_begin refuses it. receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash,now:Maybe(time.Time)=nil) -> Error { if s==nil || s.root=="" {return .Invalid_Argument} - if size<0 || len(pieces)!=piece_count(size) {return .Invalid_Argument} + if size<0 || len(pieces)!=piece_count(size) || len(pieces)==1 {return .Invalid_Argument} if address_value(size,pieces)!=hash {return .Corrupt} refreshed:=blob_refresh(s,hash,now.? or_else time.now()) if refreshed==.None { @@ -80,8 +82,9 @@ receive_begin :: proc(s:^Store,hash:Hash,size:i64,pieces:[]Hash,now:Maybe(time.T // Keeps one received piece: checked against the table and written at its // offset; a duplicate is harmless. The piece that leaves none missing // completes the blob. A last piece that matches its table entry at another -// length is Wrong_Size: the blob was begun with a wrong size, and is removed -// and begun again with the one the piece gives. +// length is Wrong_Size: the blob was begun with a wrong size, and its partial +// folder is discarded (never a complete blob) to begin again with the size +// the piece gives. receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8,now:Maybe(time.Time)=nil) -> Error { if s==nil || s.root=="" {return .Invalid_Argument} p,error:=partial_open(s,hash,{.Data,.Have}) @@ -95,7 +98,10 @@ receive_piece :: proc(s:^Store,hash:Hash,index:int,piece:[]u8,now:Maybe(time.Tim if index=0;index-=1 { - testing.expect_value(t,receive_piece(&r,put,index,test_piece(data,index)),Error.None) + if len(pieces)==1 { + // A table of one piece proves nothing: the blob is put whole once its bytes are here. + testing.expect_value(t,receive_begin(&r,put,i64(c.size),pieces),Error.Invalid_Argument) + testing.expect_value(t,receive_begin(&r,put,i64(c.size)/2+1,pieces),Error.Invalid_Argument) + whole,whole_error:=put_bytes(&r,data) + testing.expect(t,whole_error==.None && whole==put) + } else { + testing.expect_value(t,receive_begin(&r,put,i64(c.size),pieces),Error.None) + for index:=len(pieces)-1;index>=0;index-=1 { + testing.expect_value(t,receive_piece(&r,put,index,test_piece(data,index)),Error.None) + } } for s in ([]^Store{&a,&b,&r}) { testing.expect(t,has(s,put),"every path ends with the blob") @@ -143,12 +151,21 @@ a_wrong_table_or_piece_is_refused_and_leaves_no_blob :: proc(t:^testing.T) { testing.expect_value(t,receive_begin(&s,hash,i64(len(data))-1,pieces),Error.None) last:=len(pieces)-1 for index in 0.. Date: Wed, 18 Nov 2026 20:07:09 -0300 Subject: [PATCH 019/123] net: a bulk stream keeps its rate on a path that reorders A datagram is lost only once reordering cannot explain its absence: datagrams sent after it were acked and it has been out longer than the RTT and a window from its variation, widened by any loss a late ack takes back. Late datagrams are acked when everything in them was taken, and stay in the acks until the other end read one; late acks are read. The RTT comes from the datagram that arrived last, the queue test against the lowest interval mean with its uncertainty, and unreliable game channels keep their order per channel. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-networking/SKILL.md | 9 +- .../mirai-networking/references/networking.md | 59 +++-- .../skills/mirai-store/references/exchange.md | 3 +- exchange/exchange_test.odin | 6 +- net/channel.odin | 40 ++-- net/flow_test.odin | 68 ++++-- net/peer.odin | 218 ++++++++++++++---- 7 files changed, 291 insertions(+), 112 deletions(-) diff --git a/.agents/skills/mirai-networking/SKILL.md b/.agents/skills/mirai-networking/SKILL.md index ea2d82a07..d129d8b28 100644 --- a/.agents/skills/mirai-networking/SKILL.md +++ b/.agents/skills/mirai-networking/SKILL.md @@ -158,9 +158,12 @@ In `plugins/multiplayer/runtime`: - **A connection finds its rate by itself.** Nothing in `Host_Desc` starts or caps it; a reliable channel holds in flight what its `queue_bytes` holds, and the game channels of a connection stay on time next to a stream on it. -- **Datagram acks and losses are final.** Late datagrams are not acked and only allow - [the partial delivery described in connections](references/networking.md#connections). Do not - extend that exception without reviewing channels, links and baselines. +- **A datagram is lost only when reordering cannot explain its absence**, and an ack is final. A + late datagram is acked only if everything in it that a resend would bring was taken + ([connections](references/networking.md#connections)); objects, the server time and the library + channels are never taken late. Do not extend what is taken late without reviewing channels, + links and baselines. A loss taken back by a late ack does not count for the rate; what was + resent for it stays resent. - **Nothing changes until it is known that the entry fits.** `entry_write` writes, checks and only then modifies the link; if it does not fit, it restores the writer. - **A baseline is used only after its ack**, and the receiver stores it from the state that results diff --git a/.agents/skills/mirai-networking/references/networking.md b/.agents/skills/mirai-networking/references/networking.md index 6f645fd5f..6778d5722 100644 --- a/.agents/skills/mirai-networking/references/networking.md +++ b/.agents/skills/mirai-networking/references/networking.md @@ -130,19 +130,29 @@ like `replication.hysteresis`) and the values it admits. Elias gamma code: a stream with few losses acks thousands of datagrams in a few bytes. Everything the sender has in flight arrives this way with its real fate, also after a receiver stall. The ack takes only what the clock or the inputs leave of the datagram: runs that do not fit are left - out, and the sender takes those datagrams as lost. + out, and the sender takes those datagrams as lost. It also names the datagram that arrived last + and how long ago, which the RTT is taken from: on a path that reorders, the newest one received + is the one that came fastest. - A receiver acks within `ACK_DELAY`, or with what it sends anyway, and at once after - `ACK_EVERY` datagrams that carried reliable chunks: a stream is acked as it arrives, and a - connection without one sends no datagram more for it. - - A datagram older than the last accepted is not acked: when a later one is acked, the fate of - the earlier ones is final and what was in a lost one is resent. - - If it arrives late (at most `LATE_WINDOW` behind) and had not been received, only what is safe - out of order is taken (`packets_late`): the chunks of the game's reliable channels, which - assemble only once, and on the server the inputs of ticks after the last received. - - Skipped are unreliable and sequenced messages, the server time and the objects, because what - arrived later is newer; and the library channels (reliable events and replication messages), - because they talk about objects whose state in that datagram is discarded and delivering them - earlier would change the order in which the game applies them. + `ACK_EVERY` datagrams that carried reliable chunks or one that came late: a stream is acked as + it arrives, and a connection without one sends no datagram more for it. + - **Loss.** A datagram missing from an ack is lost only when reordering can no longer explain + it, as RACK and QUIC judge it: `REORDER_COUNT` datagrams sent after it were acked, and it has + been out longer than the RTT and a window from its variation (`loss_delay`). Its records are + resolved then; acked ones are resolved as they are acked, in any order. A lost datagram that + an ack later reports as arrived takes its loss back, so it does not count for the rate, widens + the window to what it needed (fading over seconds), and what was resent for it is a duplicate. + - A datagram older than the last accepted came late. If it had not been received and lies within + `ACK_WINDOW`, its ack is read and what is safe out of order is taken (`packets_late`): the + chunks of the game's reliable channels, which assemble only once, the messages of a game + channel that delivered none from a later datagram (each channel keeps its order), and on the + server the inputs of ticks after the last received. If that was all it carried it is acked, + and the receiver keeps it in its acks until the other end read one that has it. + - Skipped are the server time and the objects, because what arrived later is newer; and the + library channels (reliable events and replication messages), because they talk about objects + whose state in that datagram is discarded and delivering them earlier would change the order + in which the game applies them. A late datagram with something skipped is not acked: what was + in it is resent. - With nothing to say it is sent anyway to ack, keep the connection alive or probe a queue with no ack. - **Send limit**: a token bucket with the connection's rate that stores at most one tick of rate @@ -155,13 +165,14 @@ like `replication.hysteresis`) and the values it admits. replication follows the rate. - **Congestion**: the connection finds the rate of its path; nothing in `Host_Desc` sets it. It starts at `CC_START_RATE` and doubles each interval that wanted more, until the queue appears - (the mean RTT of the interval above the lowest of the recent window, beyond `CC_QUEUE_DELAY` and - two datagrams at the rate). Then it drops below what the path delivered and grows by the same - bytes per second for every connection (at most a quarter each interval), so connections on one - path converge to equal shares and the queue stays short for the game traffic on it. Losses cut it - only with a queue or past `CC_OVERRUN_SHARE`; a link that loses at random keeps its rate, which - never drops below `CC_MIN_RATE`. An interval that delivered next to nothing, as after an outage, - starts again from `CC_START_RATE`. + (the mean RTT of the interval above the lowest mean of an interval in the recent window, beyond + `CC_QUEUE_DELAY` and two datagrams at the rate; each mean counts with the uncertainty its spread + and its number of samples leave, so jitter does not read as a queue). Then it drops below what + the path delivered and grows by the same bytes per second for every connection (at most a + quarter each interval), so connections on one path converge to equal shares and the queue stays + short for the game traffic on it. Losses cut it only with a queue or past `CC_OVERRUN_SHARE`; a + link that loses at random keeps its rate, which never drops below `CC_MIN_RATE`. An interval + that delivered next to nothing, as after an outage, starts again from `CC_START_RATE`. - **Close**: `host_disconnect` tells the other end with its reason; silence closes with `Timeout`, and an authentic datagram that cannot be interpreted, with `Protocol_Error`. The reasons for rejection and closing are `Disconnect_Reason` (`host.odin`). @@ -186,12 +197,12 @@ others their share. receiver joins; the library's own reliable channel takes only whole messages. - **Window.** The chunks a reliable channel keeps in flight grow by what the path acknowledges (`channel_window`, each congestion interval), so they double each round trip while they hold the - channel back. Twice what the connection's rate sends in a round trip, in chunks of the size the - channel's acknowledged ones had, bounds them, so congestion + channel back. What the connection's rate sends in a round trip and the time a loss takes to be + found, in chunks of the size the channel's acknowledged ones had, bounds them, so congestion control decides what a stream takes, and the channel's `queue_bytes` sets the most it holds in - flight and receives ahead (`channel_chunks`, a power of two up to `MAX_WINDOW`): a stream that - fills a path holds twice what the path carries in a round trip. A loss does not shrink the - window: its resend waits a round trip with the chunks after it acknowledged, ahead of new ones. + flight and receives ahead (`channel_chunks`, a power of two up to `MAX_WINDOW`). A loss does not + shrink the window: its resend waits with the chunks after it acknowledged until the loss is + found, ahead of new ones. - While objects wait for the update, the reliable game channels leave half of each datagram to them: a part that streams does not starve the state. - **Receiving.** A reliable channel delivers what follows what it delivered only while the command diff --git a/.agents/skills/mirai-store/references/exchange.md b/.agents/skills/mirai-store/references/exchange.md index 2f690c993..4cd59703d 100644 --- a/.agents/skills/mirai-store/references/exchange.md +++ b/.agents/skills/mirai-store/references/exchange.md @@ -53,7 +53,8 @@ headless peer and the editor link the same code. Its messages are in `exchange/w - `mirai check exchange` and `mirai test exchange` (no GPU): every test runs peers on `net`'s `Sim_Network` in virtual time with temporary stores. -- Focal tests: `blobs_around_the_piece_boundaries_and_a_large_one_arrive_verified_under_loss`, +- Focal tests: + `blobs_around_the_piece_boundaries_and_a_large_one_arrive_verified_under_loss_and_jitter`, `a_high_priority_want_asked_later_completes_before_a_large_low_one`, `a_transfer_resumes_after_the_connection_drops_and_after_the_store_reopens`, `a_liar_in_the_table_a_piece_or_the_size_is_marked_and_the_blob_still_arrives`, diff --git a/exchange/exchange_test.odin b/exchange/exchange_test.odin index 29d3c4343..ca73edd6d 100644 --- a/exchange/exchange_test.odin +++ b/exchange/exchange_test.odin @@ -24,7 +24,7 @@ test_put :: proc(t:^testing.T,s:^store.Store,size:int,seed:u64) -> store.Hash { } @(test) -blobs_around_the_piece_boundaries_and_a_large_one_arrive_verified_under_loss :: proc(t:^testing.T) { +blobs_around_the_piece_boundaries_and_a_large_one_arrive_verified_under_loss_and_jitter :: proc(t:^testing.T) { w:Test_World defer test_world_destroy(&w) if !test_pair(t,&w,"exchange-loss") {return} @@ -32,8 +32,8 @@ blobs_around_the_piece_boundaries_and_a_large_one_arrive_verified_under_loss :: sizes:=[]int{0,1,P-1,P,P+1,2*P,3*P+17,6<<20} addresses:=make([]store.Hash,len(sizes),context.temp_allocator) for size,k in sizes {addresses[k]=test_put(t,&w.nodes[0].s,size,u64(k)+1)} - for e in w.nodes[0].endpoints[:1] {wire.sim_endpoint_conditions(&w.sim,e,{latency=0.02,loss=0.05})} - wire.sim_endpoint_conditions(&w.sim,w.nodes[1].endpoints[0],{latency=0.02,loss=0.05}) + for e in w.nodes[0].endpoints[:1] {wire.sim_endpoint_conditions(&w.sim,e,{latency=0.02,jitter=0.005,loss=0.05})} + wire.sim_endpoint_conditions(&w.sim,w.nodes[1].endpoints[0],{latency=0.02,jitter=0.005,loss=0.05}) for a in addresses {testing.expect_value(t,want(&w.nodes[1].x,a),Error.None)} testing.expect(t,test_until_complete(&w,1,addresses,60),"every blob arrives") for a,k in addresses { diff --git a/net/channel.odin b/net/channel.odin index b8041a1dc..6c571a787 100644 --- a/net/channel.odin +++ b/net/channel.odin @@ -3,8 +3,8 @@ package mirai_net import "mirai:std/serial" // Messages travel inside the datagrams of a connection, on one of its -// channels. A datagram is delivered whole and in order or not at all, so what -// a kind adds is on top of that: +// channels. Each channel delivers in order: a message overtaken by a later one +// of its channel is dropped. What a kind adds is on top of that: // // Unreliable sent once; lost with its datagram. // Sequenced as Unreliable, and a message still waiting to be sent is @@ -97,6 +97,7 @@ Channel :: struct { budget,spent,max_message:int, queue:Byte_Ring, // Reliable: a message streamed in is cut as its bytes come applied:u64, // bytes of the messages for this connection alone taken from the frames + delivered_after:u64, // Unreliable and Sequenced: one past the datagram of the last message delivered // Reliable, sending: chunks send_base..0) || !(rate>0) {return} + if !(span>0) || !(rate>0) {return} wire:=acked>0?f64(bytes)/f64(acked)+4:CHUNK_BYTES+4 // bytes a chunk takes in a datagram - chunks:=min(f64(c.window)+f64(acked),2*rate*rtt/wire,f64(len(c.chunks))) + chunks:=min(f64(c.window)+f64(acked),rate*span/wire,f64(len(c.chunks))) c.window=max(int(chunks),min(CHUNK_WINDOW,len(c.chunks))) } @@ -343,11 +344,12 @@ channel_resolve :: proc(p:^Peer,record:Record,received:bool) { } @(private) -// A late datagram delivers only the reliable chunks of the game's channels, which are numbered and taken -// once. Its unreliable and sequenced messages are older than those delivered since. The channels of the -// library carry events about objects whose state in the same datagram is skipped, and keep their order -// with it by waiting for the resend. -channels_read :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,late:=false) -> bool { +// A late datagram delivers the reliable chunks of the game's channels, which are numbered and taken once, +// and the unreliable and sequenced messages of a game channel that delivered none from a later datagram: +// each channel keeps its order, and one beside a stream on a path that reorders still arrives. The +// channels of the library carry events about objects whose state in the same datagram is skipped, and +// keep their order with it by waiting for the resend. counter: the datagram's. +channels_read :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,counter:u64,late:=false,skipped:^bool=nil) -> bool { scratch:=h.back.scratch[:] chunked:=false defer if chunked {p.ack_chunks+=1} @@ -358,7 +360,8 @@ channels_read :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,late:=false) -> bool if c.kind!=.Reliable { length:=int(serial.bit_read(r,11)) if length>c.max_message || !serial.bit_read_bytes(r,scratch[:length]) {return false} - if late {continue} + if late && (index>=USER_CHANNELS || counter+1 bool length:=int(serial.bit_read(r,9)) if length>CHUNK_BYTES || !serial.bit_read_bytes(r,scratch[:length]) {return false} chunked=true - if late && index>=USER_CHANNELS {continue} + if late && index>=USER_CHANNELS { + if skipped!=nil {skipped^=true} + continue + } size:=len(c.recv_chunks) if int(u16(seq-c.recv_base))>=size {continue} slot:=int(seq)%size diff --git a/net/flow_test.odin b/net/flow_test.odin index e355866dd..4c3fa99bc 100644 --- a/net/flow_test.odin +++ b/net/flow_test.odin @@ -83,7 +83,7 @@ a_large_reliable_part_streams_at_what_the_path_carries_and_the_state_keeps_its_r } // An acknowledgement of a datagram never sent closes the connection, one that comes again for a chunk -// counts it once, and no measure grows a window past twice what the rate sends in a round trip nor past +// counts it once, and no measure grows a window past what the rate sends in its span nor past // what the channel holds, nor shrinks it below where it starts. @(test) hostile_acknowledgements_are_refused_and_never_push_a_window_past_its_bounds :: proc(t:^testing.T) { @@ -110,22 +110,22 @@ hostile_acknowledgements_are_refused_and_never_push_a_window_past_its_bounds :: channel_resolve(p,record,true) testing.expect_value(t,c.acked,1) - Case :: struct {start,acked,bytes:int,rtt,rate:f64,low,high:int} + Case :: struct {start,acked,bytes:int,span,rate:f64,low,high:int} nan:=math.nan_f64() cases:=[]Case{ - {start=64,acked=max(int)/2,rtt=1e9,rate=1e30,low=1024,high=1024}, - {start=64,acked=1<<20,bytes=(1<<20)*CHUNK_BYTES,rtt=0.05,rate=1<<20,low=1,high=409}, - {start=64,acked=max(int)/2,bytes=max(int)/2,rtt=0.05,rate=1<<20,low=1,high=1024}, - {start=512,acked=0,rtt=0.05,rate=16<<20,low=512,high=512}, - {start=512,acked=1<<20,rtt=nan,rate=1<<20,low=512,high=512}, - {start=512,acked=1<<20,rtt=0.05,rate=nan,low=512,high=512}, - {start=512,acked=1<<20,rtt=-1,rate=1<<20,low=512,high=512}, - {start=512,acked=1<<20,rtt=1e-9,rate=0,low=512,high=512}, - {start=512,acked=1<<20,rtt=1e-9,rate=1,low=CHUNK_WINDOW,high=CHUNK_WINDOW}, + {start=64,acked=max(int)/2,span=1e9,rate=1e30,low=1024,high=1024}, + {start=64,acked=1<<20,bytes=(1<<20)*CHUNK_BYTES,span=0.1,rate=1<<20,low=1,high=409}, + {start=64,acked=max(int)/2,bytes=max(int)/2,span=0.1,rate=1<<20,low=1,high=1024}, + {start=512,acked=0,span=0.1,rate=16<<20,low=512,high=512}, + {start=512,acked=1<<20,span=nan,rate=1<<20,low=512,high=512}, + {start=512,acked=1<<20,span=0.1,rate=nan,low=512,high=512}, + {start=512,acked=1<<20,span=-1,rate=1<<20,low=512,high=512}, + {start=512,acked=1<<20,span=1e-9,rate=0,low=512,high=512}, + {start=512,acked=1<<20,span=1e-9,rate=1,low=CHUNK_WINDOW,high=CHUNK_WINDOW}, } for k in cases { c.window,c.acked,c.acked_bytes=k.start,k.acked,k.bytes - channel_window(c,k.rtt,k.rate) + channel_window(c,k.span,k.rate) testing.expectf(t,c.window>=k.low && c.window<=k.high && c.acked==0 && c.acked_bytes==0,"%v gives a window of %d",k,c.window) } @@ -135,6 +135,7 @@ hostile_acknowledgements_are_refused_and_never_push_a_window_past_its_bounds :: serial.bit_write_bool(&wr,true) serial.bit_write(&wr,u64(p.send_seq+10),16) serial.bit_write(&wr,0,ACK_COUNT_BITS) + serial.bit_write_bool(&wr,false) serial.bit_write(&wr,0,8) rd:serial.Bit_Reader serial.bit_reader_init(&rd,serial.bit_writer_flush(&wr)) @@ -240,12 +241,13 @@ Bulk_Result :: struct { game_mean,game_worst:f64, // s a 60 Hz unreliable message took above the latency, once warm game_sent,game_got:int, drop_share:f64, // of the datagrams the server sent, those its link dropped for lack of queue + resent_share:f64, // of the bytes the first client received once warm, those its server sent again } // A server streams reliable parts to each client over its 100 Mbit/s link with every setting at its // default, the worker's cadence of 1 ms, and a 60 Hz unreliable message to each on the same connection. @(private="file") -bulk_stream :: proc(t:^testing.T,rtt:f64,loss:f32,clients:int,seconds:f64) -> (result:Bulk_Result,ok:bool) { +bulk_stream :: proc(t:^testing.T,rtt:f64,loss:f32,clients:int,seconds:f64,jitter:=0.0) -> (result:Bulk_Result,ok:bool) { blob:=make([]u8,BULK_PART);defer delete(blob) w:Test_World;test_world_init(&w,1+clients,seed=31,packets=1<<16);defer test_world_destroy(&w) server:=test_host(&w,{mode=.Server,channels=BULK_CHANNELS}) @@ -256,14 +258,15 @@ bulk_stream :: proc(t:^testing.T,rtt:f64,loss:f32,clients:int,seconds:f64) -> (r on_server[i],_=test_connect(&w,server,hosts[i]) if !testing.expect(t,on_server[i]!=0,"the client connects") {return} } - sim_endpoint_conditions(&w.sim,0,{latency=rtt/2,loss=loss,bandwidth=BULK_LINK}) - for i in 1..=clients {sim_endpoint_conditions(&w.sim,i,{latency=rtt/2,loss=loss})} + sim_endpoint_conditions(&w.sim,0,{latency=rtt/2,jitter=jitter,loss=loss,bandwidth=BULK_LINK}) + for i in 1..=clients {sim_endpoint_conditions(&w.sim,i,{latency=rtt/2,jitter=jitter,loss=loss})} sent:[2]int got:[2]f64 start:=w.now next_game:=start game_total:f64 sim_before:=w.sim.stats + resent_before:u64 for w.now-start (r if command.kind==.Disconnected {testing.expect(t,false,"the connection holds");return} } warm:=w.now-start>=BULK_WARM + if !warm { + stats,_:=connection_stats(server,on_server[0]) + resent_before=stats.chunks_resent + } for i in 0.. (r result.game_mean=game_total/f64(max(result.game_got,1)) sent_datagrams:=w.sim.stats.sent-sim_before.sent result.drop_share=f64(w.sim.stats.dropped_queue-sim_before.dropped_queue)/f64(max(sent_datagrams,1)) + stats,_:=connection_stats(server,on_server[0]) + result.resent_share=f64(stats.chunks_resent-resent_before)*CHUNK_BYTES/max(got[0],1) return result,true } @@ -329,6 +338,35 @@ a_bulk_stream_finds_the_rate_of_the_path_and_leaves_the_game_channel_on_time :: } } +// A path that reorders its datagrams by jitter, as internet paths do, neither slows a reliable stream +// nor makes it resend what arrived: with jitter alone it takes most of the link, with jitter and loss +// it keeps near what the same loss without jitter gives, and the 60 Hz unreliable game channel on the +// same connection stays on time and loses only what a later message of its own overtook. +@(test) +a_bulk_stream_keeps_its_rate_on_a_path_that_reorders :: proc(t:^testing.T) { + Case :: struct {jitter:f64,loss:f32} + steady:f64 + for c in ([]Case{{0.005,0},{0.02,0},{0,0.01},{0.005,0.01},{0.02,0.01}}) { + r,ok:=bulk_stream(t,0.04,c.loss,1,12,c.jitter) + if !ok {return} + fmt.printfln("bulk 40 ms, %.0f ms jitter, %.0f%%: %.2f MB/s, %.2f%% resent, game %.1f ms mean, %.1f ms worst, %d/%d, %.2f%% dropped",c.jitter*1000,c.loss*100,r.delivered[0]/1e6,r.resent_share*100,r.game_mean*1000,r.game_worst*1000,r.game_got,r.game_sent,r.drop_share*100) + if c.jitter==0 {steady=r.delivered[0];continue} + if c.loss==0 { + testing.expectf(t,r.delivered[0]>0.6*BULK_LINK,"%v: %.0f B/s arrive over a link of %.0f",c,r.delivered[0],BULK_LINK) + testing.expectf(t,r.resent_share<0.02,"%v: %.1f%% of what arrived was sent again",c,r.resent_share*100) + } else { + testing.expectf(t,r.delivered[0]>0.8*steady,"%v: %.0f B/s arrive, %.0f with the same loss and no jitter",c,r.delivered[0],steady) + testing.expectf(t,r.resent_share<4*f64(c.loss),"%v: %.1f%% of what arrived was sent again",c,r.resent_share*100) + } + // The channel keeps its order: a message that a later one overtook by jitter is dropped, which + // happens to two consecutive ones when their spread of 2 jitter exceeds the 1/60 s between them. + overtaken:=max(1-1.0/60/(2*c.jitter),0) + overtaken=overtaken*overtaken/2 + testing.expectf(t,r.game_got>=int(0.9*(1-f64(c.loss))*(1-overtaken)*f64(r.game_sent)),"%v: %d of %d game messages arrive",c,r.game_got,r.game_sent) + testing.expectf(t,r.game_mean<0.03 && r.game_worst<0.08+c.jitter,"%v: game messages take %.0f ms more than the latency on average, %.0f at worst",c,r.game_mean*1000,r.game_worst*1000) + } +} + // Two connections that stream over the same link each get a fair share of it. @(test) two_bulk_streams_on_one_link_share_it :: proc(t:^testing.T) { diff --git a/net/peer.odin b/net/peer.odin index b1cc21d65..cfa2d8ec1 100644 --- a/net/peer.odin +++ b/net/peer.odin @@ -4,6 +4,7 @@ import "core:crypto" import "core:crypto/siphash" import "base:intrinsics" import "core:hash/xxhash" +import "core:math" import "core:mem" import "core:slice" import "mirai:std/base" @@ -46,7 +47,9 @@ MAX_CONNECTIONS :: 1024 @(private) ACK_WINDOW :: SENT_WINDOW-1 // datagrams before the last received an acknowledgement covers at most @(private) RECORD_WINDOW :: SENT_WINDOW-1-64 // datagrams in flight past which only one a keepalive goes @(private) ACK_COUNT_BITS :: 12 // of how many datagrams an acknowledgement covers: up to ACK_WINDOW -@(private) LATE_WINDOW :: 32 // datagrams behind the last received that are still taken late +@(private) LATE_WINDOW :: 32 // behind the last received, taken late with something skipped +@(private) NO_REACK :: max(u64) +@(private) REORDER_COUNT :: 3 // datagrams acknowledged after one before it is taken as lost @(private) HANDSHAKE_BYTES :: 256 @(private) HANDSHAKE_INTERVAL :: 0.2 @(private) COOKIE_SECONDS :: 10 @@ -57,7 +60,7 @@ MAX_CONNECTIONS :: 1024 @(private) CHECKSUM_BYTES :: 4 @(private) ACCEPT_BYTES :: 18 // server salt, object capacity, ticks of history and the slot of the connection, sealed @(private) CANDIDATES :: 2 // addresses a client asks at once: those of an invitation -@(private) TRANSPORT_VERSION :: 9 // of the layout of the handshake and of the datagrams after it: raised when it changes +@(private) TRANSPORT_VERSION :: 10 // of the layout of the handshake and of the datagrams after it: raised when it changes Connection_Stats :: struct { rtt,rtt_variance,jitter:f32, // seconds @@ -126,19 +129,31 @@ Peer :: struct { recv_bits:Ack_Bits, late_bits:u32, // received late and taken, the first LATE_WINDOW of recv_bits; never acknowledged ack_known:u64, // the oldest datagram received that the other end may not have resolved yet + // Datagrams that came late and that the acknowledgement must cover until the other end read one that + // does: the oldest that one of this end's datagrams from reack_seq on carries, and the oldest of + // those that came after, which the datagrams from reack_next_seq on carry. NO_REACK when none. + reack,reack_next:u64, + reack_seq,reack_next_seq:u16, recv_any,ack_pending,has_rtt:bool, ack_chunks:int, // datagrams with reliable chunks received since this end last sent - recv_time:f64, + recv_time:f64, // when the newest datagram received arrived + last_arrival:u64, // the datagram acknowledged that arrived last, which the other end takes its RTT from + last_time:f64, // and when sent:[SENT_WINDOW]Sent, + sent_acked,sent_lost:[SENT_WINDOW/64]u64, // per slot of sent: its datagram was acknowledged, or taken as lost + largest_acked:u16, // the newest datagram acknowledged, once acked_any + acked_any:bool, records:[]Record, record_head,record_tail:u32, rtt,rtt_variance,jitter:f64, + latest_rtt:f64, // the last sample + reorder:f64, // s past the round trip that the latest arrival of a datagram taken as lost showed, fading tokens,token_time:f64, // Congestion control: the rate the connection may send at, judged over intervals of two round trips. rate:f64, starting:bool, // doubling the rate each interval until the path shows a queue - min_rtt,window_min,window_time:f64, // smallest RTT of the last window, and of the current one - cc_time,cc_rtt:f64, // start of the interval and the sum of its RTT samples + min_rtt,window_min,window_time:f64, // lowest mean RTT of an interval in the last window, and in the current one + cc_time,cc_rtt,cc_rtt2:f64, // start of the interval, the sum of its RTT samples and of their squares cc_samples:int, cc_acked,cc_lost:u64, // acked and lost datagrams at its start delivered,cc_delivered:u64, // bytes of datagrams acknowledged, and at the interval's start @@ -174,10 +189,10 @@ ack_bits_advance :: proc "contextless" (bits:^Ack_Bits,shift:u64) { // lengths of the runs of arrived and missing ones in turn, each in Elias gamma code. Within budget bits: // the runs that do not fit are left out, and the other end takes those datagrams as lost. @(private) -ack_bits_write :: proc(w:^serial.Bit_Writer,bits:^Ack_Bits,count,budget:int) { +ack_bits_write :: proc(w:^serial.Bit_Writer,bits:^Ack_Bits,count,budget:int) -> int { count:=ack_fit(bits,count,budget) serial.bit_write(w,u64(count),ACK_COUNT_BITS) - if count==0 {return} + if count==0 {return 0} set:=ack_bit(bits,1) serial.bit_write_bool(w,set) for at:=0;at0 && p.cc_rtt/f64(p.cc_samples)-p.min_rtt>CC_QUEUE_DELAY+2*PACKET_BYTES/p.rate + // Jitter spreads the samples around their mean, and the far end of that spread is not a queue: the + // floor is the lowest mean of an interval, and a mean counts with the uncertainty that its spread + // and its number of samples leave, above when it sets the floor and below when it is judged. + n:=f64(p.cc_samples) + mean:=n>0?p.cc_rtt/n:0 + spread:=n>1?math.sqrt(max(p.cc_rtt2-n*mean*mean,0)/(n-1)/n):p.rtt_variance + if n>0 { + if p.min_rtt==0 || mean+spread0 && mean-spread-p.min_rtt>CC_QUEUE_DELAY+2*PACKET_BYTES/p.rate overrun:=acked+lost>=16 && f64(lost)>CC_OVERRUN_SHARE*f64(acked+lost) if queued || overrun { congested:=overrun || lost>0 && f64(lost)>CC_LOSS_SHARE*f64(acked+lost) @@ -467,11 +499,12 @@ congestion_update :: proc(h:^Host,p:^Peer) { if p.window_min>0 {p.min_rtt=p.window_min} p.window_min,p.window_time=0,now } + p.reorder-=p.reorder*min(elapsed/8,1) for &c in p.channels { - if c.active && c.kind==.Reliable {channel_window(&c,p.rtt,p.rate)} + if c.active && c.kind==.Reliable {channel_window(&c,p.rtt+loss_delay(p),p.rate)} } p.stats.send_limit=f32(p.rate) - p.cc_time,p.cc_rtt,p.cc_samples,p.cc_limited=now,0,0,false + p.cc_time,p.cc_rtt,p.cc_rtt2,p.cc_samples,p.cc_limited=now,0,0,0,false p.cc_acked,p.cc_lost,p.cc_delivered=p.stats.packets_acked,p.stats.packets_lost,p.delivered } @@ -817,7 +850,91 @@ peer_resolve :: proc(h:^Host,p:^Peer,seq:u16,received:bool) { record:=p.records[i%u32(len(p.records))] if record.kind==.Chunk {channel_resolve(p,record,received)} else {repl_resolve(h,p,record,seq,received)} } - p.record_head=sent.first+u32(sent.count) +} + +@(private) sent_flag :: #force_inline proc "contextless" (bits:^[SENT_WINDOW/64]u64,seq:u16) -> bool { + slot:=int(seq)%SENT_WINDOW + return bits[slot/64]&(1< f64 { + return max(p.rtt,p.latest_rtt)+max(4*p.rtt_variance,p.rtt/8,p.reorder,0.001) +} + +// A datagram is taken as lost only once reordering cannot explain its absence: REORDER_COUNT datagrams +// sent after it were acknowledged, and it has been out longer than a round trip and the window that +// the variation of the round trip gives, as RACK and QUIC judge it. The tail of what is in flight +// moves past what is settled; the records stay until the oldest datagram that holds them is. +@(private) +peer_settle :: proc(h:^Host,p:^Peer) { + now:=h.back.now + window:=loss_delay(p) + for p.sent_tail!=p.send_seq { + seq:=p.sent_tail + if !sent_flag(&p.sent_acked,seq) && !sent_flag(&p.sent_lost,seq) { + if !p.acked_any || i16(p.largest_acked-seq) bool { + now:=h.back.now + in_flight:=u16(p.send_seq-p.sent_tail) + for behind:u64=0;behind<=ack_count;behind+=1 { + seq:=ack-u16(behind) + if u16(p.send_seq-seq)>SENT_WINDOW {break} + if behind>0 && !ack_bit(ack_bits,behind) || sent_flag(&p.sent_acked,seq) {continue} + sent_mark(&p.sent_acked,seq,true) + if behind==arrival { + sample:=max(now-p.sent[seq%SENT_WINDOW].time-delay,0) + if !p.has_rtt { + p.rtt,p.rtt_variance,p.has_rtt=sample,sample*0.5,true + } else { + deviation:=abs(sample-p.rtt) + p.rtt_variance+=(deviation-p.rtt_variance)*0.25 + p.jitter+=(deviation-p.jitter)*0.0625 + p.rtt+=(sample-p.rtt)*0.125 + } + p.latest_rtt=sample + p.stats.rtt,p.stats.rtt_variance,p.stats.jitter=f32(p.rtt),f32(p.rtt_variance),f32(p.jitter) + p.cc_rtt+=sample + p.cc_rtt2+=sample*sample + p.cc_samples+=1 + } + if sent_flag(&p.sent_lost,seq) { + // The window grows to what this one needed, with a margin, so the next does not repeat it. + sent_mark(&p.sent_lost,seq,false) + p.stats.packets_lost-=1 + p.reorder=min(max(p.reorder,1.25*(now-p.sent[seq%SENT_WINDOW].time-max(p.rtt,p.latest_rtt))),1) + continue + } + if u16(seq-p.sent_tail)>=in_flight {continue} + if !p.acked_any || seq_newer(seq,p.largest_acked) {p.largest_acked,p.acked_any=seq,true} + p.ack_known=max(p.ack_known,p.sent[seq%SENT_WINDOW].known) + for p.reack!=NO_REACK && !seq_newer(p.reack_seq,seq) { + p.reack,p.reack_seq,p.reack_next=p.reack_next,p.reack_next_seq,NO_REACK + } + peer_resolve(h,p,seq,true) + if p.state!=.Connected {return false} + } + peer_settle(h,p) + return p.state==.Connected } // False when the datagram is malformed: the caller closes the connection. @@ -827,18 +944,35 @@ peer_receive :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,counter:u64) -> bool { has_ack:=serial.bit_read_bool(r) ack:=u16(serial.bit_read(r,16)) ack_bits,ack_count,acks_ok:=ack_bits_read(r) + arrival:=serial.bit_read_bool(r)?serial.bit_read(r,ACK_COUNT_BITS):0 delay:=f64(serial.bit_read(r,8))*0.001 - if !acks_ok || r.error {return false} + if !acks_ok || r.error || arrival>ack_count || arrival>0 && !ack_bit(&ack_bits,arrival) {return false} if p.recv_any && counter<=p.recv_counter { - // Within the window of acknowledgements and not received before: it came late. What in it is safe - // out of order is taken; it is not acknowledged, so the sender resends the rest as if it were lost. + // Within the window of acknowledgements and not received before: it came late. Its + // acknowledgement and what in it is safe out of order are taken. behind:=p.recv_counter-counter - if behind==0 || behind>LATE_WINDOW || ack_bit(&p.recv_bits,behind) || p.late_bits&(1<<(behind-1))!=0 {p.stats.packets_out_of_order+=1;return true} - p.late_bits|=1<<(behind-1) + if behind==0 || behind>ACK_WINDOW || ack_bit(&p.recv_bits,behind) || behind<=LATE_WINDOW && p.late_bits&(1<<(behind-1))!=0 {p.stats.packets_out_of_order+=1;return true} + if has_ack && !seq_newer(p.send_seq,ack) {return false} p.stats.packets_late+=1 p.last_receive=now + if has_ack && !acks_take(h,p,ack,&ack_bits,ack_count,arrival,delay) {return true} if !time_read(h,p,r,late=true) || p.state!=.Connected {return p.state!=.Connected} - if !channels_read(h,p,r,late=true) || p.state!=.Connected {return p.state!=.Connected} + skipped:bool + if !channels_read(h,p,r,counter,late=true,skipped=&skipped) || p.state!=.Connected {return p.state!=.Connected} + // Everything in it that a resend would bring again was taken: it is acknowledged, from as far + // back as it lies, so that the sender knows it arrived even if it already took it as lost. + if !skipped && (h.back.repl.protocol==nil || !serial.bit_read_bool(r)) && !r.error { + p.recv_bits[(behind-1)/64]|=1<<((behind-1)%64) + if p.reack==NO_REACK { + p.reack,p.reack_seq=counter,p.send_seq + } else { + p.reack_next,p.reack_next_seq=min(p.reack_next,counter),p.send_seq + } + // At once, as a receiver acks a segment that fills a gap: the sender waits for it. + p.ack_pending,p.ack_chunks,p.last_arrival,p.last_time=true,ACK_EVERY,counter,now + } else if behind<=LATE_WINDOW { + p.late_bits|=1<<(behind-1) + } return true } if has_ack && !seq_newer(p.send_seq,ack) {return false} @@ -849,35 +983,12 @@ peer_receive :: proc(h:^Host,p:^Peer,r:^serial.Bit_Reader,counter:u64) -> bool { p.stats.packets_missed+=shift-1 } p.recv_counter,p.recv_any,p.ack_pending=counter,true,true - p.last_receive,p.recv_time=now,now + p.last_receive,p.recv_time,p.last_arrival,p.last_time=now,now,counter,now if has_ack { - for p.sent_tail!=p.send_seq && !seq_newer(p.sent_tail,ack) { - behind:=u16(ack-p.sent_tail) - received:=behind==0 || (u64(behind)<=ack_count && ack_bit(&ack_bits,u64(behind))) - if behind==0 { - sample:=max(now-p.sent[ack%SENT_WINDOW].time-delay,0) - if !p.has_rtt { - p.rtt,p.rtt_variance,p.has_rtt=sample,sample*0.5,true - } else { - deviation:=abs(sample-p.rtt) - p.rtt_variance+=(deviation-p.rtt_variance)*0.25 - p.jitter+=(deviation-p.jitter)*0.0625 - p.rtt+=(sample-p.rtt)*0.125 - } - p.stats.rtt,p.stats.rtt_variance,p.stats.jitter=f32(p.rtt),f32(p.rtt_variance),f32(p.jitter) - if p.min_rtt==0 || samplep.ack_known?int(min(p.recv_counter-p.ack_known,ACK_WINDOW)):0,ack_budget) - serial.bit_write(&w,u64(clamp((b.now-p.recv_time)*1000,0,255)),8) + ack_budget:=serial.bit_writer_remaining(&w)-1-ACK_COUNT_BITS-8-time_max_bits(h)-SECTION_RESERVE + from:=min(p.ack_known,p.reack,p.reack_next) + covered:=ack_bits_write(&w,&p.recv_bits,p.recv_any && p.recv_counter>from?int(min(p.recv_counter-from,ACK_WINDOW)):0,ack_budget) + // The datagram that arrived last, and how long ago: on a path that reorders, the newest one + // received is the one that came fastest, and an RTT taken only from it reads short. + arrival,arrived:=u64(0),p.recv_time + if p.recv_counter-p.last_arrival<=u64(covered) {arrival,arrived=p.recv_counter-p.last_arrival,p.last_time} + serial.bit_write_bool(&w,arrival>0) + if arrival>0 {serial.bit_write(&w,arrival,ACK_COUNT_BITS)} + serial.bit_write(&w,u64(clamp((b.now-arrived)*1000,0,255)),8) wrote:=time_write(h,p,&w,seq) yield:=b.repl.key_cursor0?min(budget,PACKET_BYTES*4):0 if channels_write(h,p,&w,SECTION_RESERVE,yield) {wrote=true} @@ -931,6 +1049,8 @@ peer_send :: proc(h:^Host,p:^Peer) { break } bytes:=data_seal(h,p,&w) + sent_mark(&p.sent_acked,seq,false) + sent_mark(&p.sent_lost,seq,false) p.sent[seq%SENT_WINDOW]={time=b.now,known=p.recv_any?p.recv_counter+1:0,first=first_record,count=u16(p.record_tail-first_record),bytes=u16(len(bytes))} p.send_seq+=1 p.send_counter+=1 From faf21dfbffb4c0568cd529209aa33f208e89bfa3 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 20:08:44 -0300 Subject: [PATCH 020/123] editor: import through the machine library Importing a file or folder ingests it into the library where it is and pins its package (sources.project_ingest); nothing is copied and bring is gone from Source_Importer. The session keeps a store (--store, else the machine's) that reaches the library as a value. Deleting the root of a package unpins it (sources.project_unpin) and moving it moves where the package is listed; other listed assets are refused naming the root. packages.list and packages.add, with Add from library in the content browser, pin a library package or some of its assets. The prepare of plugin importers takes the library store and the root instead of the project registry, and sources.Library passes the store. Co-Authored-By: Claude Opus 5.5 --- .../mirai-agent-api/references/methods.md | 32 +- .agents/skills/mirai-editor/SKILL.md | 4 + .../mirai-editor/references/extensions.md | 25 +- .../skills/mirai-editor/references/panels.md | 66 +-- .../mirai-imports/references/imports.md | 29 +- asset/registry.odin | 21 + cook/import/sources/command.odin | 2 +- cook/import/sources/library.odin | 37 +- cook/import/sources/reimport_test.odin | 4 +- cook/import/sources/support_test.odin | 2 +- docs/game/assets.md | 10 +- editor/api_assets.odin | 9 +- editor/asset_delete.odin | 64 ++- editor/asset_move.odin | 120 ++++- editor/assets_parts.odin | 2 +- editor/content.odin | 1 + editor/editor.odin | 21 +- editor/import.odin | 464 ++++++++---------- editor/import_api_test.odin | 206 ++++++++ editor/library.odin | 337 +++++++++++++ 20 files changed, 1071 insertions(+), 385 deletions(-) create mode 100644 editor/library.odin diff --git a/.agents/skills/mirai-agent-api/references/methods.md b/.agents/skills/mirai-agent-api/references/methods.md index c2546a123..4faa30c81 100644 --- a/.agents/skills/mirai-agent-api/references/methods.md +++ b/.agents/skills/mirai-agent-api/references/methods.md @@ -823,17 +823,24 @@ of an instance" are `-32602`; the rest, `-32000` another asset (that of an imported prefab), is its owner. - `asset.import`: what the Import button of the content browser does with any source it imports ([import](../../mirai-editor/references/panels.md#import)), wherever it is on disk: the - editor reads it and writes only in the assets folder. `folder` replaces the folder of its type; a - source that a plugin imports whole (a map) does not accept it. It imports, writing to disk (it - is an operation on files) while the editor goes on, and answers when it ends: whether it wrote - (`ok`), its message, the scene of a map and the assets it wrote (`kept`: its file was already - the one it gives). `asset.import_status` gives the same, and whether one runs (`importing`), at - any moment. While it runs, another import and whatever writes assets fail saying what is being - imported. Reimporting keeps ids and files, which the sidecar of the source lists + source stays where it is, its package goes to the store of the session and the project pins it, + listed in `folder` (else the root, or the folder of a source in the assets). It imports, + writing to disk (it is an operation on files) while the editor goes on, and answers when it + ends: whether it wrote (`ok`), its message, the scene of a map and the assets it wrote + (`kept`: its file was already the one it gives). `asset.import_status` gives the same, and + whether one runs (`importing`), at any moment. While it runs, another import and whatever + writes assets fail saying what is being imported. Reimporting keeps ids and files, which the + package of the source lists ([identity](../../mirai-imports/references/imports.md#identity-and-reimport)). `image` (`color`, `data`, `normal`, `normal_green_down`) says what an image imported on its own holds, over its settings and its name (a packed metallic-roughness or a mask is `data`); a source in the project keeps the choice in its settings +- `packages.list {name?}`: the packages of the library of the machine (the store of the session) + with their assets (`name`, `kind`, `bytes`, `built`) and whether the project pins each (`pinned`, + `folder`). `packages.add {name, assets?, folder?}` pins one, by id or name, or some of its + assets by id, listed in `folder`, as Add from library of the content browser; it runs as an + import and answers as `asset.import`. An unknown package or asset, an invalid folder or a + session without a store fail saying how to fix it. - `asset.detach`: `asset` (path or id) is a part that leaves its owner: it is left without `owner` and shown alone; the import keeps writing it and leaves it out ([parts](../../mirai-imports/references/imports.md#parts)). It is an edit of its file, with undo; @@ -847,14 +854,17 @@ of an instance" are `-32602`; the rest, `-32000` `{id,path}`; an unknown type fails naming the valid ones, and a taken or invalid name says why. - `asset.delete`: `asset` (path or id) goes to the system trash with the import parts it owns, like Delete of the [content browser](../../mirai-editor/references/panels.md#content-browser): at - once, without undo. Returns `{moved,used_by}`; those that use it keep the - reference. Fails saying why with an asset of a plugin, of the engine or of the editor, a scene - open in a tab or outside Edit. + once, without undo; the root of a package also takes the package out of the project, and what + it listed leaves with it while the store keeps its blobs. Returns `{moved,used_by}`; those that + use it keep the reference. Fails saying why with another asset a package lists, an asset of a + plugin, of the engine or of the editor, a scene open in a tab or outside Edit. - `asset.move {asset, folder?, name?, copy?}` takes an asset by path or id, or a folder by path, and writes at once, without undo; it returns the resulting path and id ([rename, move and duplicate](../../mirai-editor/references/panels.md#content-browser)). No `folder` keeps the one it is in and `/` is the root. Moving keeps the id and takes the parts - along; what is being edited fails asking to finish the edit. With `copy` it + along; the root of a package moves where the project lists it, keeping its name, and another + asset a package lists fails naming that root; what is being edited fails asking to finish the + edit. With `copy` it copies with new ids and the copy is not an output of the import; without `name` it takes a free one. - `asset.users`: `asset` (path or id). Returns `{used_by}`, the files that reference it or a part diff --git a/.agents/skills/mirai-editor/SKILL.md b/.agents/skills/mirai-editor/SKILL.md index 06d13fbf1..0d5e5ec66 100644 --- a/.agents/skills/mirai-editor/SKILL.md +++ b/.agents/skills/mirai-editor/SKILL.md @@ -159,6 +159,10 @@ of the `Editor_World` and its functions are `@(step)` of the `editor` loop, like are off, and the project library does not swap while a plugin's handler is on the stack. - Asset folders are thematic, never by type (the browser filters by type): what the editor imports or creates goes where the user is, and it creates no folder named after a type. +- The editor imports through the library and the store of its session + ([import](references/panels.md#import)): a source is read where it is and never copied into a + project, and importing, deleting or moving the root of a package changes only the project's + `packages.mirai` and its documents, never a blob of the store nor a source outside the project. - State located according to [the session](references/editor.md#the-session). `Api` keeps only a pointer because the server must stay where it was born. - A procedure takes the storage it reads and writes (`^Ai`, the chat store, `^Git`, `^Edits`), diff --git a/.agents/skills/mirai-editor/references/extensions.md b/.agents/skills/mirai-editor/references/extensions.md index bc99bc652..70f0f32c9 100644 --- a/.agents/skills/mirai-editor/references/extensions.md +++ b/.agents/skills/mirai-editor/references/extensions.md @@ -393,29 +393,28 @@ and joins them with the engine's. those of `sources.FORMATS`, with the same reading, writing, cache and identity ([plugin formats](../../mirai-imports/references/imports.md#plugin-formats)); its `Format.folder` chooses the folder of the import. Its extensions enter the dialog's Models - filter and what the editor recognizes on dropping a file. With `prepare(assets, info, + filter and what the editor recognizes on dropping a file. With `prepare(library, root, info, allocator)` it gives, before a model is imported, the format's options for that import - (`Format.data`) from the registry of the project and the library of its plugins (for example - what other providers give), with no session: the import program of the engine prepares it the - same way when a build syncs the project (`import_library`). `release(data, allocator)` - releases them on finishing. + (`Format.data`), with no session: `library` is the store the packages are in (the session's in + the editor, the one of the command in the import program), `root` the project or plugin the + import is for (`asset.root_packages(root, library.root)` lists its packages) and `info` the + types of its game, with what other providers give. The import program of the engine prepares + it the same way when a build refreshes the project (`import_library`). `release(data, + allocator)` releases them on finishing. - `editor.Import_Source_Provider{source}` gives a `Source_Importer`, a source the plugin imports whole with `importer.Outputs`, the single source of every asset it makes ([plugin formats](../../mirai-imports/references/imports.md#plugin-formats)). Its fields are `name` (the dialog's filter and the messages), `extensions`, `claims(path, head)` (claims the file by its extension and its first `importer.SNIFF_BYTES`; it is tried before the formats), - `prepare(assets, info, allocator)`, `run(state, source, options, allocator)`, - `release(state, allocator)` and `bring(source, folder, allocator)`. + `prepare(library, root, info, allocator)`, `run(state, source, options, allocator)` and + `release(state, allocator)`. The source is read where it is: nothing is copied into a project. - `prepare` runs before starting, like that of a format, with no session: what `run` needs from - the registry of the project and the library of its plugins (for example, what other providers - give) is gathered there, in the received allocator, and is the `state`. `run` runs in the + the library, the root and the types of its game (for example, what other providers give) is + gathered there, in the received allocator, and is the `state`. `run` runs in the import's thread with a job system of its own in `options.worker` and returns the report, the scene it wrote (`""` if none) and the error. `release` releases `state` on finishing. The - import program prepares and runs it the same way when a build syncs the project + import program prepares and runs it the same way when a build refreshes the project (`import_library`). - - `bring`, when given, takes a source from outside the assets: it copies it with the files it - reads into `folder` (absolute, in the assets) and returns where the source is now. Without it - the editor copies the source alone. - A plugin can define its own providers so that other plugins join it: the one that gives the source gathers them in `prepare` and passes them to `run` through the `state`. This way an import plugin delegates to automatic bridge plugins the actors of optional domains, without diff --git a/.agents/skills/mirai-editor/references/panels.md b/.agents/skills/mirai-editor/references/panels.md index 098c68e3e..9d13689aa 100644 --- a/.agents/skills/mirai-editor/references/panels.md +++ b/.agents/skills/mirai-editor/references/panels.md @@ -558,9 +558,9 @@ assets. They are off outside Edit and while watching. (`content_focused`, which `shortcuts_run` looks at before the entity Delete), opens `Modal.Delete_Asset` with the asset, its parts and what uses it. Confirming sends it to the trash (`platform.trash`) immediately, without undo: it is an operation on files, like import. -What uses it keeps the reference. Assets of a plugin, the -engine or the editor, an open scene or prefab, are not deleted, nor outside Edit; the modal says -why. +What uses it keeps the reference. The root of a package takes the package out of the project +([import](#import)). Assets of a plugin, the engine or the editor, an open scene or prefab, are +not deleted, nor outside Edit; the modal says why. **Rename, move and duplicate** (`asset_move.odin`) also change the files immediately, without undo, and rescan the registry. Identity is the id, so references, Recent, Favorites and open @@ -579,8 +579,9 @@ documents follow the asset; the selection passes to it (`Content.reveal`). registry paths that the scan replaces. - **Parts and imports.** An asset moves with the parts it owns; each keeps its place relative to the owner's folder (the one outside it goes to the destination folder), and if one cannot go, - nothing moves. The link with the source does not change: `Import_Info.source` names the source - from the project and reimporting finds the outputs by their `Import_Info` + nothing moves; a part a package lists goes with the listing of its package ([import](#import)). + The link with the source does not change: `Import_Info.source` names the source from its + package and reimporting finds the outputs by their ids ([identity](../../mirai-imports/references/imports.md#identity-and-reimport)). - **History and references.** The history names the moved files at their new path, so that undo and redo write there, and the project's `start_scene` follows its scene. @@ -846,15 +847,16 @@ plugin [vfx](../../mirai-world/references/vfx-editor.md#particle-document). ## Import -`import_file(s, path)` converts a source file into project assets, only in `Edit`, with -`sources.import_source` like `mirai import`. A source in the assets of the project or of a plugin -imports where it is; one from outside the project is first copied into `folder` -(`asset_folder_clean`, the folder of the browser), and one in the project but outside its assets -is refused. The source is versioned with its sidecar beside what it makes, and what it builds -that nobody edits goes to the imported folder -([identity](../../mirai-imports/references/imports.md#identity-and-reimport)). The imported folder -is brought up to date with the sources by the build that opens the editor -([refresh](../../mirai-imports/references/imports.md#library)). +`import_file(s, path)` ingests a source file or folder from anywhere on disk into the library of +the machine, where it is, and pins its package in the project, only in `Edit`, with +`sources.project_ingest` like `mirai import` +([library](../../mirai-imports/references/imports.md#library)). +Nothing is copied: the package is listed in `folder` (`asset_folder_clean`, the folder of the +browser; for a source in the assets without one, its own folder), in the project or in the plugin +whose assets that folder is. What nobody edits is listed from the blobs of the store, and the +documents are files of that folder. The store is the session's: `--store`, else the machine's +(`store.default_root`); the registry lists from the same one. The build that opens the editor +refreshes the packages ([refresh](../../mirai-imports/references/imports.md#library)). File > Import, the content browser, a file dropped on the window and `asset.import` call it. It imports in a thread of its own, with its own job system, while the editor keeps drawing; the content browser shows the dot of what works. `import_poll`, between frames, takes what it wrote on @@ -870,28 +872,36 @@ output folders in [importers](../../mirai-imports/SKILL.md). imported before the formats and takes no folder. - An image is sRGB unless its settings or its name say it is data or a normal (`sources.image_use`, which `mirai import` applies too); `asset.import {image}` chooses it. -- The sidecar of a source (`.import`) keeps its settings and the id of each asset; - reimporting writes the same assets with their ids where they are and keeps what did not change. - Textures with +- The package of a source keeps its settings and the id of each asset; reimporting writes the + same assets with their ids where they are and keeps what did not change. Textures with the same image and settings and meshes with the same geometry and settings come from their caches ([textures](../../mirai-imports/references/imports.md#texture-cache), [meshes](../../mirai-imports/references/imports.md#mesh-cache)). -- On finishing, the written files are refreshed in the registry and the replaced assets are - reloaded; if Surface programs were written (`Import_Report.programs`), the project's programs - are published (`programs_publish_request`) without waiting for another change. +- On finishing, the registry lists the packages of the root again and rescans, and the replaced + assets are reloaded; if Surface programs were written (`Import_Report.programs`), the + project's programs are published (`programs_publish_request`) without waiting for another change. - **Parts.** Meshes and skeletons of a model, and the materials and textures that are only its, are parts of its prefab (`asset.Header.owner`, [rule](../../mirai-imports/references/imports.md#parts)). `asset_detach` (`editor/assets_parts.odin`, `asset.detach`) takes one out of its owner as an edit of its file with undo: it only clears `owner`, and the import leaves it out; there is no UI that calls it yet. -- **What an import keeps in the imported folder is not edited where it is.** Moving, renaming, - detaching or deleting a part on its own is refused, saying to duplicate it. Moving the root of - an import of the cache (`asset_move`, the texture of an image) moves its source and sidecar, and - moving a folder moves the sources in it: the registry lists their assets there at once, without - an import. Deleting the root of an import (its prefab, the texture of an image) gives the import - up: its source and its sidecar go to the trash with what only the import can build. A - copy made outside the editor receives an id of its own on scanning - ([repeated ids](../../mirai-assets/references/formats.md#registry)). +- **What a package lists is not edited where it is.** Moving, renaming, detaching or deleting + another asset it lists is refused, naming the root of the package and saying to duplicate it. + The root of a package (the prefab of a model, the scene of a whole source, else its one asset + that is no part: the texture of an image) stands for it: moving it (`asset_move`) moves where + the project lists the package (`packages.mirai`, `sources.project_pin`), keeping the name its + source gives a listed asset, and moving a folder moves the packages listed in it. Deleting it + takes the package out of the project (`sources.project_unpin`): its documents go to the trash, + what it listed leaves the registry, its blobs stay in the store, and a source of it inside the + project goes to the trash while one outside stays. None of these is undone by undo; version + control brings `packages.mirai` back. A copy made outside the editor receives an id of its own + on scanning ([repeated ids](../../mirai-assets/references/formats.md#registry)). +- **Add from library.** `packages.list` lists the packages of the store with their assets (name, + kind, bytes, whether this machine has them built) and whether the project pins each; + `packages.add` pins one, or some of its assets, in a folder, as an import that runs in the + thread of imports and answers as `asset.import` (`sources.project_add`, `library.odin`). Add + from library of the menu of the content browser offers the packages and adds one whole to the + folder of the browser; one the project pins is shown checked and off. ## Graphics diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index 02f900cc1..2b4c845b5 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -9,9 +9,10 @@ Packages, all *engine-time*: - `cook/import/obj`, `cook/import/gltf`, `cook/import/usd` and `cook/import/fbx` (on top of ufbx): one format each, with its `FORMAT`. They import the core and nothing from the other formats. - `cook/import/sources`: the `FORMATS` table (the engine's formats), `source_kind` and - `import_source`, used by the editor (`editor/import.odin`), `mirai import` (`command.odin`, in the - engine's `mirai-import` program) and the [library](#library) (`ingest`, `project_add`, - `project_refresh`). The editor adds the formats of plugins ([plugin formats](#plugin-formats)). + `import_source`, used by the [library](#library) (`ingest`, `project_ingest`, `project_add`, + `project_refresh`), through which the editor (`editor/import.odin`) and `mirai import` + (`command.odin`, in the engine's `mirai-import` program) import. The editor adds the formats of + plugins ([plugin formats](#plugin-formats)). - `cook/textures`: images to `.tex` levels. ## API @@ -153,11 +154,13 @@ part ([extension](../../mirai-editor/references/extensions.md#importers-of-a-plu - `Import_Format_Provider{format}` is one more `importer.Format`: it is imported like the engine's, with the same storage (the cache and the documents, [identity](#identity-and-reimport)). `Format.folder` says the folder of its assets. Its extensions are in the Models filter of the - dialog. Its `prepare` gives the format's options for an import (`Format.data`) from the assets - of the project and the library of its plugins, and `release` frees them when it ends. -- The editor imports with the providers of its library; the import program of the engine - (`mirai import`, the [refresh](#library) of every build) loads the editor library of the project - and takes the same providers from it (`editor.import_library`, `sources.Library`), only when a + dialog. Its `prepare` gives the format's options for an import (`Format.data`) from the store of + the library, the root the import is for and the types of its game, and `release` frees them + when it ends. +- The editor imports with the providers of its session, which it prepares and gives the library + (`Library_Options.imports`); the import program of the engine (`mirai import`, the + [refresh](#library) of every build) loads the editor library of the project and takes the same + providers from it with its store (`editor.import_library`, `sources.Library`), only when a source is read by no format of the engine or a model imports, whose prefab is written in the types of the game the library has. A clone without that library built fails those sources and says to build the editor. @@ -292,7 +295,8 @@ shared by every checkout, worktree and project of the machine, keeps their manif or worktrees that ingest one origin take turns with a lock per origin (`cook/workspace`, under `/locks/`), and the second finds the work done. - **`project_add(root, manifest, folder, assets, options)`** pins a package in the project - (`project_pin`, the only writer of `packages.mirai`) and follows it there. **Following** writes + (`project_pin` and `project_unpin`, the only writers of `packages.mirai`) and follows it there; + `project_ingest` ingests a source and pins its package in one import. **Following** writes or follows its documents under `folder` of the assets: from the baselines of its build (`documents_follow`), and for a model whose source is here and whose game types the library of the project's editor gives, by importing it with that folder, which also writes its prefab @@ -364,9 +368,10 @@ machine that shares blobs with another finds what that one built. - `cook/cache` keeps each result as a blob of `mirai:store` named by the ref (`cache-`, `-`) (`asset.cache_namespace`). The store reaches it as a value - (`Import_Options.store`, `Sync_Options.store`, `Cook_Desc.store`; nil, none): the editor's - imports, `mirai import` (the machine's store, or `--store`) and the cook of a build open the - machine's (`store.default_root`). Tests make one in a temporary folder. + (`Import_Options.store`, `Cook_Desc.store`; nil, none): the editor's + imports (the session's: `--store`, else the machine's), `mirai import` (the machine's store, or + `--store`) and the cook of a build (the machine's, `store.default_root`). Tests make one in a + temporary folder. - A miss asks `Import_Options.cache_source` (a `cache.Source`) once before building: it makes the blob present in the store and sets the ref, or says it has none. - A texture that is not kept looks for its `textures.cache_key_build` (the bytes of the image, or diff --git a/asset/registry.odin b/asset/registry.odin index 859558b14..a7996cbec 100644 --- a/asset/registry.odin +++ b/asset/registry.odin @@ -184,6 +184,27 @@ registry_mount :: proc(r:^Registry,folder,prefix:string,listed:[]Package_Listed= return registry_mount_add(r,folder,prefix,false,listed) } +// Replaces the outputs of packages the next scan lists: those of the root +// ("") or of the mount of prefix. False without memory for them, or with no +// mount of prefix, which keep what they had. +registry_listed_set :: proc(r:^Registry,prefix:string,listed:[]Package_Listed) -> bool { + kept,ok:=registry_listed_clone(listed,r.allocator) + if !ok {return false} + if prefix=="" { + registry_listed_free(r.listed,r.allocator) + r.listed=kept + return true + } + for &mount in r.mounts { + if mount.prefix!=prefix {continue} + registry_listed_free(mount.listed,r.allocator) + mount.listed=kept + return true + } + registry_listed_free(kept,r.allocator) + return false +} + // A copy of entries whose strings live in allocator. False without memory for it. @(private="file") registry_listed_clone :: proc(entries:[]Package_Listed,allocator:runtime.Allocator) -> (kept:[]Package_Listed,ok:bool) { diff --git a/cook/import/sources/command.odin b/cook/import/sources/command.odin index 8384fd966..aed72afa2 100644 --- a/cook/import/sources/command.odin +++ b/cook/import/sources/command.odin @@ -216,7 +216,7 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in plugins:Library_Imports defer if plugins.release!=nil {plugins.release(plugins.state)} if (kind==.None || kind==.Model) && library!=nil && root!="" { - plugins=library(root,context.temp_allocator) + plugins=library(root,options.store,context.temp_allocator) all:=make([dynamic]importer.Format,0,len(FORMATS)+len(plugins.formats),context.temp_allocator) append(&all,..FORMATS[:]) append(&all,..plugins.formats) diff --git a/cook/import/sources/library.odin b/cook/import/sources/library.odin index 4b86b360f..3fd05c991 100644 --- a/cook/import/sources/library.odin +++ b/cook/import/sources/library.odin @@ -27,7 +27,8 @@ import "mirai:store" // ingest one origin at once take turns (a lock per origin), and the second // finds the work done. // - project_add: pins a package in the project's packages.mirai -// (project_pin, its only writer) and follows it into the project. +// (project_pin and project_unpin, its only writers) and follows it into +// the project. // - project_refresh: for each package a project pins, the build this engine // reads, from the local refs, else from the caller's source, else by // ingesting its origin when it is on this machine (also when the source @@ -43,10 +44,10 @@ import "mirai:store" // What the library of the editor of the project at root gives its imports: // the formats of models and the importers of whole sources its plugins add, // with what they read prepared for its assets, and the types of its game, -// which the prefab of a model is written in; release(state) lets them go. -// problem: why there are none. Only a program that links the editor reads it -// (editor.import_library). -Library :: #type proc(root:string,allocator:mem.Allocator) -> Library_Imports +// which the prefab of a model is written in, with the packages of root listed +// from the store s; release(state) lets them go. problem: why there are none. +// Only a program that links the editor reads it (editor.import_library). +Library :: #type proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports Library_Imports :: struct { formats:[]importer.Format, wholes:[]Whole_Source, @@ -71,6 +72,9 @@ Library_Options :: struct { // (plugins) and its game's types hold prefabs. "": the formats of the engine alone. root:string, plugins:Library, // nil: no formats of plugins, and no prefabs of models + // What the plugins give, already prepared by the caller, which releases + // it (the editor, from its session): taken in place of plugins. + imports:^Library_Imports, worker:jobs.Worker_Ref, progress:io.Writer, // where each source imported is said as it goes; nil: nowhere } @@ -206,7 +210,7 @@ project_refresh :: proc(root:string,options:Library_Options,allocator:=context.a } // Pins entry in the packages.mirai of the project at root, in place of the -// entry of the same package: the only writer of that file. +// entry of the same package: with project_unpin, the only writer of that file. project_pin :: proc(root:string,entry:asset.Project_Package) -> asset.Error { list,_,error:=asset.packages_load(root,context.temp_allocator) if error!=.None {return error} @@ -220,6 +224,18 @@ project_pin :: proc(root:string,entry:asset.Project_Package) -> asset.Error { return asset.packages_save(root,&list) } +// Removes the package id from the packages.mirai of the project at root; +// what the store keeps of it stays. False with nothing to remove. +project_unpin :: proc(root:string,id:base.Asset_Id) -> (removed:bool,error:asset.Error) { + list,_:=asset.packages_load(root,context.temp_allocator) or_return + for &other,at in list.packages { + if other.id!=id {continue} + ordered_remove(&list.packages,at) + return true,asset.packages_save(root,&list) + } + return false,.None +} + // What ingest, project_add and project_refresh share for one call. @(private="file") Library_State :: struct { @@ -253,8 +269,13 @@ library_end :: proc(l:^Library_State) { plugins_read :: proc(l:^Library_State) { if l.plugins_read {return} l.plugins_read=true - if l.options.plugins==nil || l.options.root=="" {return} - l.plugins=l.options.plugins(l.options.root,l.temp) + switch { + case l.options.imports!=nil: + l.plugins=l.options.imports^ + l.plugins.release=nil + case l.options.plugins==nil || l.options.root=="": return + case: l.plugins=l.options.plugins(l.options.root,l.options.store,l.temp) + } all:=make([dynamic]importer.Format,0,len(FORMATS)+len(l.plugins.formats),l.temp) append(&all,..FORMATS[:]) append(&all,..l.plugins.formats) diff --git a/cook/import/sources/reimport_test.odin b/cook/import/sources/reimport_test.odin index 2c664c852..620ec8051 100644 --- a/cook/import/sources/reimport_test.odin +++ b/cook/import/sources/reimport_test.odin @@ -369,7 +369,7 @@ a_refresh_reads_a_source_again_with_the_formats_of_the_plugins :: proc(t:^testin s:=test_store(t,fmt.tprintf("%s/store",root)) if s==nil {return} context.user_ptr=prefab.import_world(t) - probing :: proc(root:string,allocator:mem.Allocator) -> Library_Imports { + probing :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports { listed:=make([]importer.Format,1,allocator) listed[0]=probe_format() return {formats=listed,world=(^core.World_Info)(context.user_ptr)} @@ -449,7 +449,7 @@ a_refresh_ingests_again_a_whole_source_when_what_it_is_built_from_changes :: pro s:=test_store(t,fmt.tprintf("%s/store",root)) if s==nil {return} context.user_ptr=&state - leveling :: proc(root:string,allocator:mem.Allocator) -> Library_Imports { + leveling :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports { wholes:=make([]Whole_Source,1,allocator) wholes[0]={claims=level_claims,run=level_run,state=context.user_ptr,code=(^Level_Importer)(context.user_ptr).code} return {wholes=wholes} diff --git a/cook/import/sources/support_test.odin b/cook/import/sources/support_test.odin index 2d2e27d16..569a29f3b 100644 --- a/cook/import/sources/support_test.odin +++ b/cook/import/sources/support_test.odin @@ -38,7 +38,7 @@ write_text :: proc(folder,name,text:string) -> string { // A Library that gives no formats and the types of the game context.user_ptr // points to: what sync and mirai import write the prefab of a model in. -test_library :: proc(root:string,allocator:mem.Allocator) -> Library_Imports { +test_library :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> Library_Imports { return {world=(^core.World_Info)(context.user_ptr)} } diff --git a/docs/game/assets.md b/docs/game/assets.md index 461d419f9..78478cdbb 100644 --- a/docs/game/assets.md +++ b/docs/game/assets.md @@ -32,11 +32,11 @@ mirai import assets/player/step.wav # assets/player/step.sound - A model writes meshes, materials, textures, skeletons, clips and a prefab: instantiate that. - An image is color unless its name says otherwise (`normal`, `_n`, `rough`, `metal`, `orm`, `_ao`, `height`, `mask`); force it with `--srgb`, `--data`, `--normal`, `--normal-green-down`. -- `.import` beside the source keeps the flags, the folder of its assets and the id of each - asset, so importing again updates in place, keeping ids and material edits: commit it, and move - it with its source. A part renamed in the source is a new asset. -- Meshes, textures, sounds and fonts are built into `.mirai/imported/`, never committed: `mirai - build` makes them again on a fresh clone. Edit their source, not them. +- An import leaves the source where it is: its package in the store of the machine keeps the + flags and ids, so importing again updates in place, keeping ids and material edits. The project + pins it in `packages.mirai`: commit that. A part renamed in the source is a new asset. +- Meshes, textures, sounds and fonts are blobs of the store the project lists, never committed: + `mirai build` refreshes them on a fresh clone. Edit their source, not them. - A mesh keeps only the colliders its source brings; placed, it collides only with them. An `Auto` `Collider` without them is its triangles when still, convex pieces of them when it moves. - A model without a skeleton that must animate: `--rig` with the template that fits it. Clips of diff --git a/editor/api_assets.odin b/editor/api_assets.odin index ffff0bae0..6cff1da29 100644 --- a/editor/api_assets.odin +++ b/editor/api_assets.odin @@ -83,7 +83,7 @@ Api_Detach_Result :: struct { // What api_asset_delete (asset_delete.odin) writes. Api_Asset_Delete_Result :: struct { - moved:[]string `mirai:"label=The paths moved to the trash: the asset and its parts"`, + moved:[]string `mirai:"label=The paths that left the project: the asset and its parts, moved to the trash, and what the package it is the root of listed"`, used_by:[]string `mirai:"label=The files that use what moved; they keep the reference"`, error:string `mirai:"optional,label=Why the rest could not be moved after some were"`, } @@ -91,13 +91,16 @@ Api_Asset_Delete_Result :: struct { @(private) api_assets_register :: proc(s:^Session,st:^Api_State) { assets:=Api_Asset_Params{limit=256} + add_default:=Api_Packages_Add_Params{seconds=API_IMPORT_SECONDS} api_register(s,st,{name="component.list",summary="The components a scene can hold, with their defaults and their fields as scene.write takes them.",params=type_info_of(Api_Component_Params),result=type_info_of(Api_Component_List),flags={.Core,.Read_Only},run=api_component_list}) api_register(s,st,{name="asset.list",summary="The assets of the registry by kind and folder, with a count of its broken files, duplicated ids and missing dependencies.",params=type_info_of(Api_Asset_Params),result=type_info_of(Api_Asset_List),defaults=&assets,flags={.Core,.Read_Only},run=api_asset_list}) api_register(s,st,{name="asset.detach",summary="Takes a part of an import out of the asset that owns it so it shows on its own; the import still writes it and leaves it out of the owner.",params=type_info_of(Api_Detach_Params),result=type_info_of(Api_Detach_Result),run=api_asset_detach}) api_register(s,st,{name="asset.create",summary="Makes an empty asset in a folder of the assets as Create of the content browser does; material.set, particle.set and the like fill it.",params=type_info_of(Api_Asset_Create_Params),result=type_info_of(Api_Asset_Create_Result),run=api_asset_create}) api_register(s,st,{name="asset.users",summary="The files that use an asset or a part it owns, as the Delete modal of the content browser lists them.",params=type_info_of(Api_Asset_Users_Params),result=type_info_of(Api_Asset_Users_Result),flags={.Read_Only},run=api_asset_users}) - api_register(s,st,{name="asset.delete",summary="Moves an asset and the parts it owns to the trash of the system at once and without undo. Fails for files of plugins, the engine or the editor, open tabs and outside Edit.",params=type_info_of(Api_Asset_Delete_Params),result=type_info_of(Api_Asset_Delete_Result),flags={.Destructive},run=api_asset_delete}) - api_register(s,st,{name="asset.move",summary="Moves, renames or copies an asset with the parts it owns, or a folder with all it holds, at once and without undo; a move keeps ids and references, an asset its extension. A move fails for files of plugins, the engine or the editor, open tabs, files being edited, a folder into itself and outside Edit.",params=type_info_of(Api_Asset_Move_Params),result=type_info_of(Api_Asset_Moved_Result),run=api_asset_move}) + api_register(s,st,{name="asset.delete",summary="Moves an asset and the parts it owns to the trash of the system at once and without undo; the root of a package of the library also takes the package out of the project, and the store keeps it. Fails for another asset a package lists, files of plugins, the engine or the editor, open tabs and outside Edit.",params=type_info_of(Api_Asset_Delete_Params),result=type_info_of(Api_Asset_Delete_Result),flags={.Destructive},run=api_asset_delete}) + api_register(s,st,{name="asset.move",summary="Moves, renames or copies an asset with the parts it owns, or a folder with all it holds, at once and without undo; a move keeps ids and references, an asset its extension, and moving the root of a package of the library moves where the project lists the package. A move fails for another asset a package lists, files of plugins, the engine or the editor, open tabs, files being edited, a folder into itself and outside Edit.",params=type_info_of(Api_Asset_Move_Params),result=type_info_of(Api_Asset_Moved_Result),run=api_asset_move}) + api_register(s,st,{name="packages.list",summary="The packages of the library of this machine, what every import ingests into its store, with their assets: name, kind, bytes, whether built, and whether the project pins the package.",params=type_info_of(Api_Packages_List_Params),result=type_info_of(Api_Packages_List_Result),flags={.Read_Only},run=api_packages_list}) + api_register(s,st,{name="packages.add",summary="Pins a package of the library, or some of its assets, in the project, listed in a folder of the assets, as Add from library of the content browser does; its documents are written there. Answers as asset.import does when it ends.",params=type_info_of(Api_Packages_Add_Params),result=type_info_of(Import_Status_Result),defaults=&add_default,run=api_packages_add}) } // ---------------------------------------------------------------- component.list diff --git a/editor/asset_delete.odin b/editor/asset_delete.odin index 010461a33..48d286e33 100644 --- a/editor/asset_delete.odin +++ b/editor/asset_delete.odin @@ -5,6 +5,7 @@ import "core:path/filepath" import "core:slice" import "core:strings" import "mirai:asset" +import "mirai:cook/import/sources" import "mirai:core" import "mirai:editor/api" import "mirai:host" @@ -16,7 +17,11 @@ import "mirai:ui" // trash of the system, from which the user brings it back, at once and not as // an edit, as an import writes. The parts of an import go with the asset that // owns them; what uses it, saved or in an open document, is named before and -// keeps the reference. +// keeps the reference. The root of a package (library.odin) takes the package +// out of the packages.mirai of the project, which version control brings +// back: what it listed leaves the project, and its blobs stay in the store. A +// source of it inside the project goes to the trash with it; one outside +// stays. Another asset a package lists is refused. Api_Asset_Delete_Params :: struct { asset:[API_PATH_BYTES]u8 `mirai:"text,label=The asset to delete: its path under the asset folder or its id"`, @@ -46,19 +51,19 @@ asset_delete_files :: proc(r:^asset.Registry,id:base.Asset_Id) -> []u32 { return files[:] } -// As asset_delete_files, with what goes when the asset is the root of an -// import: what that import keeps in the imported folder, which nothing builds -// once its source and its sidecar are gone. +// As asset_delete_files, with the package the asset is the root of, which +// deleting it unpins, and what that package lists. @(private) -asset_delete_all :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (files:[]u32,sidecar:string) { - listed:=make([dynamic]u32,context.temp_allocator) - append(&listed,..asset_delete_files(r,id)) - cached:[]u32 - sidecar,cached=import_root(s,r,id) - for at in cached { - if !slice.contains(listed[:],at) {append(&listed,at)} +asset_delete_all :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (files:[]u32,pin:Library_Pin,unpins:bool) { + out:=make([dynamic]u32,context.temp_allocator) + append(&out,..asset_delete_files(r,id)) + file,found:=asset.registry_find(r,id) + if !found {return out[:],{},false} + if pin,unpins=library_pin_of(s,r,file);!unpins || library_package_root(&pin.p)!=id {return out[:],{},false} + for &a in pin.p.assets { + if at,known:=r.by_id[a.id];known && r.files[at].listed && !slice.contains(out[:],at) {append(&out,at)} } - return listed[:],sidecar + return out[:],pin,true } // What uses what deleting moves and is not moved with it, by path: the files @@ -79,7 +84,13 @@ asset_delete_users :: proc(s:^Session,r:^asset.Registry,files:[]u32) -> []string asset_delete_refusal :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> string { if refusal:=asset_owned_refusal(r,file);refusal!="" {return refusal} if file.listed { - if sidecar,_:=import_root(s,r,file.id);sidecar=="" {return "It is built by the import of its source, which makes it again: delete or move the asset that source gives (its prefab), change the source, or duplicate it to have a copy of its own"} + pin,found:=library_pin_of(s,r,file) + switch { + case !found: + return "It is listed from a package the project pins and the store at hand does not have: refresh the project, or duplicate it to have a copy of its own" + case library_package_root(&pin.p)!=file.id: + return fmt.tprintf("It is listed from the package %s: delete the asset that package gives, %s, which takes the package out of the project, or duplicate it to have a copy of its own",pin.p.name,library_root_named(r,&pin.p)) + } } if watching(s) {return "The run of an agent is being watched; stop watching it first"} if busy:=import_busy(s);busy!="" {return busy} @@ -106,23 +117,34 @@ asset_delete :: proc(s:^Session,id:base.Asset_Id) -> (moved:[]string,message:str file,found:=asset.registry_find(r,id) if !found {return nil,"The asset is not in the registry"} if refusal:=asset_delete_refusal(s,r,file);refusal!="" {return nil,refusal} - files,sidecar:=asset_delete_all(s,r,id) + files,pin,unpins:=asset_delete_all(s,r,id) paths:=make([]string,len(files),context.temp_allocator) - for at,i in files {paths[i]=strings.clone(r.files[at].path,context.temp_allocator)} + listed:=make([]bool,len(files),context.temp_allocator) + for at,i in files {paths[i],listed[i]=strings.clone(r.files[at].path,context.temp_allocator),r.files[at].listed} // What the registry holds of them goes first, or it would keep answering for them. for path in paths {_,_=asset.registry_memory_drop(r,path)} done:=make([dynamic]string,context.temp_allocator) - for path in paths { + for path,i in paths { + // What a package lists is a blob of the store, which stays: it leaves with the package. + if listed[i] {continue} if !platform.trash(asset.registry_file_path(r,path,context.temp_allocator)) { message=fmt.tprintf("%s could not be moved to the trash",path) break } append(&done,path) } - // The import is given up with its root: its source stays, and importing it again is a new import. - if message=="" && sidecar!="" { - _=platform.trash(sidecar) - _=platform.trash(strings.trim_suffix(sidecar,asset.SIDECAR_EXTENSION)) + // The package is given up with its root: importing its source again is a new package. + if message=="" && unpins { + if _,error:=sources.project_unpin(pin.root.root,pin.entry.id);error!=.None { + message=fmt.tprintf("%s/%s could not be written (%v): check that it is not read-only, then delete again",pin.root.root,asset.PACKAGES_FILE,error) + } else { + for path,i in paths { + if listed[i] {append(&done,path)} + } + origin:=pin.p.origin.path + if origin!="" && asset_path_inside(origin,pin.root.root) {_=platform.trash(origin)} + library_relist(s,pin.root) + } } if len(done)>0 { host.assets_refresh(s.app.assets,done[:]) @@ -153,7 +175,7 @@ asset_delete_modal :: proc(s:^Session) { closed:=false defer if closed {g.modal,g.delete_asset=.None,{}} if ui.modal(ctx,shell_id(g,"delete_asset"),{width=420,padding=16,gap=14,close=&closed}) { - files,_:=asset_delete_all(s,r,file.id) + files,_,_:=asset_delete_all(s,r,file.id) users:=asset_delete_users(s,r,files) refusal:=asset_delete_refusal(s,r,file) ui.label(ctx,"Delete asset",ui.metric(ctx,"font_heading"),ui.tone(ctx,"text"),font=ui.FONT_SEMIBOLD) diff --git a/editor/asset_move.odin b/editor/asset_move.odin index a0fa5ea24..4a8f37500 100644 --- a/editor/asset_move.odin +++ b/editor/asset_move.odin @@ -7,6 +7,7 @@ import "core:os" import "core:path/filepath" import "core:strings" import "mirai:asset" +import "mirai:cook/import/sources" import "mirai:editor/api" import "mirai:editor/history" import "mirai:host" @@ -20,9 +21,12 @@ import "mirai:std/serial" // asset names its id, which moving keeps: references, Recent, Favorites and // open documents follow it. An asset moves with the parts it owns, which keep // where they are beside it; an import keeps its source, which Import_Info -// names from the project. What is being edited does not move until its step -// ends. A duplicate is a new asset with ids of its own, and no -// output of an import. +// names from its package. Moving the root of a package of the library +// (library.odin) moves where the project lists the package, and what it +// lists goes there; a folder takes the packages listed in it. Another asset +// a package lists does not move on its own. What is being edited does not +// move until its step ends. A duplicate is a new asset with ids of its own, +// and no output of an import. Api_Asset_Move_Params :: struct { asset:[API_PATH_BYTES]u8 `mirai:"text,label=The asset or folder: its path under the asset folder or the id of an asset"`, @@ -83,7 +87,6 @@ asset_held_refusal :: proc(r:^asset.Registry,paths:[]string,folder:="") -> strin // Why an asset cannot be renamed or moved now; "" when it can. @(private) asset_move_refusal :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> string { - if file.listed {return "It is a part of what its source gives, listed in the folder of its source: move that folder, or duplicate it to have a copy that moves"} if refusal:=asset_create_refusal(s);refusal!="" {return refusal} if refusal:=asset_delete_refusal(s,r,file);refusal!="" {return refusal} files:=asset_delete_files(r,file.id) @@ -101,25 +104,20 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess if r==nil {return "","The editor has no asset folder it can change"} file,found:=asset.registry_find(r,id) if !found {return "","The asset is not in the registry: asset.list lists them"} - // What a source gives is listed where the source is: its source moves, with its sidecar, and nothing is imported. - if sidecar,_:=import_root(s,r,id);file.listed && sidecar!="" { - into,refused:=asset_folder_clean(r,folder) - if refused!="" {return "",fmt.tprintf("folder: %s",refused)} - if busy:=import_busy(s);busy!="" {return "",busy} - source:=strings.trim_suffix(sidecar,asset.SIDECAR_EXTENSION) - target:=fmt.tprintf("%s/%s",asset.registry_file_path(r,into,context.temp_allocator),filepath.base(source)) - if dir:=filepath.dir(target);!os.is_dir(dir) {_=os.make_directory_all(dir)} - if os.exists(target) || os.rename(source,target)!=nil || os.rename(sidecar,fmt.tprintf("%s%s",target,asset.SIDECAR_EXTENSION))!=nil { - return "",fmt.tprintf("%s could not be moved to %s: a file is there, or another program has it open",source,target) - } - host.assets_scan(s.app.assets) - if moved,listed:=asset.registry_find(r,id);listed {path=strings.clone(moved.path,context.temp_allocator)} - return - } + if file.listed {return asset_move_listed(s,r,file,folder,name)} if refusal:=asset_move_refusal(s,r,file);refusal!="" {return "",refusal} + pin,pinned:=library_pin_of(s,r,file) + pinned&&=library_package_root(&pin.p)==id into,refused:=asset_folder_clean(r,folder) if refused!="" {return "",fmt.tprintf("folder: %s",refused)} into=strings.clone(into,context.temp_allocator) + // The listing of its package goes where it goes, in the root it is pinned in. + listing:string + if pinned { + root,rest,rooted:=import_root_of(s,r,into) + if !rooted || root.root!=pin.root.root {return "",fmt.tprintf("folder: the package %s is pinned in %s: give a folder of its assets",pin.p.name,pin.root.root)} + listing=strings.clone(rest,context.temp_allocator) + } extension:=asset_path_extension(file.path) named:=strings.trim_space(name) if named=="" {named=asset_name(file.path)} @@ -128,7 +126,7 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess source:=content_folder_of(file.path) moves:=make([dynamic]Asset_Move,context.temp_allocator) for at,i in asset_delete_files(r,id) { - // A part of the imported folder stays at the path its sidecar lists. + // A part a package lists goes with the listing of its package. if i>0 && r.files[at].listed {continue} from:=strings.clone(r.files[at].path,context.temp_allocator) to:=asset_create_join(into,named,extension) @@ -141,7 +139,7 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess } append(&moves,Asset_Move{from,to}) } - if len(moves)==0 {return} + if len(moves)==0 && !(pinned && listing!=pin.entry.folder) {return} done:=0 for move in moves { from:=asset.registry_file_path(r,move.from,context.temp_allocator) @@ -164,6 +162,11 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess asset_history_moved(s,asset.registry_file_path(r,move.from,context.temp_allocator),asset.registry_file_path(r,move.to,context.temp_allocator),false) asset_start_scene_moved(s,move.from,move.to,false) } + if pinned && listing!=pin.entry.folder { + entry:=pin.entry + entry.folder=listing + if problem:=library_repin(s,pin.root,entry);problem!="" {message=problem} + } host.assets_scan(s.app.assets) if c:=s.content;c!=nil { c.reveal=id @@ -172,6 +175,64 @@ asset_move :: proc(s:^Session,id:base.Asset_Id,folder,name:string) -> (path,mess return } +// Moves what a package lists: its root moves where the project lists the +// package, keeping the name its source gives it; any other is refused. +@(private) +asset_move_listed :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File,folder,name:string) -> (path,message:string) { + id:=file.id + pin,found:=library_pin_of(s,r,file) + if !found {return "","It is listed from a package the project pins and the store at hand does not have: refresh the project, or duplicate it to have a copy that moves"} + if library_package_root(&pin.p)!=id {return "",fmt.tprintf("It is listed from the package %s: move the asset that package gives, %s, which moves where the project lists the package, or duplicate it to have a copy that moves",pin.p.name,library_root_named(r,&pin.p))} + if refusal:=asset_create_refusal(s);refusal!="" {return "",refusal} + into,refused:=asset_folder_clean(r,folder) + if refused!="" {return "",fmt.tprintf("folder: %s",refused)} + extension:=asset_path_extension(file.path) + named:=strings.trim_space(name) + if extension!="" && strings.has_suffix(named,extension) {named=named[:len(named)-len(extension)]} + if named!="" && named!=asset_name(file.path) {return "",fmt.tprintf("name: %s keeps the name its source gives it: move it without a name, or duplicate it to name a copy",file.path)} + root,rest,rooted:=import_root_of(s,r,into) + if !rooted || root.root!=pin.root.root {return "",fmt.tprintf("folder: the package %s is pinned in %s: give a folder of its assets",pin.p.name,pin.root.root)} + entry:=pin.entry + entry.folder=rest + if entry.folder!=pin.entry.folder { + if problem:=library_repin(s,pin.root,entry);problem!="" {return "",problem} + } + moved,there:=asset.registry_find(r,id) + if !there {return "",fmt.tprintf("The package %s was listed in %s and its asset is not there: refresh the project",pin.p.name,into)} + path=strings.clone(moved.path,context.temp_allocator) + if c:=s.content;c!=nil { + c.reveal=id + content_place_set(c,into) + } + return +} + +// The entries of the packages root pins that are listed in the folder from +// of its assets or under it, each with the folder it is listed in once from +// is to; in the temp allocator. +@(private) +asset_folder_pins :: proc(root:Library_Root,from,to:string) -> []asset.Project_Package { + list,_,error:=asset.packages_load(root.root,context.temp_allocator) + if error!=.None {return nil} + out:=make([dynamic]asset.Project_Package,context.temp_allocator) + for entry in list.packages { + if entry.folder!=from && !asset_path_inside(entry.folder,from) {continue} + moved:=entry + moved.folder=strings.concatenate({to,entry.folder[len(from):]},context.temp_allocator) + append(&out,moved) + } + return out[:] +} + +// Whether the folder of the registry holds what a package lists. +@(private) +asset_folder_listed :: proc(r:^asset.Registry,folder:string) -> bool { + for &file in r.files { + if file.listed && asset_path_inside(file.path,folder) {return true} + } + return false +} + // Moves the folder from, with all it holds, to folder under name ("" keeps // the one it has). Returns where it is, or what failed as how to correct it. asset_folder_move :: proc(s:^Session,from,folder,name:string) -> (path,message:string) { @@ -190,15 +251,26 @@ asset_folder_move :: proc(s:^Session,from,folder,name:string) -> (path,message:s for &file in r.files { if file.state==.Asset && asset_path_inside(file.path,source) && scene_tab(s,file.id)>=0 {return "",fmt.tprintf("%s is open in a tab: close the tab first",file.path)} } + // The packages listed in it go with it, in the root they are pinned in. + root,from_rest,_:=import_root_of(s,r,source) + to_root,to_rest,rooted:=import_root_of(s,r,path) + pins:=asset_folder_pins(root,from_rest,to_rest) + if len(pins)>0 && (!rooted || to_root.root!=root.root) {return "",fmt.tprintf("%s lists packages pinned in %s: give a folder of its assets",source,root.root)} full:=asset.registry_file_path(r,source,context.temp_allocator) target:=asset.registry_file_path(r,path,context.temp_allocator) - // The sources of the folder go with it, with their sidecars: what they give is listed where they are, and nothing is imported. if os.is_dir(full) { if dir:=filepath.dir(target);!os.is_dir(dir) {_=os.make_directory_all(dir)} if os.rename(full,target)!=nil {return "",fmt.tprintf("%s could not be moved to %s; another program may have a file of it open",source,path)} } asset_history_moved(s,full,target,true) asset_start_scene_moved(s,source,path,true) + for entry in pins { + if error:=sources.project_pin(root.root,entry);error!=.None { + message=fmt.tprintf("%s/%s could not be written (%v): the folder moved and some of its packages are listed where it was; check that the file is not read-only and move them",root.root,asset.PACKAGES_FILE,error) + break + } + } + if len(pins)>0 {library_relist(s,root)} host.assets_scan(s.app.assets) if c:=s.content;c!=nil {content_place_set(c,path)} return @@ -321,7 +393,7 @@ asset_folder_subject :: proc(s:^Session,from:string) -> (r:^asset.Registry,folde switch { case refusal!="": return case folder=="": refusal="The assets folder itself stays: give one of its folders" - case !os.is_dir(asset.registry_file_path(r,folder,context.temp_allocator)): refusal=fmt.tprintf("%s is not a folder of the assets: asset.list lists what they hold",folder) + case !os.is_dir(asset.registry_file_path(r,folder,context.temp_allocator)) && !asset_folder_listed(r,folder): refusal=fmt.tprintf("%s is not a folder of the assets: asset.list lists what they hold",folder) } return } @@ -422,7 +494,7 @@ api_asset_subject :: proc(call:^api.Call,r:^asset.Registry,name:string) -> (id:b } if file,found:=asset.registry_find_path(r,name);found && file.state==.Asset {return file.id,"",true} clean,refusal:=asset_folder_clean(r,name) - if refusal=="" && clean!="" && os.is_dir(asset.registry_file_path(r,clean,context.temp_allocator)) {return {},clean,true} + if refusal=="" && clean!="" && (os.is_dir(asset.registry_file_path(r,clean,context.temp_allocator)) || asset_folder_listed(r,clean)) {return {},clean,true} api.call_failf(call,api.INVALID_PARAMS,"%q is no asset nor folder: give a path under the asset folder or the id of an asset as asset.list lists them",name) return } diff --git a/editor/assets_parts.odin b/editor/assets_parts.odin index b940fe111..df5af1a5b 100644 --- a/editor/assets_parts.odin +++ b/editor/assets_parts.odin @@ -30,7 +30,7 @@ asset_detach :: proc(s:^Session,id:base.Asset_Id) -> (owner:base.Asset_Id,messag if !found {return {},fmt.tprintf("No asset has the id %s; asset.list lists them",base.asset_id_format(id,buffer[:]))} path:=file.path if file.owner=={} {return {},fmt.tprintf("%s is a part of no asset: it already is one of its own",path)} - if file.listed {return {},fmt.tprintf("%s is built by the import of its source, whose sidecar says it is a part: import the source with parts none, or duplicate the asset to have one of its own",path)} + if file.listed {return {},fmt.tprintf("%s is listed from a package whose manifest says it is a part: import its source again with parts none, or duplicate the asset to have one of its own",path)} owner=file.owner bytes,error:=asset.registry_read(r,path,context.temp_allocator) if error==.None {error=asset.header_owner_set(bytes,{})} diff --git a/editor/content.odin b/editor/content.odin index 4057b80a2..82d382be6 100644 --- a/editor/content.odin +++ b/editor/content.odin @@ -1687,6 +1687,7 @@ content_settings_build :: proc(s:^Session,content_view:bool) { if asset.registry_scan(&s.app.assets.registry)!=.None {notify(s,"The asset folder could not be read")} } command_menu_item(s,ui.id(ctx,"import"),"import","Import") + if ui.submenu(ctx,ui.id(ctx,"library"),"Add from library",{icon=.Import,disabled=s.store.root=="" || s.mode!=.Edit}) {content_library_items(s)} ui.id_pop(ctx) } } diff --git a/editor/editor.odin b/editor/editor.odin index 18377d941..ee7793071 100644 --- a/editor/editor.odin +++ b/editor/editor.odin @@ -20,6 +20,7 @@ import "mirai:project" import "mirai:std/base" import "mirai:std/jobs" import "mirai:std/subprocess" +import "mirai:store" import "mirai:ui" // The editor of a project: the application with its window, the World that @@ -95,7 +96,7 @@ Run :: struct { @(private) USAGE_TEXT :: `Usage: [--project ] [--scene ] [--headless] [--hidden] [--no-gpu] [--frames ] [--capture ] [--validation] [--width ] [--height ] [--density ] [--engine ] [--api-port ] [--no-api] - [--share [port]] [--join ] [--name ] [--preferences] + [--share [port]] [--join ] [--name ] [--preferences] [--store ] --api-port port of the agent API (JSON-RPC and MCP); 7940 if missing, or a free one when it is taken; 0: a free one, never 7940 --no-api without the agent API --share shares the scene with other editors; port 7950 if missing. Prints the invitation code @@ -110,11 +111,13 @@ USAGE_TEXT :: `Usage: [--project ] [--scene ] [--headless] [--hidd --frames ends after n frames --capture writes the last frame as a PNG at the end --validation turns on the validation of the graphics API - --engine engine root for Build; if missing, MIRAI_ROOT or the engine above the executable or the working folder` + --engine engine root for Build; if missing, MIRAI_ROOT or the engine above the executable or the working folder + --store the store of the packages it imports and lists; the machine's if missing` // The arguments the editor was started with. Options :: struct { project,scene,engine,capture:string, + store:string, // the store of the library; "": the machine's documents:string, // the tabs a restart left (documents_restore) gpu_lost:string, // what stopped the GPU of the editor whose restart this is built:bool, // a Build of the editor whose restart this is succeeded: its library is the one loaded @@ -178,6 +181,9 @@ Session :: struct { engine_stamp:string, // of the binaries of the engine it runs (build.engine_stamp_of): a build of the engine afterwards changes the one beside them library_stamp:string, // of the library of the project it loaded last (build.target_stamp_of) assets_root:string, + // The machine store the library keeps the packages of imports in (--store, + // else the machine's); its root is "" when the machine has none. + store:store.Store, has_project:bool, // the folder has a project.mirai settings_unsaved:bool, // a setting changed that the file of its layer does not have yet settings_layer:project.Settings_Layer, // the one the settings dialog edits @@ -414,6 +420,7 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { if title:=project.window_title(&p.manifest);title!="" {window.title=strings.clone(title,context.temp_allocator)} } mounts:=[1]asset.Registry_Mount{{folder=project.path(engine,EDITOR_ASSETS,allocator=context.temp_allocator),prefix=EDITOR_ASSETS_PREFIX}} + store_root:=options.store!=""?project.path_absolute(options.store,context.temp_allocator):store.default_root(context.temp_allocator) windowed:=!options.headless app,app_error:=host.app_create({ info=info,root=root,engine_root=engine,threads=jobs.THREADS_AUTO, @@ -424,7 +431,7 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { assets_root=os.is_dir(assets)?assets:"",assets_cache=project.cache_path(root,project.REGISTRY_CACHE,allocator=context.temp_allocator), animation_cache=project.cache_path(root,project.ANIMATION_CACHE,allocator=context.temp_allocator), collision_cache=project.content_cache_path(root,project.COLLISION_CACHE,allocator=context.temp_allocator), - asset_mounts=mounts[:],shaders_root=root, + asset_mounts=mounts[:],shaders_root=root,store_root=store_root, }) if app==nil { fmt.eprintfln("mirai: the editor could not start: %v",app_error) @@ -449,6 +456,9 @@ open :: proc(info:^core.World_Info,args:[]string) -> (s:^Session,code:int) { s.engine_stamp=engine_stamp_here(s) s.library_stamp=library_stamp_here(s) s.assets_root=strings.clone(assets) + if store_root!="" { + if opened,error:=store.open(store_root);error==.None {s.store=opened} else {log.warnf("the store at %s does not open (%v): nothing imports until the editor starts with --store ",store_root,error)} + } s.has_project=has_project if has_project {s.build.sources=code_fingerprint(root)} s.headless=options.headless @@ -1169,6 +1179,8 @@ session_destroy :: proc(s:^Session) { s.recipes=nil console_destroy(s.console) settings_engine_free(s) + if s.store.root!="" {store.close(&s.store)} + s.store={} delete(s.project_root);delete(s.engine_root);delete(s.program);delete(s.engine_stamp);delete(s.library_stamp);delete(s.assets_root);delete(s.capture);delete(s.gpu_lost_before) for arg in s.args {delete(arg)} delete(s.args) @@ -1192,6 +1204,7 @@ options_read :: proc(args:[]string) -> (o:Options,ok:bool) { case "--project": o.project=value(args,&i) or_return case "--scene": o.scene=value(args,&i) or_return case "--engine": o.engine=value(args,&i) or_return + case "--store": o.store=value(args,&i) or_return case "--capture": o.capture=value(args,&i) or_return case "--documents": o.documents=value(args,&i) or_return case "--gpu-lost": o.gpu_lost=value(args,&i) or_return @@ -1246,7 +1259,7 @@ options_passed :: proc(args:[]string) -> []string { for i:=0;i.import) keeps its settings and the id of each -// asset, and importing it again gives the same assets; what an import builds -// that nobody edits is not a file of the asset folder but of the imported -// folder of the project, which mirai build brings up to date with the -// sources (sources.sync). The Import button of the Assets tab and File > -// Import ask for the file; a file dropped on the window is imported the same -// way. +// Import from the content browser: a source file or folder anywhere on disk +// is ingested into the library of the machine where it is, nothing copied, +// and its package is pinned in the project (sources.project_ingest), as mirai +// import does, with the formats and whole sources of plugins too +// (Import_Format_Provider, Import_Source_Provider). Its package is listed in +// the folder of the browser, or for a source in the assets, in its own +// folder: its meshes, textures, sounds and fonts are blobs of the store the +// registry lists there, and its documents (materials, skeletons, clips, the +// prefab of a model) are files of that folder. Importing it again gives the +// same assets. The Import button of the Assets tab and File > Import ask for +// the file; a file dropped on the window is imported the same way, and Add +// from library pins a package the library has (library.odin). // // The import runs in a thread of its own, with its own job system, while the // editor goes on; the content browser shows the dot of what works. When it -// ends, between frames, the registry looks again at what it wrote, what was -// loaded from the files it replaced is read again and, when it wrote Surface -// programs, the programs of the project are published, so its materials -// show. One import runs at a time, and while it does nothing else writes -// assets (import_busy). +// ends, between frames, the registry lists the packages again and looks at +// what it wrote, what was loaded of it is read again and, when it wrote +// Surface programs, the programs of the project are published, so its +// materials show. One import runs at a time, and while it does nothing else +// writes assets (import_busy). It is no edit: undo does not take it back. DIALOG_IMPORT :: u32(0x4d494d50) DIALOG_AI_REFERENCE :: u32(0x4d414952) @@ -65,27 +60,34 @@ Importer :: struct { zones:[dynamic]importer.Zone_Total, // of the last one that ended, when it was profiled publishing:time.Tick, // since the last one asked for its programs to be published, while they are; zero otherwise profiled:bool, // the last one was profiled: the publication it asked for is a zone too + library:[dynamic]Library_Shown, // what Add from library shows, read when it opens + library_frame:u64, // the frame it was last shown in } -// An import in a thread of its own. Its memory is that of allocator, which +// An import in a thread of its own: of a source (source), or the add of a +// package of the library (manifest). Its memory is that of allocator, which // any thread can call. Import_Job :: struct { thread:^thread.Thread, - source:string, - kind:sources.Source_Kind, - formats:[]importer.Format, // of a model: the engine's, then plugin_formats' + source:string, // the source, or the name of the package added + root,folder:string, // the project or plugin it is pinned in, and the folder of its assets it is listed in + prefix:string, // of that root in the registry: "" for the project + manifest:store.Hash, // of the package added; zero for a source + assets:[]base.Asset_Id, // of the package added, those it uses; none: all + settings:importer.Import_Settings, + settled:bool, // settings replace those its package keeps + claimed:bool, // a plugin imports the source whole + imports:sources.Library_Imports, // what the plugins give, prepared on the thread of the editor plugin_formats:[]Import_Format_Provider, // with the format.data their prepare gave - whole:Source_Importer, // of a source a plugin imports whole; run is nil otherwise - state:rawptr, // what whole.prepare gave - options:importer.Import_Options, - store:store.Store, // the machine store, whose caches options.store names; unopened: none + wholes:[]Source_Importer, // with the state their prepare gave, in imports.wholes + store:store.Store, // the session's + options:sources.Library_Options, allocator:mem.Allocator, started:time.Tick, profiled:bool, profile:base.Profile, // its zones, when profiled // What the thread leaves once done is set. report:importer.Import_Report, - scene:string, error:importer.Error, seconds:f64, done:bool, @@ -130,19 +132,30 @@ import_place :: proc(s:^Session) -> string { return s.content!=nil?string(s.content.place[:s.content.place_length]):"" } -// Starts importing a source file; false, with the reason shown, when it -// cannot. A source in the assets of the project or of a plugin imports where -// it is; one from outside the project is brought first into folder, a folder -// of the assets (asset_folder_clean; none is the root), and imports there: -// it is versioned with its sidecar beside what it makes, and its meshes, -// textures, sounds and fonts go to the imported folder. A source a plugin -// imports whole is brought by its importer with the files it reads -// (Source_Importer.bring), any other is copied alone. A source imported -// before writes the assets its sidecar lists, wherever they are. profiled -// records the zones of the import (importer.profile_totals) for -// asset.import_status. +// The root of the registry folder is of, the project or a plugin, and +// where the folder is under its assets; false for a folder of the editor or +// of the engine, which pins nothing. +@(private) +import_root_of :: proc(s:^Session,r:^asset.Registry,folder:string) -> (root:Library_Root,rest:string,ok:bool) { + for mount in r.mounts { + if folder!=mount.prefix && !asset_path_inside(folder,mount.prefix) {continue} + if strings.has_prefix(mount.prefix,".") || filepath.base(mount.folder)!=project.FOLDER_NAMES[.Assets] {return} + rest=folder[min(len(mount.prefix)+1,len(folder)):] + return {filepath.dir(mount.folder),mount.prefix},rest,true + } + if s.project_root=="" {return} + return {s.project_root,""},folder,true +} + +// Starts importing a source file or folder from anywhere on disk; false, +// with the reason shown, when it cannot. It is ingested into the library +// where it is and its package is pinned in the project, or the plugin whose +// folder folder is, listed in folder, a folder of the assets +// (asset_folder_clean; none is the root, or for a source in the assets, its +// own folder). profiled records the zones of the import +// (importer.profile_totals) for asset.import_status. // rig, when given, has the rig, rig_crotch and rig_neck that replace the -// source's settings, which a model inside the project keeps for its next import. +// settings its package keeps for its next import. // image, when given, is the use of an image that replaces its settings the same // way; without it, its settings say, else its name (sources.image_use). // skeleton, when not zero, is the skeleton of the project the clips of a model @@ -158,130 +171,117 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb import_say(s,"%s",busy) return false } - name:=filepath.base(source) - source:=source - // A source a plugin imports whole goes before the formats, which may read its bytes too. - whole,code,claimed:=import_whole(s,source) - if _,in_assets:=sources.source_root(source,context.temp_allocator);!in_assets { - for at:=filepath.dir(project.path_absolute(source,context.temp_allocator));filepath.dir(at)!=at;at=filepath.dir(at) { - if !project.path_same(at,s.project_root) {continue} - import_say(s,"%s is in the project but in no assets folder: move it under assets/, beside what it makes, and import it there",name) - return false - } - into:=asset.registry_file_path(&a.registry,folder,context.temp_allocator) - if claimed && whole.bring!=nil { - brought,bring_error:=whole.bring(source,into,context.temp_allocator) - if bring_error.code!=.None { - import_say(s,"%s: %s",name,bring_error.message) - return false - } - if _,inside:=sources.source_root(brought,context.temp_allocator);!inside { - import_say(s,"%s: %s brought it to %s, which is in no assets folder",name,whole.name,brought) - return false - } - source=brought - } else { - copied:=fmt.tprintf("%s/%s",into,name) - if os.exists(copied) { - import_say(s,"%s is already in %s: import that one, or rename the file to import",name,folder==""?"the assets":folder) - return false - } - if os.copy_file(copied,source)!=nil { - import_say(s,"%s could not be copied into %s",name,folder==""?"the assets":folder) - return false - } - source=copied - } + if s.store.root=="" { + import_say(s,"The editor has no store to keep imports in: start it with --store ") + return false } - root,_:=sources.source_root(source,context.temp_allocator) - allocator:=runtime.heap_allocator() - options:=importer.Import_Options{ - root=strings.clone(root,allocator), - world=s.info, - fast_textures=true, - } - formats:=import_formats(s) - kind:=sources.Source_Kind.None if claimed else sources.source_kind(source,formats) - switch kind { - case .None: - case .Model: - _,rigged:=rig.? - _,faced:=front.? - if rigged || faced || skeleton!={} { - stored,_,settings_error:=importer.settings_load(options,source) - if settings_error.code!=.None { - import_say(s,"%s: %s",name,settings_error.message) - return false - } - options.settings,options.ignore_stored=stored,true - if wanted,rigged:=rig.?;rigged {options.rig,options.rig_crotch,options.rig_neck=wanted.rig,wanted.rig_crotch,wanted.rig_neck} - if skeleton!={} {options.skeleton=skeleton} - if stated,given:=front.?;given {options.front=stated} - } - case .Image: - options.image=sources.image_use(source) - if use,given:=image.?;given { - stored,_,settings_error:=importer.settings_load(options,source) - if settings_error.code!=.None { - import_say(s,"%s: %s",name,settings_error.message) - return false - } - options.settings,options.ignore_stored=stored,true - options.image=use - } - case .Sound,.Font,.Prompts: + if importer.unsafe_path(source) || !os.exists(source) { + import_say(s,"%s does not exist: give the path of a file or a folder on disk",source) + return false } - job:=new(Import_Job,allocator) - job^={kind=kind,options=options,allocator=allocator,started=time.tick_now(),profiled=profiled} - if store_root:=store.default_root(context.temp_allocator);store_root!="" { - if opened,error:=store.open(store_root,allocator);error==.None { - job.store=opened - job.options.store=&job.store - } + origin:=importer.path_key(source) + name:=filepath.base(origin) + place:=folder + if place=="" { + if relative,inside:=asset.registry_relative(&a.registry,origin);inside {place=content_folder_of(relative)} } - job.source=strings.clone(source,allocator) - if kind==.Model {import_formats_prepare(s,job)} - if kind==.None && !claimed { - import_job_free(job) - import_say(s,"%s is not a file the editor imports",name) + root,rest,rooted:=import_root_of(s,&a.registry,place) + if !rooted { + import_say(s,"%s is a folder of the editor or the engine: import into a folder of the project or of a plugin",place) return false } - if _,given:=rig.?;given && kind!=.Model { - import_job_free(job) + // A source a plugin imports whole goes before the formats, which may read its bytes too. + whole,code,claimed:=import_whole(s,origin) + kind:=sources.Source_Kind.None if claimed else sources.source_kind(origin,import_formats(s)) + switch { + case kind==.None && !claimed: + import_say(s,"%s is not a file the editor imports",name) + return false + case rig!=nil && kind!=.Model: import_say(s,"%s is not a model: only a model takes a rig",name) return false - } - if skeleton!={} && kind!=.Model { - import_job_free(job) + case skeleton!={} && kind!=.Model: import_say(s,"%s is not a model: only the clips of a model go on a skeleton",name) return false - } - if _,given:=front.?;given && kind!=.Model { - import_job_free(job) + case front!=nil && kind!=.Model: import_say(s,"%s is not a model: only a model has a front",name) return false - } - if _,given:=image.?;given && kind!=.Image { - import_job_free(job) + case image!=nil && kind!=.Image: import_say(s,"%s is not an image: only an image imported on its own takes a use; the textures of a model take theirs from its materials",name) return false } - if claimed { - job.whole=whole - job.options.importer_code=code - if whole.prepare!=nil {job.state=whole.prepare(&a.registry,s.info,allocator)} + allocator:=runtime.heap_allocator() + job:=new(Import_Job,allocator) + job^={allocator=allocator,started=time.tick_now(),profiled=profiled,claimed=claimed} + job.store=s.store + job.source=strings.clone(origin,allocator) + job.root,job.prefix,job.folder=strings.clone(root.root,allocator),strings.clone(root.prefix,allocator),strings.clone(rest,allocator) + // The settings its package keeps, changed by what was asked: given only when they change it. + if kind==.Model || kind==.Image { + stored,found,settings_error:=importer.settings_load({store=&job.store},origin) + if settings_error.code!=.None { + import_job_free(job) + import_say(s,"%s: %s",name,settings_error.message) + return false + } + chosen:=stored + // An image with no settings yet holds what its name says. + if kind==.Image && !found {chosen.image=sources.image_use(origin)} + if use,given:=image.?;given {chosen.image=use} + if wanted,given:=rig.?;given {chosen.rig,chosen.rig_crotch,chosen.rig_neck=wanted.rig,wanted.rig_crotch,wanted.rig_neck} + if skeleton!={} {chosen.skeleton=skeleton} + if stated,given:=front.?;given {chosen.front=stated} + job.settings=chosen + job.settled=chosen!=stored || !found + } + import_plugins_prepare(s,job,kind==.Model,whole if claimed else {},code) + import_job_start(s,job,"Importing %s",name) + return job.thread!=nil +} + +// The formats and the whole importers of the plugins job imports with, with +// what their prepare gives from the library: models takes the formats, +// whole the one importer that claims its source, with the identity of the +// code of its plugin. +@(private) +import_plugins_prepare :: proc(s:^Session,job:^Import_Job,models:bool,whole:Source_Importer,code:u64) { + job.imports.world=s.info + if models { + found:[PROVIDER_CAPACITY]Import_Format_Provider + plugins:=found[:provided_all(s,found[:])] + job.plugin_formats=slice.clone(plugins,job.allocator) + formats:=make([]importer.Format,len(plugins),job.allocator) + for &provider,i in job.plugin_formats { + if provider.prepare!=nil {provider.format.data=provider.prepare(&job.store,job.root,s.info,job.allocator)} + formats[i]=provider.format + } + job.imports.formats=formats + } + if whole.run!=nil { + job.wholes=make([]Source_Importer,1,job.allocator) + job.wholes[0]=whole + wholes:=make([]sources.Whole_Source,1,job.allocator) + wholes[0]={claims=whole.claims,run=whole.run,code=code} + if whole.prepare!=nil {wholes[0].state=whole.prepare(&job.store,job.root,s.info,job.allocator)} + job.imports.wholes=wholes } + job.options=sources.Library_Options{store=&job.store,fast_textures=true,root=job.root,imports=&job.imports} +} + +// Starts the thread of job, which is the import that runs once started; it +// is freed when it cannot start, with the reason shown. +@(private) +import_job_start :: proc(s:^Session,job:^Import_Job,format:string,args:..any) { job.thread=thread.create_and_start_with_poly_data(job,import_run) if job.thread==nil { + import_say(s,"%s: no thread could be started to import it",filepath.base(job.source)) import_job_free(job) - import_say(s,"%s: no thread could be started to import it",name) - return false + return } s.importer.running=job s.importer.status={importing=true} - base.text_set(s.importer.status.path[:],source) - import_say(s,"Importing %s",name) - return true + base.text_set(s.importer.status.path[:],job.source) + import_say(s,format,..args) } // The import of job, in its thread: a job system of its own cooks on the @@ -296,37 +296,60 @@ import_run :: proc(job:^Import_Job) { defer if threaded {jobs.system_destroy(&js)} if threaded {job.options.worker={system=&js,worker=jobs.OWNER}} defer job.options.worker={} - if job.profiled && !importer.profile_begin(&job.profile,&job.options,job.allocator) {job.profiled=false} + if job.profiled { + timed:=importer.Import_Options{worker=job.options.worker} + if importer.profile_begin(&job.profile,&timed,job.allocator) {job.options.worker=timed.worker} else {job.profiled=false} + } + added:sources.Library_Report error:importer.Error - if job.whole.run!=nil { - // One store per value until odin-lang/Odin#7937 (see ODIN_WORKAROUNDS). - report,scene,run_error:=job.whole.run(job.state,job.source,job.options,job.allocator) - job.report=report - job.scene,error=scene,run_error + if job.manifest!={} { + added=sources.project_add(job.root,job.manifest,job.folder,job.assets,job.options,job.allocator) + import_added_report(job) } else { - report,import_error:=sources.import_source(job.source,job.options,job.formats,allocator=job.allocator) - job.report=report - error=import_error + settings:=&job.settings if job.settled else nil + job.report,added,error=sources.project_ingest(job.root,job.source,job.folder,job.options,settings,job.allocator) } + defer sources.library_report_destroy(&added) + if error.code==.None && len(added.failures)+len(added.missing)>0 { + error={.Write,strings.join(len(added.failures)>0?added.failures[:]:added.missing[:],"; ",context.temp_allocator)} + } + for warning in added.warnings {append(&job.report.warnings,strings.clone(warning,job.allocator))} if error.code!=.None {job.error={error.code,strings.clone(error.message,job.allocator)}} job.seconds=time.duration_seconds(time.tick_since(job.started)) intrinsics.atomic_store(&job.done,true) } +// What the add of a package gives: its assets the project uses. +@(private) +import_added_report :: proc(job:^Import_Job) { + importer.report_init(&job.report,job.allocator) + p,error:=asset.package_get(&job.store,job.manifest,context.temp_allocator) + if error!=.None {return} + job.report.package_id,job.report.manifest=p.id,job.manifest + for &a in p.assets { + if len(job.assets)>0 && !slice.contains(job.assets,a.id) {continue} + append(&job.report.assets,importer.Written{id=a.id,kind=a.kind}) + } +} + @(private) import_job_free :: proc(job:^Import_Job) { allocator:=job.allocator if job.thread!=nil {thread.destroy(job.thread)} importer.report_destroy(&job.report) - delete(job.source,allocator);delete(job.scene,allocator) - delete(job.options.root,allocator) - if job.store.root!="" {store.close(&job.store)} + delete(job.source,allocator) + delete(job.root,allocator);delete(job.prefix,allocator);delete(job.folder,allocator) + delete(job.assets,allocator) for provider in job.plugin_formats { if provider.prepare!=nil && provider.release!=nil {provider.release(provider.format.data,allocator)} } delete(job.plugin_formats,allocator) - delete(job.formats,allocator) - if job.whole.release!=nil {job.whole.release(job.state,allocator)} + delete(job.imports.formats,allocator) + for source,i in job.wholes { + if source.prepare!=nil && source.release!=nil {source.release(job.imports.wholes[i].state,allocator)} + } + delete(job.wholes,allocator) + delete(job.imports.wholes,allocator) delete(job.error.message,allocator) if job.profiled {base.profile_destroy(&job.profile)} free(job,allocator) @@ -351,25 +374,20 @@ import_poll :: proc(s:^Session) { status.seconds=time.duration_seconds(time.tick_since(job.started)) if job.profiled {import_zones_take(s,profile)} } + a:=s.app.assets + // What its package lists is listed again, also after a failure that pinned it. + if a!=nil { + registry_zone:=base.profile_begin(profile,jobs.OWNER,"editor.registry") + library_relist(s,{job.root,job.prefix}) + base.profile_end(registry_zone) + } if job.error.code!=.None { import_say(s,"%s: %s",name,job.error.message) base.text_set(status.message[:],string(s.importer.message[:s.importer.message_length])) return } - a:=s.app.assets report:=&job.report if a!=nil { - registry_zone:=base.profile_begin(profile,jobs.OWNER,"editor.registry") - // The registry looks again at the files it wrote, and what was loaded from the ones it replaced is read again. - paths:=make([dynamic]string,0,len(report.assets)+1,context.temp_allocator) - written:=true - for entry in report.assets { - path,inside:=asset.registry_relative(&a.registry,entry.path) - if !inside {written=false;break} - append(&paths,path) - } - if written {host.assets_refresh(a,paths[:])} else {host.assets_scan(a)} - base.profile_end(registry_zone) materials_zone:=base.profile_begin(profile,jobs.OWNER,"editor.materials") for entry in report.assets { if entry.kind==asset.KIND_MATERIAL {material_rebase(s,entry.id)} @@ -385,23 +403,25 @@ import_poll :: proc(s:^Session) { base.profile_end(programs_zone) s.importer.publishing=time.tick_now() } - s.content.revision+=1 + if s.content!=nil {s.content.revision+=1} s.importer.imports+=1 status.ok=true for warning in report.warnings {append(&s.importer.warnings,strings.clone(warning))} + scene:="" for entry in report.assets { path:=entry.path if a!=nil { - if relative,inside:=asset.registry_relative(&a.registry,entry.path);inside {path=relative} + if file,found:=asset.registry_find(&a.registry,entry.id);found {path=file.path} } + if job.claimed && entry.kind==asset.KIND_SCENE && scene=="" {scene=path} append(&s.importer.written,Import_Written{id=entry.id,kind=entry.kind,kept=entry.kept,path=strings.clone(path)}) } switch { - case job.scene!="": - scene:=job.scene - if a!=nil {scene,_=asset.registry_relative(&a.registry,job.scene)} + case scene!="": base.text_set(status.scene[:],scene) import_say(s,"Imported %s: the scene %s and %d assets, %d warnings, in %.1f s",name,scene,len(report.assets)-1,len(report.warnings),job.seconds) + case job.manifest!={}: + import_say(s,"Added %s: %d assets",name,len(report.assets)) case: import_say(s,"Imported %s: %d assets",name,len(report.assets)) } @@ -409,40 +429,6 @@ import_poll :: proc(s:^Session) { for warning in report.warnings {log.warn(warning)} } -// The sidecar of the import an asset is the root of (the prefab of a model, -// the texture of an image, the sound of a sound file), and the registry files -// of what that import keeps in the imported folder; "" for any other asset. -// Deleting the root gives the import up: its sidecar goes with it, and with -// the sidecar what only the import can build. -@(private) -import_root :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (sidecar:string,cached:[]u32) { - file,found:=asset.registry_find(r,id) - if !found || s.project_root=="" {return} - types:asset.Types - asset.types_init(&types,context.temp_allocator) - defer asset.types_destroy(&types) - // Every imported asset names the source it was imported from, relative to the root it is of. - full:=asset.registry_file_path(r,file.path,context.temp_allocator) - _,info,imported,error:=asset.import_peek(full,&types) - if error!=.None || !imported {return} - source:=base.text_get(info.source[:]) - root,in_assets:=sources.source_root(full,context.temp_allocator) - if file.listed || !in_assets {root=s.project_root} - for mount in r.mounts { - if file.listed && strings.has_prefix(file.path,mount.prefix) {root=filepath.dir(mount.folder)} - } - source=project.path(root,source,allocator=context.temp_allocator) - text,read_error:=os.read_entire_file(strings.concatenate({source,asset.SIDECAR_EXTENSION},context.temp_allocator),context.temp_allocator) - if read_error!=nil {return} - listed,valid:=asset.sidecar_parse(string(text),context.temp_allocator) - if !valid || len(listed.outputs)==0 || listed.outputs[0].id!=id {return} - files:=make([dynamic]u32,context.temp_allocator) - for output in listed.outputs { - if at,known:=r.by_id[output.id];known && r.files[at].listed {append(&files,at)} - } - return strings.concatenate({source,asset.SIDECAR_EXTENSION},context.temp_allocator),files[:] -} - // Waits for an import that runs and lets go of what the last one wrote: // when the editor ends. import_end :: proc(s:^Session) { @@ -455,7 +441,8 @@ import_end :: proc(s:^Session) { delete(s.importer.written) delete(s.importer.warnings) delete(s.importer.zones) - s.importer.written,s.importer.warnings,s.importer.zones=nil,nil,nil + delete(s.importer.library) + s.importer.written,s.importer.warnings,s.importer.zones,s.importer.library=nil,nil,nil,nil } @(private) @@ -505,35 +492,32 @@ import_working :: proc(s:^Session) -> bool { } // A format of models a plugin adds: its files import as those of the engine, -// in the editor and in the sync of every build (import_library). +// in the editor and in the refresh of every build (import_library). Import_Format_Provider :: struct { format:importer.Format, // Before a model is imported: the options of the format for it - // (Format.data) from the assets of the project and the library of its - // plugins, in allocator; nil keeps format.data. - prepare:proc(assets:^asset.Registry,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, + // (Format.data), in allocator; nil keeps format.data. library: the store + // the packages of the root the import is for are in (asset.root_packages + // of root lists them), info: the types of its game. + prepare:proc(library:^store.Store,root:string,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, release:proc(data:rawptr,allocator:mem.Allocator), // of what prepare gave, once the import ended } // A source a plugin imports whole with importer.Outputs, the one source of // every asset it makes (an Unreal map: a scene and the meshes, materials and // textures it names, the packages it reads being its inputs), in the editor -// and in the sync of every build (import_library). +// and in the refresh of every build (import_library). It is read where it is. Source_Importer :: struct { name:string, // "Unreal maps": its filter in the file dialog and in messages extensions:[importer.FORMAT_EXTENSIONS]string, // lower case, with the dot, "" after the last claims:proc(path:string,head:[]u8) -> bool, // head: up to importer.SNIFF_BYTES from the start of the file - // Before the import starts: what run needs, from the assets of the - // project and the library of its plugins, in allocator; nil for nothing. - prepare:proc(assets:^asset.Registry,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, + // Before the import starts: what run needs, in allocator; nil for + // nothing. library, root and info as Import_Format_Provider.prepare has them. + prepare:proc(library:^store.Store,root:string,info:^core.World_Info,allocator:mem.Allocator) -> rawptr, // In the thread of the import, with a job system of its own in // options.worker. scene: the scene it wrote, in allocator, "" for none. run:proc(state:rawptr,source:string,options:importer.Import_Options,allocator:mem.Allocator) -> (report:importer.Import_Report,scene:string,error:importer.Error), release:proc(state:rawptr,allocator:mem.Allocator), // of what prepare gave, once the import ended - // A source outside the assets: copies it with the files it reads into - // folder (absolute, in the assets) and gives where the source is now, in - // allocator. nil: the source is copied alone. - bring:proc(source,folder:string,allocator:mem.Allocator) -> (brought:string,error:importer.Error), } Import_Source_Provider :: struct {source:Source_Importer} @@ -548,27 +532,10 @@ import_formats :: proc(s:^Session) -> []importer.Format { return out[:] } -// The formats of the import of a model by job, with the options the plugins -// prepare for it. -@(private) -import_formats_prepare :: proc(s:^Session,job:^Import_Job) { - found:[PROVIDER_CAPACITY]Import_Format_Provider - plugins:=found[:provided_all(s,found[:])] - for &provider in plugins { - if provider.prepare!=nil {provider.format.data=provider.prepare(&s.app.assets.registry,s.info,job.allocator)} - } - formats:=make([dynamic]importer.Format,0,len(sources.FORMATS)+len(plugins),job.allocator) - append(&formats,..sources.FORMATS[:]) - for provider in plugins {append(&formats,provider.format)} - job.formats=formats[:] - job.plugin_formats=slice.clone(plugins,job.allocator) -} - // What import_library holds until its release. @(private="file") Import_Library_Held :: struct { library:host.Project_Library, - registry:asset.Registry, providers:[]Import_Format_Provider, formats:[]importer.Format, importers:[]Source_Importer, @@ -577,10 +544,10 @@ Import_Library_Held :: struct { } // The formats of models and the importers of whole sources the plugins of -// the project at root add, prepared for its assets, and the types of its +// the project at root add, prepared with the store s, and the types of its // game, from the library of its editor (sources.Library): what the import -// program of the engine syncs and imports its sources with. -import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_Imports { +// program of the engine refreshes and imports its sources with. +import_library :: proc(root:string,s:^store.Store,allocator:mem.Allocator) -> sources.Library_Imports { path:=project.project_library(root,"editor",context.temp_allocator) if !os.is_file(path) {return {problem="the project has no built editor library, where the importers of its plugins and the types of its game are: mirai build editor builds it"}} held:=new(Import_Library_Held,allocator) @@ -589,16 +556,12 @@ import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_I library,error:=host.project_library_load(path,0) if error!=.None {return {problem=host.library_error_text(error,path)}} held.library=library - // The type of sources.Library names no store: the packages are the machine's. - packages,_:=asset.root_packages(root,"",context.temp_allocator) - asset.registry_init(&held.registry,project.folder_path(root,.Assets,context.temp_allocator),"",allocator,packages.listed) - asset.registry_scan(&held.registry) found:[PROVIDER_CAPACITY]Import_Format_Provider held.providers=slice.clone(found[:provided_in(&held.library.info,found[:])],allocator) held.formats=make([]importer.Format,len(held.providers),allocator) for provider,i in held.providers { held.formats[i]=provider.format - if provider.prepare!=nil {held.formats[i].data=provider.prepare(&held.registry,&held.library.info,allocator)} + if provider.prepare!=nil {held.formats[i].data=provider.prepare(s,root,&held.library.info,allocator)} } claimed:[PROVIDER_CAPACITY]Import_Source_Provider plugins:[PROVIDER_CAPACITY]string @@ -609,7 +572,7 @@ import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_I for provider,i in claimed[:count] { held.importers[i]=provider.source held.wholes[i]={claims=provider.source.claims,run=provider.source.run,code=plugin_code(root,engine,plugins[i])} - if provider.source.prepare!=nil {held.wholes[i].state=provider.source.prepare(&held.registry,&held.library.info,allocator)} + if provider.source.prepare!=nil {held.wholes[i].state=provider.source.prepare(s,root,&held.library.info,allocator)} } return {formats=held.formats,wholes=held.wholes,world=&held.library.info,state=held,release=proc(state:rawptr) { held:=(^Import_Library_Held)(state) @@ -619,7 +582,6 @@ import_library :: proc(root:string,allocator:mem.Allocator) -> sources.Library_I for source,i in held.importers { if source.prepare!=nil && source.release!=nil {source.release(held.wholes[i].state,held.allocator)} } - asset.registry_destroy(&held.registry) host.project_library_unload(&held.library) }} } diff --git a/editor/import_api_test.odin b/editor/import_api_test.odin index 59cc33046..7d3ad252b 100644 --- a/editor/import_api_test.odin +++ b/editor/import_api_test.odin @@ -1,11 +1,20 @@ package editor +import "base:intrinsics" +import "core:fmt" +import "core:os" +import "core:path/filepath" import "core:strings" import "core:testing" +import "core:thread" +import "mirai:asset" +import img "mirai:cook/image" import "mirai:cook/import/sources" import "mirai:editor/api" import "mirai:host" +import "mirai:project" import "mirai:std/base" +import "mirai:store" // asset.import takes by name what an image holds and where a model stands // facing, and by its id the skeleton the clips of a model go on, and says @@ -50,3 +59,200 @@ asset_import_reads_a_relative_source_from_the_project :: proc(t:^testing.T) { testing.expect_value(t,asset_import_source(&s,"assets/hero/walk.glb"),"C:/game/assets/hero/walk.glb") testing.expect_value(t,asset_import_source(&s,"C:/elsewhere/walk.glb"),"C:/elsewhere/walk.glb") } + +// A project of the editor over a store of its own, as a session has them. +@(private="file") +Library_Test :: struct { + using files:Files_Test, + assets:host.Assets, + app:host.App, + importer:Importer, +} + +@(private="file") +library_test_open :: proc(t:^testing.T,l:^Library_Test,name,store_folder:string) -> bool { + files_test_open(t,&l.files,name) or_return + if !testing.expect(t,host.assets_init(&l.assets,{root=project.path(l.root,"assets",allocator=context.temp_allocator),capacity=16})) {return false} + l.app=host.App{assets=&l.assets} + l.s.app,l.s.importer=&l.app,&l.importer + opened,error:=store.open(store_folder) + l.s.store=opened + return testing.expect_value(t,error,store.Error.None) +} + +@(private="file") +library_test_close :: proc(l:^Library_Test) { + import_end(&l.s) + store.close(&l.s.store) + host.assets_destroy(&l.assets) + files_test_close(&l.files) +} + +// Waits for the import that runs and takes what it wrote, as the frames do. +@(private="file") +library_test_end :: proc(t:^testing.T,l:^Library_Test,loc:=#caller_location) -> bool { + job:=l.s.importer.running + if !testing.expect(t,job!=nil,"an import runs",loc=loc) {return false} + for !intrinsics.atomic_load(&job.done) {thread.yield()} + import_poll(&l.s) + status:=&l.s.importer.status + return testing.expectf(t,status.ok,"%s",base.text_get(status.message[:]),loc=loc) +} + +// The files under folder, by path. +@(private="file") +library_test_files :: proc(folder:string) -> []string { + out:=make([dynamic]string,context.temp_allocator) + todo:=make([dynamic]string,context.temp_allocator) + append(&todo,folder) + for next:=0;next []asset.Project_Package { + list,_,_:=asset.packages_load(root,context.temp_allocator) + return list.packages[:] +} + +// A PNG outside the project, as a download is. +@(private="file") +library_test_source :: proc(t:^testing.T,folder:string) -> string { + pixels:=make([]u8,8*8*4,context.temp_allocator) + for &p,at in pixels {p=u8(at*7)} + path:=fmt.tprintf("%s/downloads/sign.png",folder) + os.make_directory_all(filepath.dir(path)) + testing.expect_value(t,img.image_write(path,{pixels=pixels,width=8,height=8}),img.Error.None) + return path +} + +// Importing a source from outside the project ingests it where it is and +// pins its package: the project lists its texture from the store and copies +// nothing. Moving its root changes where the package is listed, and deleting +// it unpins the package and leaves the store and the source as they were. +@(test) +a_source_from_outside_is_pinned_listed_moved_and_unpinned_without_copies :: proc(t:^testing.T) { + l:=new(Library_Test,context.temp_allocator) + top:=base.test_folder(t,"editor-library") + if top=="" || !library_test_open(t,l,"editor-library-game",fmt.tprintf("%s/store",top)) {return} + defer library_test_close(l) + s:=&l.s + r:=&l.assets.registry + source:=library_test_source(t,top) + before,_:=os.read_entire_file(source,context.temp_allocator) + + if !testing.expect(t,import_file(s,source,"props"),string(s.importer.message[:s.importer.message_length])) || !library_test_end(t,l) {return} + pins:=library_test_pins(l.root) + if !testing.expect_value(t,len(pins),1) {return} + testing.expect_value(t,pins[0].folder,"props") + file,found:=asset.registry_find_path(r,"props/sign.png.tex") + if !found {file,found=asset.registry_find_path(r,"props/sign.tex")} + if !testing.expectf(t,found && file.listed,"the texture is listed from the package: %v",r.listed) {return} + id:=file.id + blob:=asset.registry_file_path(r,file.path,context.temp_allocator) + testing.expectf(t,strings.has_prefix(blob,s.store.root),"%s is a blob of the store",blob) + testing.expectf(t,len(library_test_files(project.path(l.root,"assets",allocator=context.temp_allocator)))==0,"nothing is copied into the assets") + after,_:=os.read_entire_file(source,context.temp_allocator) + testing.expect(t,string(before)==string(after),"the source is as it was") + + // Its root moves where the package is listed, keeping its name. + _,message:=asset_move(s,id,"props","other") + testing.expect(t,strings.has_prefix(message,"name:"),message) + path:string + path,message=asset_move(s,id,"textures","") + testing.expectf(t,message=="" && strings.has_prefix(path,"textures/"),"%s %s",path,message) + testing.expect_value(t,library_test_pins(l.root)[0].folder,"textures") + moved,_:=asset.registry_find(r,id) + testing.expect(t,moved!=nil && moved.path==path) + + // Deleting it unpins the package: the store and the source stay. + gone,refused:=asset_delete(s,id) + testing.expectf(t,refused=="" && len(gone)==1 && gone[0]==path,"%v %s",gone,refused) + testing.expect_value(t,len(library_test_pins(l.root)),0) + _,listed:=asset.registry_find(r,id) + testing.expect(t,!listed,"what it listed left the project") + testing.expect(t,os.is_file(blob),"the blob stays in the store") + testing.expect(t,os.is_file(source),"the source outside the project stays") +} + +// A package of the library added to a second project lists the blobs the +// first one does; packages.list and packages.add say for each hostile +// argument what failed and how to fix it. +@(test) +a_package_of_the_library_goes_to_a_second_project_with_the_same_blobs :: proc(t:^testing.T) { + top:=base.test_folder(t,"editor-library-add") + if top=="" {return} + kept:=fmt.tprintf("%s/store",top) + first:=new(Library_Test,context.temp_allocator) + if !library_test_open(t,first,"editor-library-first",kept) {return} + defer library_test_close(first) + source:=library_test_source(t,top) + if !testing.expect(t,import_file(&first.s,source,"")) || !library_test_end(t,first) {return} + pin:=library_test_pins(first.root)[0] + file:^asset.Registry_File + for &f in first.assets.registry.files { + if f.listed {file=&f} + } + if !testing.expect(t,file!=nil) {return} + blob:=strings.clone(asset.registry_file_path(&first.assets.registry,file.path,context.temp_allocator),context.temp_allocator) + + second:=new(Library_Test,context.temp_allocator) + if !library_test_open(t,second,"editor-library-second",kept) {return} + defer library_test_close(second) + s:=&second.s + call :: proc(s:^Session,run:proc(s:^Session,call:^api.Call),params:^$P,$R:typeid) -> api.Call { + method:=api.Method{params_id=P,result_id=R} + call:=api.Call{method=&method,params=params,result=new(R,context.temp_allocator),allocator=context.temp_allocator} + run(s,&call) + call.method,call.params=nil,nil + return call + } + listing:Api_Packages_List_Params + listed:=call(s,api_packages_list,&listing,Api_Packages_List_Result) + result:=(^Api_Packages_List_Result)(listed.result) + if !testing.expectf(t,!listed.failed && result!=nil && len(result.packages)==1,"%s",listed.error.message) {return} + p:=result.packages[0] + testing.expect(t,p.id==pin.id && !p.pinned && len(p.assets)==1 && p.assets[0].built && p.assets[0].bytes>0 && p.assets[0].kind=="texture") + + Hostile :: struct {name:string,assets:[]string,folder,says:string} + buffer:[base.ASSET_ID_TEXT]u8 + named:=strings.clone(base.asset_id_format(pin.id,buffer[:]),context.temp_allocator) + hostile:=[?]Hostile{ + {"",nil,"","packages.list"}, + {"nothing such",nil,"","packages.list"}, + {named,{"zz"},"","packages.list"}, + {named,{"0123456789abcdef0123456789abcdef"},"","packages.list"}, + {named,nil,"../out","folder:"}, + {named,nil,".editor/x","folder:"}, + } + for h in hostile { + params:=Api_Packages_Add_Params{name=h.name,assets=h.assets,folder=h.folder} + answer:=call(s,api_packages_add,¶ms,Import_Status_Result) + testing.expectf(t,answer.failed && answer.error.code==api.INVALID_PARAMS && strings.contains(answer.error.message,h.says),"%v: %s",h,answer.error.message) + testing.expect(t,s.importer.running==nil,"nothing starts") + } + params:=Api_Packages_Add_Params{name=named,folder="shared"} + added:=call(s,api_packages_add,¶ms,Import_Status_Result) + if !testing.expectf(t,!added.failed,"%s",added.error.message) || !library_test_end(t,second) {return} + pins:=library_test_pins(second.root) + testing.expect(t,len(pins)==1 && pins[0].id==pin.id && pins[0].folder=="shared") + there,found:=asset.registry_find(&second.assets.registry,file.id) + testing.expect(t,found && strings.has_prefix(there.path,"shared/") && asset.registry_file_path(&second.assets.registry,there.path,context.temp_allocator)==blob,"the same blob") + + // Without a store nothing lists or imports, and each says how to give one. + store.close(&s.store) + without:=call(s,api_packages_list,&listing,Api_Packages_List_Result) + testing.expect(t,without.failed && strings.contains(without.error.message,"--store"),without.error.message) + testing.expect(t,!import_file(s,source,"")) + testing.expect(t,strings.contains(string(s.importer.message[:s.importer.message_length]),"--store")) + opened,_:=store.open(kept) + s.store=opened + testing.expect(t,!import_file(s,fmt.tprintf("%s/none.png",top),"")) + testing.expect(t,strings.contains(string(s.importer.message[:s.importer.message_length]),"does not exist")) +} diff --git a/editor/library.odin b/editor/library.odin new file mode 100644 index 000000000..1fb80952d --- /dev/null +++ b/editor/library.odin @@ -0,0 +1,337 @@ +package editor + +import "base:runtime" +import "core:fmt" +import "core:log" +import "core:slice" +import "core:strings" +import "core:time" +import "mirai:asset" +import "mirai:cook/import/sources" +import "mirai:editor/api" +import "mirai:host" +import "mirai:std/base" +import "mirai:std/serial" +import "mirai:store" +import "mirai:ui" + +// The library of the machine in the editor: the packages the store of the +// session keeps (cook/import/sources/library.odin) and what a root of the +// editor, the project or a plugin, pins of them in its packages.mirai. +// Importing ingests a source where it is and pins its package (import.odin); +// packages.list lists the packages of the library with their assets, and +// packages.add, or Add from library of the content browser, pins one or some +// of its assets in a folder of the project, as an import that runs in its +// thread. Deleting the root of a package unpins it and moving it moves where +// the package is listed (asset_delete.odin, asset_move.odin); none of them +// removes a blob of the store or touches a source outside the project. + +// A root of the registry: the project (prefix "") or a plugin mounted under prefix. +Library_Root :: struct {root,prefix:string} + +// The package a root pins that gives an asset: its entry in the list of the +// root and its manifest, in the temp allocator. +@(private) +Library_Pin :: struct { + root:Library_Root, + entry:asset.Project_Package, + p:asset.Package, +} + +// A package of the library, as packages.list tells it. +@(private) +Library_Package :: struct { + id:base.Asset_Id, + manifest:store.Hash, + name,origin:string, + pinned:bool, // the project pins it + folder:string, // where the project lists it, when pinned + assets:[]Library_Asset, +} + +@(private) +Library_Asset :: struct { + id:base.Asset_Id, + name:string, + kind:asset.Fourcc, + bytes:i64, // of its blob in the build of this machine; 0 when not built + built:bool, +} + +// What Add from library shows of a package: its name, cut to fit. +@(private) +Library_Shown :: struct { + id:base.Asset_Id, + name:[64]u8, + name_length:int, + pinned:bool, +} + +// Lists again what the packages of root list, from the store of the session, +// and scans the registry. What they cannot list is told in the log. +@(private) +library_relist :: proc(s:^Session,root:Library_Root) { + r:=files_registry(s) + if r==nil {return} + packages,error:=asset.root_packages(root.root,s.store.root,context.temp_allocator) + if error!=.None { + log.warnf("the packages of %s could not be listed: %v",root.root,error) + return + } + for problem in packages.problems {log.warnf("assets: %s",problem)} + if !asset.registry_listed_set(r,root.prefix,packages.listed) {log.warnf("the packages of %s could not be listed: no memory",root.root)} + host.assets_scan(s.app.assets) +} + +// The package a root pins that gives the asset of file, with its manifest, +// for what a package lists (by the package the registry lists it from) and +// for a scene, which may be the root of one; false for any other asset, an +// asset no package gives, or without the store. +@(private) +library_pin_of :: proc(s:^Session,r:^asset.Registry,file:^asset.Registry_File) -> (pin:Library_Pin,found:bool) { + if s.store.root=="" || !file.listed && file.kind!=asset.KIND_SCENE {return} + root,_,rooted:=import_root_of(s,r,content_folder_of(file.path)) + if !rooted {return} + // What a package lists names it: only its manifest is read. + listed:base.Asset_Id + if file.listed { + entries:=r.listed + for mount in r.mounts { + if mount.prefix==root.prefix && root.prefix!="" {entries=mount.listed} + } + for &l in entries { + if l.id==file.id {listed=l.package_id} + } + if listed=={} {return} + } + list,_,error:=asset.packages_load(root.root,context.temp_allocator) + if error!=.None {return} + for &entry in list.packages { + if listed!={} && entry.id!=listed {continue} + p,read_error:=asset.package_get(&s.store,entry.manifest,context.temp_allocator) + if read_error!=.None || p.id!=entry.id {continue} + for &a in p.assets { + if a.id==file.id {return {root,entry,p},true} + } + } + return +} + +// The root of a package: the prefab of a model or the scene of a source a +// plugin imports whole, else its one asset that is no part of another (the +// texture of an image, the sound of a sound file). Zero when it has none. +@(private) +library_package_root :: proc(p:^asset.Package) -> base.Asset_Id { + for &a in p.assets { + if a.kind==asset.KIND_SCENE && a.owner=={} {return a.id} + } + root:base.Asset_Id + for &a in p.assets { + if a.owner!={} || a.part {continue} + if root!={} {return {}} + root=a.id + } + return root +} + +// Where the root of a package is in the registry, or its id when it is not there. +@(private) +library_root_named :: proc(r:^asset.Registry,p:^asset.Package) -> string { + root:=library_package_root(p) + if file,found:=asset.registry_find(r,root);found {return file.path} + buffer:[base.ASSET_ID_TEXT]u8 + return fmt.tprintf("%s (%s)",p.name,base.asset_id_format(root,buffer[:])) +} + +// Pins entry again in its root, with what changed of it, and lists again. +@(private) +library_repin :: proc(s:^Session,root:Library_Root,entry:asset.Project_Package) -> string { + if error:=sources.project_pin(root.root,entry);error!=.None { + return fmt.tprintf("%s/%s could not be written (%v): check that it is not read-only, then try again",root.root,asset.PACKAGES_FILE,error) + } + library_relist(s,root) + return "" +} + +// The packages of the library of the session, by name, with whether the +// root at root pins them. In allocator. +@(private) +library_list :: proc(s:^Session,root:string,allocator:=context.allocator) -> (list:[]Library_Package,problem:string) { + if s.store.root=="" {return nil,"The editor has no store, where the library keeps its packages: start it with --store "} + refs,error:=store.ref_list(&s.store,asset.REF_PACKAGES,context.temp_allocator) + if error!=.None && error!=.Missing {return nil,fmt.tprintf("The packages of the store at %s could not be listed (%v): check that the folder reads",s.store.root,error)} + pins:asset.Packages + if root!="" {pins,_,_=asset.packages_load(root,context.temp_allocator)} + out:=make([dynamic]Library_Package,0,len(refs),allocator) + for ref in refs { + p,read_error:=asset.package_get(&s.store,ref.hash,allocator) + if read_error!=.None {continue} + entry:=Library_Package{id=p.id,manifest=ref.hash,name=p.name,origin=p.origin.path} + for &pinned in pins.packages { + if pinned.id==p.id {entry.pinned,entry.folder=true,strings.clone(pinned.folder,allocator)} + } + outputs:=make(map[base.Asset_Id]asset.Build_Output,context.temp_allocator) + digits:[64]u8 + if built,ref_error:=store.ref_get(&s.store,asset.REF_BUILDS,store.hex(asset.build_select(ref.hash,&p),&digits));ref_error==.None { + if b,build_error:=asset.build_get(&s.store,built,context.temp_allocator);build_error==.None && b.package_id==p.id { + for o in b.outputs {outputs[o.id]=o} + } + } + assets:=make([]Library_Asset,len(p.assets),allocator) + for &a,at in p.assets { + assets[at]={id=a.id,name=a.name,kind=a.kind} + if o,found:=outputs[a.id];found && o.kind==a.kind && store.has(&s.store,o.blob) { + assets[at].built=true + assets[at].bytes,_=store.blob_size(&s.store,o.blob) + } + } + entry.assets=assets + append(&out,entry) + } + slice.sort_by(out[:],proc(a,b:Library_Package) -> bool {return a.name string { + r:=files_registry(s) + if r==nil {return "The project has no assets folder to add into"} + if busy:=import_busy(s);busy!="" {return busy} + clean,refused:=asset_folder_clean(r,folder) + if refused!="" {return fmt.tprintf("folder: %s",refused)} + root,rest,rooted:=import_root_of(s,r,clean) + if !rooted {return fmt.tprintf("folder: %s is a folder of the editor or the engine: give a folder of the project or of a plugin",clean)} + packages,problem:=library_list(s,root.root,context.temp_allocator) + if problem!="" {return problem} + wanted:=strings.trim_space(named) + if wanted=="" {return "name: give the id or the name of a package, as packages.list lists them"} + chosen:=-1 + id,is_id:=serial.text_parse_asset_id(wanted) + for &p,at in packages { + if !(is_id && p.id==id || !is_id && p.name==wanted) {continue} + if chosen>=0 {return fmt.tprintf("name: more than one package is named %q: give its id, as packages.list lists them",wanted)} + chosen=at + } + if chosen<0 {return fmt.tprintf("name: the library at %s has no package %q: packages.list lists them, and asset.import ingests a source",s.store.root,wanted)} + p:=&packages[chosen] + used:=make([dynamic]base.Asset_Id,0,len(ids),context.temp_allocator) + for text in ids { + asset_id,parsed:=serial.text_parse_asset_id(strings.trim_space(text)) + known:=false + for &a in p.assets {known||=parsed && a.id==asset_id} + if !known {return fmt.tprintf("assets: %q is no asset of the package %s: give the ids packages.list lists for it, or none for all",text,p.name)} + if !slice.contains(used[:],asset_id) {append(&used,asset_id)} + } + allocator:=runtime.heap_allocator() + job:=new(Import_Job,allocator) + job^={allocator=allocator,manifest=p.manifest,started=time.tick_now()} + job.store=s.store + job.source=strings.clone(p.name,allocator) + job.root,job.prefix,job.folder=strings.clone(root.root,allocator),strings.clone(root.prefix,allocator),strings.clone(rest,allocator) + job.assets=slice.clone(used[:],allocator) + import_plugins_prepare(s,job,true,{},0) + import_job_start(s,job,"Adding %s",p.name) + if job.thread==nil {return string(s.importer.message[:s.importer.message_length])} + return "" +} + +// ---------------------------------------------------------------- packages.list, packages.add + +Api_Packages_List_Params :: struct { + name:string `mirai:"optional,label=Only the packages whose name holds it, in any case; empty: all"`, +} + +Api_Packages_List_Result :: struct { + store:string `mirai:"label=The folder of the store the library keeps its packages in"`, + packages:[]Api_Package, +} +Api_Package :: struct { + id:base.Asset_Id, + name:string, + source:string `mirai:"label=Where its source was when it was ingested on this machine"`, + pinned:bool `mirai:"label=The project pins it"`, + folder:string `mirai:"optional,label=Of the assets, where the project lists it, when pinned"`, + assets:[]Api_Package_Asset, +} +Api_Package_Asset :: struct { + id:base.Asset_Id, + name:string `mirai:"label=In the source"`, + kind:string, + bytes:i64 `mirai:"label=Of its blob in the build of this machine; 0 when not built"`, + built:bool `mirai:"label=This machine has a build of it this engine reads"`, +} + +Api_Packages_Add_Params :: struct { + name:string `mirai:"label=The package: its id or its name, as packages.list lists them"`, + assets:[]string `mirai:"optional,label=Of the package, the ids of the assets the project uses, with what they depend on; none: all"`, + folder:string `mirai:"optional,label=The folder under the asset folder it is listed in; none: the root of the assets"`, + seconds:int `mirai:"optional,min=1,max=7200,unit=s,label=How long it waits for the end of the add; past that it answers importing and the add goes on"`, +} + +@(private) +api_packages_list :: proc(s:^Session,call:^api.Call) { + params:=api.call_params(call,Api_Packages_List_Params) + packages,problem:=library_list(s,s.project_root,call.allocator) + if problem!="" { + api.call_fail(call,api.SERVER_ERROR,problem) + return + } + wanted:=strings.to_lower(strings.trim_space(params.name),context.temp_allocator) + out:=make([dynamic]Api_Package,0,len(packages),call.allocator) + for &p in packages { + if wanted!="" && !strings.contains(strings.to_lower(p.name,context.temp_allocator),wanted) {continue} + assets:=make([]Api_Package_Asset,len(p.assets),call.allocator) + for a,at in p.assets { + kind:="other" + if info,known:=asset.kind_info(a.kind);known {kind=info.name} + assets[at]={a.id,a.name,kind,a.bytes,a.built} + } + append(&out,Api_Package{p.id,p.name,p.origin,p.pinned,p.folder,assets}) + } + result:=api.call_result(call,Api_Packages_List_Result) + result.store=strings.clone(s.store.root,call.allocator) + result.packages=out[:] +} + +@(private) +api_packages_add :: proc(s:^Session,call:^api.Call) { + params:=api.call_params(call,Api_Packages_Add_Params) + if problem:=library_add(s,params.name,params.assets,params.folder);problem!="" { + api.call_fail(call,api.INVALID_PARAMS,problem) + return + } + importing :: proc(s:^Session,user:rawptr) -> bool {return s.importer.running!=nil} + api_wait(s,f64(params.seconds),nil,importing,"adding") + api_import_status_answer(s,call) +} + +// The packages of the library as Add from library of the content browser +// offers them: each adds the whole package to the folder the browser is in. +// Read again each time the menu opens. +@(private) +content_library_items :: proc(s:^Session) { + ctx:=s.gui.ctx + im:=s.importer + if s.frame>im.library_frame+1 || im.library_frame==0 { + clear(&im.library) + packages,_:=library_list(s,s.project_root,context.temp_allocator) + for &p in packages { + shown:=Library_Shown{id=p.id,pinned=p.pinned} + shown.name_length=copy(shown.name[:],p.name) + append(&im.library,shown) + } + } + im.library_frame=s.frame + if len(im.library)==0 { + ui.menu_item(ctx,ui.id(ctx,"library_none"),"No packages",{disabled=true}) + return + } + for &shown,at in im.library { + if !ui.menu_item(ctx,ui.id(ctx,"library_package",at+1),string(shown.name[:shown.name_length]),{checked=shown.pinned,disabled=shown.pinned}) {continue} + buffer:[base.ASSET_ID_TEXT]u8 + if problem:=library_add(s,base.asset_id_format(shown.id,buffer[:]),nil,import_place(s));problem!="" {notify(s,problem)} + } +} From d76ca2446b9fc9c61899a35cc0c0cbfca4374a0f Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 20:09:07 -0300 Subject: [PATCH 021/123] import: a root carries the manifests it pins, and the repository's roots pin their sources A root keeps the text of each manifest packages.mirai pins in packages/.mirai, written by project_pin and removed by project_unpin. Resolving and refreshing take a manifest the store lacks from that file and put it in the store; a file that is not its pin is reported by package. The origin of a source inside a root is relative to that root, so every copy writes the same manifest and finds its source. The 39 sidecars of demo, templates/game, plugins/prompts_kenney and editor/assets/presence are gone: the first three roots pin one package per source with the ids of its sidecar; the presence sidecar had no ids and no source, and its assets stay committed as they are. Co-Authored-By: Claude Opus 5.5 --- .agents/skills/mirai-assets/SKILL.md | 16 ++- .../skills/mirai-assets/references/formats.md | 27 ++-- .agents/skills/mirai-imports/SKILL.md | 4 +- .../mirai-imports/references/imports.md | 15 ++- .../skills/mirai-projects/references/build.md | 8 +- .../mirai-projects/references/projects.md | 5 +- asset/packages.odin | 122 ++++++++++++++---- asset/packages_root.odin | 11 +- asset/packages_test.odin | 62 ++++++++- cook/import/identity.odin | 34 +++++ cook/import/outputs_test.odin | 2 +- cook/import/package.odin | 4 +- cook/import/sources/command.odin | 2 +- cook/import/sources/library.odin | 87 ++++++++----- cook/import/sources/library_test.odin | 109 +++++++++++++++- cook/import/sources/sources.odin | 18 +-- cook/import/sources/support_test.odin | 4 +- .../assets/characters/jelly/jelly.gltf.import | 14 -- demo/assets/characters/jelly/jelly/Candy.mat | Bin 4520 -> 4520 bytes .../jelly/jelly/Jelly bones.skeleton | Bin 2544 -> 2544 bytes demo/assets/characters/jelly/jelly/jelly.anim | Bin 2832 -> 2832 bytes .../assets/characters/mirai/Armature.skeleton | Bin 28704 -> 28704 bytes .../assets/characters/mirai/BakedMaterial.mat | Bin 4552 -> 4552 bytes .../characters/mirai/Mirai borde luminoso.mat | Bin 4520 -> 4520 bytes .../mirai/Mirai holograma violeta.mat | Bin 4520 -> 4520 bytes demo/assets/characters/mirai/mirai.anim | Bin 129472 -> 129472 bytes demo/assets/characters/mirai/mirai.glb.import | 17 --- .../robot/glb/13 joint robot rig.skeleton | Bin 17456 -> 17456 bytes .../characters/robot/glb/robot blue.mat | Bin 4520 -> 4520 bytes .../characters/robot/glb/robot gold.mat | Bin 4520 -> 4520 bytes .../characters/robot/glb/robot joints.mat | Bin 4520 -> 4520 bytes .../characters/robot/glb/robot visor.mat | Bin 4520 -> 4520 bytes .../robot/gltf/13 joint robot rig.skeleton | Bin 17456 -> 17456 bytes .../characters/robot/gltf/robot blue.mat | Bin 4520 -> 4520 bytes .../characters/robot/gltf/robot gold.mat | Bin 4520 -> 4520 bytes .../characters/robot/gltf/robot joints.mat | Bin 4520 -> 4520 bytes .../characters/robot/gltf/robot visor.mat | Bin 4520 -> 4520 bytes demo/assets/characters/robot/robot.glb.import | 14 -- .../assets/characters/robot/robot.gltf.import | 15 --- .../church/church_glass_color.png.import | 9 -- demo/assets/gallery/leaves_color.png.import | 10 -- demo/assets/gallery/plant.gltf.import | 14 -- demo/assets/gallery/plant/Leaves.mat | Bin 4536 -> 4536 bytes demo/assets/gallery/plant/Soil.mat | Bin 4520 -> 4520 bytes demo/assets/gallery/plant/Terracotta.mat | Bin 4520 -> 4520 bytes demo/assets/gallery/vase.obj.import | 13 -- demo/assets/gallery/vase/Ceramic.mat | Bin 4536 -> 4536 bytes demo/assets/gallery/vase/Gilding.mat | Bin 4520 -> 4520 bytes demo/assets/gallery/vase_color.png.import | 9 -- demo/assets/showcase/hum.wav.import | 7 - demo/assets/sky/sky_day.hdr.import | 7 - demo/assets/sky/sky_night.jpg.import | 9 -- demo/assets/sky/sky_overcast.jpg.import | 9 -- demo/assets/sky/sky_sunset.jpg.import | 9 -- .../surfaces/bricks/bricks_color.png.import | 9 -- .../surfaces/bricks/bricks_normal.png.import | 9 -- .../surfaces/bricks/bricks_orm.png.import | 7 - .../cobblestone/cobblestone_color.png.import | 9 -- .../cobblestone/cobblestone_normal.png.import | 9 -- .../cobblestone/cobblestone_orm.png.import | 7 - .../glass/stained_glass_color.png.import | 9 -- .../surfaces/marble/marble_color.png.import | 9 -- .../surfaces/marble/marble_normal.png.import | 9 -- .../surfaces/marble/marble_orm.png.import | 7 - .../surfaces/metal/brushed_normal.png.import | 9 -- .../surfaces/metal/brushed_orm.png.import | 7 - .../surfaces/metal/grate_color.png.import | 10 -- .../surfaces/metal/grate_normal.png.import | 9 -- .../surfaces/metal/grate_orm.png.import | 7 - .../surfaces/neon/neon_emission.png.import | 9 -- .../surfaces/tiles/tiles_color.png.import | 9 -- .../surfaces/tiles/tiles_normal.png.import | 9 -- .../surfaces/tiles/tiles_orm.png.import | 7 - .../surfaces/wood/planks_color.png.import | 9 -- .../surfaces/wood/planks_normal.png.import | 9 -- .../surfaces/wood/planks_orm.png.import | 7 - demo/packages.mirai | 41 ++++++ .../0512ac6bb76639efa1aa35746972fedb.mirai | 21 +++ .../09648c700552c36b5b4062fd1822de01.mirai | 14 ++ .../0964d5141215cea8462119f07a904e49.mirai | 11 ++ .../0ab7eefb80a0bb80c8cd0065d5c53f11.mirai | 14 ++ .../0b5202b0a19887317347ddcbfdcad5f6.mirai | 14 ++ .../0d0874d674d00d43ab66a390641369df.mirai | 14 ++ .../181b8168d78256877581cb3f1fda937e.mirai | 11 ++ .../26356114f7faffcfafff109308dd9a32.mirai | 11 ++ .../2d26a9a0d52c3edd23a71d2cc8d6e463.mirai | 17 +++ .../2d6b0454a36971219d185d940f836ac7.mirai | 14 ++ .../44342052da337fa065c60cf296a4466a.mirai | 14 ++ .../4840a3f5a151cbe8c1b403b30fe8240d.mirai | 14 ++ .../641ffc8940c220c3b2460f7a9eaa7862.mirai | 14 ++ .../6498e1a94e6c4621f925d22f8384243a.mirai | 16 +++ .../65805c17b2454730221e3d63066ce7f5.mirai | 14 ++ .../6cc37d40c23ded0fb9cc963a4dec2fbb.mirai | 14 ++ .../7572bd9ffa0dd2354005eff39b25f1bb.mirai | 11 ++ .../780bee6597fbb860e803d513bda2dee7.mirai | 16 +++ .../81d18317bfbbf0851f30ebcb2de41b67.mirai | 11 ++ .../84060b2db086d874fb6a99a638a79a10.mirai | 17 +++ .../856074124b2e8924607a1e79488b6abd.mirai | 14 ++ .../a03a7d06fb4c433e89ba1ecd314d91ed.mirai | 11 ++ .../a270e82878cb164498ba98398060ccb8.mirai | 14 ++ .../a278db7076e649a48933fc75b0dc1a4c.mirai | 14 ++ .../a3799fb21f329ae819440d1726f2aa20.mirai | 15 +++ .../a68982d58a94bfa0fcb588bf93dfafec.mirai | 11 ++ .../ae40765be8d60ddad02e47bb21203f8f.mirai | 11 ++ .../c0d7666d947e65fc7dd68a31013c0939.mirai | 14 ++ .../c1a66f64c05efe53cca206765ab98117.mirai | 14 ++ .../ca27d206e666815335b68ee9c8900ab7.mirai | 14 ++ .../caae575f45863b717aafbe7fe73862c0.mirai | 15 +++ .../de7548cbbad3494b21280744cd16ed68.mirai | 11 ++ .../e022a803a1c6fb9fe4b86ab65a48bc24.mirai | 14 ++ .../e0495081646e8954b731fc082bcd8884.mirai | 15 +++ .../e96915c067e6fe50d517b59bf3b17d61.mirai | 14 ++ .../f4ccf188ab1bf19a62c53faaef829fba.mirai | 14 ++ docs/game/assets.md | 8 +- docs/game/assets/content-tool.md | 10 +- docs/game/ship.md | 4 +- editor/assets/presence/source/pup.glb.import | 12 -- editor/import.odin | 8 +- plugins/prompts_kenney/assets/kenney.prompts | Bin 1952 -> 1952 bytes plugins/prompts_kenney/packages.mirai | 6 + .../18e1ec191e6b7a7386eba31ba4c8e829.mirai} | 15 ++- templates/game/assets/arena/hum.wav.import | 7 - templates/game/packages.mirai | 6 + .../ea1313ae2ad84fe489d5bf4061c5fda9.mirai | 11 ++ 124 files changed, 995 insertions(+), 501 deletions(-) delete mode 100644 demo/assets/characters/jelly/jelly.gltf.import delete mode 100644 demo/assets/characters/mirai/mirai.glb.import delete mode 100644 demo/assets/characters/robot/robot.glb.import delete mode 100644 demo/assets/characters/robot/robot.gltf.import delete mode 100644 demo/assets/church/church_glass_color.png.import delete mode 100644 demo/assets/gallery/leaves_color.png.import delete mode 100644 demo/assets/gallery/plant.gltf.import delete mode 100644 demo/assets/gallery/vase.obj.import delete mode 100644 demo/assets/gallery/vase_color.png.import delete mode 100644 demo/assets/showcase/hum.wav.import delete mode 100644 demo/assets/sky/sky_day.hdr.import delete mode 100644 demo/assets/sky/sky_night.jpg.import delete mode 100644 demo/assets/sky/sky_overcast.jpg.import delete mode 100644 demo/assets/sky/sky_sunset.jpg.import delete mode 100644 demo/assets/surfaces/bricks/bricks_color.png.import delete mode 100644 demo/assets/surfaces/bricks/bricks_normal.png.import delete mode 100644 demo/assets/surfaces/bricks/bricks_orm.png.import delete mode 100644 demo/assets/surfaces/cobblestone/cobblestone_color.png.import delete mode 100644 demo/assets/surfaces/cobblestone/cobblestone_normal.png.import delete mode 100644 demo/assets/surfaces/cobblestone/cobblestone_orm.png.import delete mode 100644 demo/assets/surfaces/glass/stained_glass_color.png.import delete mode 100644 demo/assets/surfaces/marble/marble_color.png.import delete mode 100644 demo/assets/surfaces/marble/marble_normal.png.import delete mode 100644 demo/assets/surfaces/marble/marble_orm.png.import delete mode 100644 demo/assets/surfaces/metal/brushed_normal.png.import delete mode 100644 demo/assets/surfaces/metal/brushed_orm.png.import delete mode 100644 demo/assets/surfaces/metal/grate_color.png.import delete mode 100644 demo/assets/surfaces/metal/grate_normal.png.import delete mode 100644 demo/assets/surfaces/metal/grate_orm.png.import delete mode 100644 demo/assets/surfaces/neon/neon_emission.png.import delete mode 100644 demo/assets/surfaces/tiles/tiles_color.png.import delete mode 100644 demo/assets/surfaces/tiles/tiles_normal.png.import delete mode 100644 demo/assets/surfaces/tiles/tiles_orm.png.import delete mode 100644 demo/assets/surfaces/wood/planks_color.png.import delete mode 100644 demo/assets/surfaces/wood/planks_normal.png.import delete mode 100644 demo/assets/surfaces/wood/planks_orm.png.import create mode 100644 demo/packages.mirai create mode 100644 demo/packages/0512ac6bb76639efa1aa35746972fedb.mirai create mode 100644 demo/packages/09648c700552c36b5b4062fd1822de01.mirai create mode 100644 demo/packages/0964d5141215cea8462119f07a904e49.mirai create mode 100644 demo/packages/0ab7eefb80a0bb80c8cd0065d5c53f11.mirai create mode 100644 demo/packages/0b5202b0a19887317347ddcbfdcad5f6.mirai create mode 100644 demo/packages/0d0874d674d00d43ab66a390641369df.mirai create mode 100644 demo/packages/181b8168d78256877581cb3f1fda937e.mirai create mode 100644 demo/packages/26356114f7faffcfafff109308dd9a32.mirai create mode 100644 demo/packages/2d26a9a0d52c3edd23a71d2cc8d6e463.mirai create mode 100644 demo/packages/2d6b0454a36971219d185d940f836ac7.mirai create mode 100644 demo/packages/44342052da337fa065c60cf296a4466a.mirai create mode 100644 demo/packages/4840a3f5a151cbe8c1b403b30fe8240d.mirai create mode 100644 demo/packages/641ffc8940c220c3b2460f7a9eaa7862.mirai create mode 100644 demo/packages/6498e1a94e6c4621f925d22f8384243a.mirai create mode 100644 demo/packages/65805c17b2454730221e3d63066ce7f5.mirai create mode 100644 demo/packages/6cc37d40c23ded0fb9cc963a4dec2fbb.mirai create mode 100644 demo/packages/7572bd9ffa0dd2354005eff39b25f1bb.mirai create mode 100644 demo/packages/780bee6597fbb860e803d513bda2dee7.mirai create mode 100644 demo/packages/81d18317bfbbf0851f30ebcb2de41b67.mirai create mode 100644 demo/packages/84060b2db086d874fb6a99a638a79a10.mirai create mode 100644 demo/packages/856074124b2e8924607a1e79488b6abd.mirai create mode 100644 demo/packages/a03a7d06fb4c433e89ba1ecd314d91ed.mirai create mode 100644 demo/packages/a270e82878cb164498ba98398060ccb8.mirai create mode 100644 demo/packages/a278db7076e649a48933fc75b0dc1a4c.mirai create mode 100644 demo/packages/a3799fb21f329ae819440d1726f2aa20.mirai create mode 100644 demo/packages/a68982d58a94bfa0fcb588bf93dfafec.mirai create mode 100644 demo/packages/ae40765be8d60ddad02e47bb21203f8f.mirai create mode 100644 demo/packages/c0d7666d947e65fc7dd68a31013c0939.mirai create mode 100644 demo/packages/c1a66f64c05efe53cca206765ab98117.mirai create mode 100644 demo/packages/ca27d206e666815335b68ee9c8900ab7.mirai create mode 100644 demo/packages/caae575f45863b717aafbe7fe73862c0.mirai create mode 100644 demo/packages/de7548cbbad3494b21280744cd16ed68.mirai create mode 100644 demo/packages/e022a803a1c6fb9fe4b86ab65a48bc24.mirai create mode 100644 demo/packages/e0495081646e8954b731fc082bcd8884.mirai create mode 100644 demo/packages/e96915c067e6fe50d517b59bf3b17d61.mirai create mode 100644 demo/packages/f4ccf188ab1bf19a62c53faaef829fba.mirai delete mode 100644 editor/assets/presence/source/pup.glb.import create mode 100644 plugins/prompts_kenney/packages.mirai rename plugins/prompts_kenney/{assets/kenney.prompts.json.import => packages/18e1ec191e6b7a7386eba31ba4c8e829.mirai} (88%) delete mode 100644 templates/game/assets/arena/hum.wav.import create mode 100644 templates/game/packages.mirai create mode 100644 templates/game/packages/ea1313ae2ad84fe489d5bf4061c5fda9.mirai diff --git a/.agents/skills/mirai-assets/SKILL.md b/.agents/skills/mirai-assets/SKILL.md index a73211151..97351da02 100644 --- a/.agents/skills/mirai-assets/SKILL.md +++ b/.agents/skills/mirai-assets/SKILL.md @@ -27,8 +27,9 @@ types, scenes and their prefab instances, registry). [container](references/formats.md#container), [typed tables](references/formats.md#typed-tables), [asset types](references/formats.md#asset-types), [versions](references/formats.md#versions), [registry](references/formats.md#registry) and [text](references/formats.md#text). -- The records of imports in the machine store (package and build manifests, `packages.mirai`), - their refs and `packages_resolve`: [packages](references/formats.md#packages). +- The records of imports in the machine store (package and build manifests, `packages.mirai` + and the manifest files a root carries), their refs and `packages_resolve`: + [packages](references/formats.md#packages). - The collection `compressor`, profiles, codecs, streams and decoders: [compression](references/formats.md#compression). - Streams, schemas, migration, paths, text: [serial](references/serial.md#stdserial). @@ -133,9 +134,9 @@ types, scenes and their prefab instances, registry). - **The header says Scene or Prefab.** One `.scene` extension and kind; `Header_Flag.Prefab` is what whoever made the file chose, never guessed from its roots, and a prefab has exactly one root ([prefab](references/scenes.md#prefab)). -- **A project pins the manifest of each package it uses, never a build**; builds are found by - key through refs, and resolving a list never removes nor rebuilds - ([packages](references/formats.md#packages)). +- **A project pins the manifest of each package it uses, never a build**, and carries the text + of each manifest it pins; builds are found by key through refs, and resolving a list never + removes nor rebuilds ([packages](references/formats.md#packages)). - **A guest mount never takes an id the registry has**: what the program shipped wins over what a player adds ([guest mounts](references/formats.md#registry)). @@ -163,8 +164,9 @@ types, scenes and their prefab instances, registry). - the document of a scene or a `.ui` from a file outside the assets (what `dump` and git show): `the_commands_read_write_and_check_the_document_of_an_asset` of `cook/author`; - records of packages: `package_records_read_back_what_they_write`, - `packages_resolve_lists_the_blobs_of_what_a_project_uses` and - `package_records_refuse_hostile_text` of `asset/packages_test.odin`; + `packages_resolve_lists_the_blobs_of_what_a_project_uses`, + `package_records_refuse_hostile_text` and `a_root_carries_the_manifests_it_pins` of + `asset/packages_test.odin`; - registry: `asset/registry_test.odin` and, in the editor, `editor/content_test.odin`; - layers of an imported material: `asset/kinds_test.odin`, and the overrides shared with prefabs in the editor, `editor/prefabs_test.odin`; diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index 857c2b287..9db227c1b 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -279,30 +279,39 @@ scan opens every file and writes it again. ## Packages The records of the imports kept in the [machine store](../../mirai-store/SKILL.md) -(`asset/packages.odin`), three texts with a version each. They are written beside the -sidecars; the importers, the registry and the build do not use them yet. +(`asset/packages.odin`), three texts with a version each. - **Package manifest** (`Package`): a blob, named by `packages/` on the machine - that ingested it. The import that ingests a source writes it. + that ingested it. The import that ingests a source writes it. A root that pins it also keeps + its text, versioned, in `packages/.mirai` beside `packages.mirai` + (`package_file`). - **Build manifest** (`Build`): a blob, named by `imports/` (Final, which peers share) or `imports-fast/` (never shared) by the import that builds it, and by `builds/` by the refresh of a project that chose it. -- **Project list** (`Packages`): `packages.mirai` beside `project.mirai`, versioned. The editor - and the commands that add or remove a package write it. +- **Project list** (`Packages`): `packages.mirai` beside `project.mirai`, versioned. The library + (`project_pin`, `project_unpin`) writes it and the manifest files, and nothing else does. - **A project pins the manifest, never a build.** The manifest holds what the source is (its `identity`), where it was (`origin`, a hint) and each asset with its id, and nothing that depends on the engine version: an engine that changes a version leaves the list as it is. +- **A root carries the manifests it pins.** The address a pin names is that of its manifest + file's bytes, so a clone or a copy of the root, with an empty store, has every id and setting; + only builds need the store, a source or a peer. `package_load` takes a manifest the store + lacks from that file and puts it in the store; a file that is not its pin's bytes is a problem + of that package, also when the store has the manifest. +- **The origin of a source in a root is relative to it** (`assets/...`, from the root whose + assets folder holds it), so every copy of the root writes the same manifest and finds the + source; a source in no root keeps its absolute path. - **Builds are found by name.** `import_key` combines the identity with the versions of what imports; any machine with the manifest computes it, without the source, and a peer that built that key gives it. `build_select` combines the manifest with the version of each kind of the cache it holds, so a program without importers (the dev player) finds the build the refresh chose. The caches of `cook/cache` name theirs in `cache-`. - `packages_resolve` lists, for each package of the list, its assets of the cache the project - uses with what they depend on, at `/` as the sidecar lists them, with - the path of their blob. A missing build lists them not built; a manifest or a build the store - lacks or does not read, and an asset the manifest lacks, are problems of the result. It - removes, builds and fetches nothing. + uses with what they depend on, at `/`, with the path of their blob. A + missing build lists them not built; a manifest that neither the store nor the root has, that + does not read or whose file is not its pin, a build that does not read and an asset the + manifest lacks are problems of the result. It removes, builds and fetches nothing. - An asset id is random at the first ingest and the manifest keeps it; an ingest again takes each id from the manifest before it. diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index 7b23b6a45..38aad5161 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -240,7 +240,9 @@ the assets. removes nor builds; a source outside any project ingests once and two projects list the same blobs; an engine change ingests again where the source is and is reported by name elsewhere, also when only Fast is here and Final is required; hostile `packages.mirai` and manifests are - reported; two ingests of one source at once import it once) and + reported; two ingests of one source at once import it once; a copy of a root with its sources + and an empty store builds the same ids, manifests and blobs, and without them reports each + package) and `cook/import/sources/reimport_test.odin` (a refresh reads a source again with the formats of the plugins, and ingests a whole source again when what it is built from changes). - `cook/import/sources/cli_test.odin` needs `MIRAI_ROOT` (defined by `mirai test`); run it when diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index 02f900cc1..c7c705c44 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -280,8 +280,9 @@ what surely belongs to that model is a part; what is doubtful stays independent The library of the machine is what its store keeps of imports and what a project does with it (`cook/import/sources/library.odin`). A project names the packages it uses in its -`packages.mirai` ([packages](../../mirai-assets/references/formats.md#packages)); the store, -shared by every checkout, worktree and project of the machine, keeps their manifests and builds. +`packages.mirai` and carries the text of their manifests beside it +([packages](../../mirai-assets/references/formats.md#packages)); the store, shared by every +checkout, worktree and project of the machine, keeps manifests and builds. - **`ingest(origin, options, settings)`** imports a file or a folder from wherever it is: in the assets of a project or a plugin, or in no project. Nothing is copied: the package goes to the @@ -292,15 +293,19 @@ shared by every checkout, worktree and project of the machine, keeps their manif or worktrees that ingest one origin take turns with a lock per origin (`cook/workspace`, under `/locks/`), and the second finds the work done. - **`project_add(root, manifest, folder, assets, options)`** pins a package in the project - (`project_pin`, the only writer of `packages.mirai`) and follows it there. **Following** writes + (`project_pin`, which with `project_unpin` is the only writer of `packages.mirai` and of the + manifest files beside it) and follows it there. **Following** writes or follows its documents under `folder` of the assets: from the baselines of its build (`documents_follow`), and for a model whose source is here and whose game types the library of the project's editor gives, by importing it with that folder, which also writes its prefab (`prefab.model_prefab`). Without the source or those types a model's prefab cannot be written, and the report says so. - **`project_refresh(root, options)`** takes each package the project pins to a build this - engine reads: the key (`asset.import_key` of the identity and the versions of what reads its - origin, an engine importer or a format of the engine or of the plugins) is looked up in + engine reads. A manifest the store lacks comes from the root's file of it, else from the + caller's source; so a clone with its sources and an empty store ingests every package again + with the same ids, manifest and key, and one without them reports each package. The key + (`asset.import_key` of the identity and the versions of what reads its origin, an engine + importer or a format of the engine or of the plugins) is looked up in `imports` (and `imports-fast` with `fast_textures`), then asked of the caller's source (`cache.Source`, the same procedure value the caches of `cook/cache` take: given `(imports, )` it makes the build and every blob it lists present, given diff --git a/.agents/skills/mirai-projects/references/build.md b/.agents/skills/mirai-projects/references/build.md index c17edc4c7..8a0b5f365 100644 --- a/.agents/skills/mirai-projects/references/build.md +++ b/.agents/skills/mirai-projects/references/build.md @@ -211,7 +211,7 @@ the published generation (no Slang sources; `mirai build game` generates the sha SDL3 if the system does not provide it. The copy replaces files and merges folders without deleting anything from the destination, except -`assets/`, which ends up exactly with what was cooked, without the sources and sidecars among the +`assets/`, which ends up exactly with what was cooked, without the sources among the assets. The code, `cook.mirai` and the `plugin.mirai` are not copied. `mirai play` copies nothing: `mirai-player` runs with `--project ` and reads assets and shaders from the project. @@ -227,9 +227,9 @@ Settings > Input changed, and SDL3, which the executable links. ## Imported sources -The meshes, textures, sounds and fonts an import builds are not versioned: they are in -`.mirai/imported/` of the project and of each plugin with assets, and are built again from the -sources and their sidecars ([library](../../mirai-imports/references/imports.md#library)). +The meshes, textures, sounds and fonts an import builds are not versioned: they are blobs of the +machine store, built again from the sources and the package manifests a project or plugin pins +([library](../../mirai-imports/references/imports.md#library)). - **A build imports, and says so.** Every build of a project runs the refresh of the project and of its active plugins once its target stands (`job_imports`, `build.imports_refresh`): the editor, diff --git a/.agents/skills/mirai-projects/references/projects.md b/.agents/skills/mirai-projects/references/projects.md index b93a37d1e..2467dcad2 100644 --- a/.agents/skills/mirai-projects/references/projects.md +++ b/.agents/skills/mirai-projects/references/projects.md @@ -12,9 +12,10 @@ Targets, jobs, fingerprint, project shaders, cooking and shipped game (`cook/bui cook.mirai cooking roots that no scene names (optional) packages.mirai the packages of the machine store the project uses, by the hash of their manifest, and the folder each is listed in (optional; a plugin may have one) + packages/ .mirai: the text of each manifest packages.mirai pins (settings, + origin and asset ids), so a clone has them without the store assets/ assets, by theme: what is authored, the art sources beside what they make, - each with its sidecar (.import: settings, folder and asset ids), and - the documents their imports write + and the documents their imports write game/ game library of the project editor/ editor package of the project (engine-time) plugins// plugin.mirai, runtime/, editor/, shaders/, assets/ diff --git a/asset/packages.odin b/asset/packages.odin index 070d0f75b..3e1848715 100644 --- a/asset/packages.odin +++ b/asset/packages.odin @@ -22,14 +22,17 @@ import "mirai:store" // importer finds. // - Packages: the list of a project (PACKAGES_FILE, beside its manifest), // versioned: the packages it uses by the hash of their manifest, the folder -// of its assets each is listed in and the assets it uses. The editor and -// the commands that add or remove a package write it. +// of its assets each is listed in and the assets it uses. Beside it, in +// PACKAGE_FILES, the text of each manifest it pins (package_file), so the +// root carries what its pins name: a store without a manifest takes it from +// there. The library of cook/import/sources writes both. // // What a parse gives lives in its allocator, an arena: nothing frees it piece // by piece. A text that is not a record of its version is an error, never a // default. PACKAGES_FILE :: "packages.mirai" +PACKAGE_FILES :: "packages" // the folder of a root, beside PACKAGES_FILE, with .mirai of each manifest it pins PACKAGES_VERSION :: 1 PACKAGE_MANIFEST_VERSION :: 1 BUILD_MANIFEST_VERSION :: 1 @@ -45,10 +48,10 @@ REF_ORIGINS :: "origins" // : the manifest of the package REF_SOURCES :: "sources" // : what the files of that source were when last read, never shared IMPORT_KEY_VERSIONS :: 32 // the most versions an import key combines -// Where the source of a package was on the machine that ingested it: a hint -// to ingest it again there, not part of what it is. +// Where the source of a package was when it was ingested: a hint to ingest it +// again, not part of what it is. Package_Origin :: struct { - path:string, + path:string, // relative to the root whose assets hold it, alike in every copy of it; else absolute folder:bool, // the source is a folder (an Unreal package), not a file importer:string, // the format or the importer of a plugin that reads it } @@ -431,15 +434,11 @@ packages_parse :: proc(text:string,allocator:=context.allocator) -> (list:Packag packages_load :: proc(root:string,allocator:=context.allocator) -> (list:Packages,found:bool,error:Error) { path:=fmt.tprintf("%s/%s",root,PACKAGES_FILE) data,read_error:=serial.file_read(path,context.temp_allocator,limit=RECORD_BYTES) - switch read_error { - case .None: - case .Not_Found: + if read_error==.Not_Found { list.packages=make([dynamic]Project_Package,allocator) return list,false,.None - case .Too_Large: return {},false,.Too_Large - case .Out_Of_Memory: return {},false,.Out_Of_Memory - case .Open,.Read,.Write,.Rename,.Invalid_Alignment: return {},false,.IO } + if read_error!=.None {return {},false,file_error(read_error)} list,error=packages_parse(string(data),allocator) return list,error==.None,error } @@ -449,12 +448,63 @@ packages_save :: proc(root:string,list:^Packages) -> Error { text:=packages_text(list,context.temp_allocator) or_return path:=fmt.tprintf("%s/%s",root,PACKAGES_FILE) if data,read_error:=os.read_entire_file(path,context.temp_allocator);read_error==nil && string(data)==text {return .None} - switch serial.file_write_atomic(path,transmute([]u8)text) { - case .None: return .None - case .Out_Of_Memory: return .Out_Of_Memory - case .Not_Found,.Open,.Read,.Write,.Rename,.Too_Large,.Invalid_Alignment: return .IO + return file_error(serial.file_write_atomic(path,transmute([]u8)text)) +} + +// The file of root that holds the text of the manifest of the package id it +// pins, in the temp allocator. +package_file :: proc(root:string,id:base.Asset_Id) -> string { + buffer:[base.ASSET_ID_TEXT]u8 + return fmt.tprintf("%s/%s/%s.mirai",root,PACKAGE_FILES,base.asset_id_format(id,buffer[:])) +} + +// Writes the manifest the store keeps under entry.manifest to the file of +// root for its package, unless the file already holds it. .Corrupt when that +// blob is not a manifest of the package entry names. +package_file_save :: proc(s:^store.Store,root:string,entry:^Project_Package) -> Error { + text:=record_get(s,entry.manifest) or_return + p:=package_parse(text,context.temp_allocator) or_return + if p.id!=entry.id {return .Corrupt} + path:=package_file(root,entry.id) + if data,read_error:=os.read_entire_file(path,context.temp_allocator);read_error==nil && string(data)==text {return .None} + if folder:=fmt.tprintf("%s/%s",root,PACKAGE_FILES);!os.is_dir(folder) && os.make_directory_all(folder)!=nil {return .IO} + return file_error(serial.file_write_atomic(path,transmute([]u8)text)) +} + +// Removes the file of root for the package id; .None also when it has none. +package_file_remove :: proc(root:string,id:base.Asset_Id) -> Error { + path:=package_file(root,id) + if !os.exists(path) {return .None} + return .None if os.remove(path)==nil else .IO +} + +// The manifest a root pins with entry: the one the store keeps, else the file +// of root for it (package_file), which then goes into the store. A file that +// is there must hold the bytes of the address entry pins: .Hash when it does +// not, also when the store has the manifest. .Not_Found when neither has it. +// root "": the store alone. In allocator, an arena. +package_load :: proc(s:^store.Store,root:string,entry:^Project_Package,allocator:=context.allocator) -> (p:Package,error:Error) { + file:[]u8 + kept:=false + if root!="" { + read_error:serial.File_Error + file,read_error=serial.file_read(package_file(root,entry.id),context.temp_allocator,limit=RECORD_BYTES) + #partial switch read_error { + case .None: + if store.address_of_bytes(file)!=entry.manifest {return {},.Hash} + kept=true + case .Not_Found: + case: return {},file_error(read_error) + } } - return .IO + p,error=package_get(s,entry.manifest,allocator) + if error==.Not_Found && kept { + p,error=package_parse(string(file),allocator) + if error!=.None {return {},error} + if _,put_error:=store.put_bytes(s,file);put_error!=.None {return {},store_error(put_error)} + } + if error==.None && p.id!=entry.id {return {},.Corrupt} + return } // Keeps a manifest in the store, by the hash of its text. @@ -499,8 +549,9 @@ Package_Listed :: struct { } Package_Problem_Code :: enum u8 { - Manifest_Missing, // the store has no blob of the manifest the project names - Manifest_Unreadable, // it has one that is not a manifest of this version, or of another package + Manifest_Missing, // neither the store nor the root has the manifest the project names + Manifest_Unreadable, // the one there is not a manifest of this version, or of another package + Manifest_Mismatch, // the file of the root for it (package_file) is not the manifest it pins Build_Unreadable, // the build its ref names is missing or does not read: its assets are not built Asset_Unknown, // the project uses an asset its manifest does not have } @@ -519,23 +570,26 @@ Packages_Resolved :: struct { // What a project lists from the packages it uses: each asset of the cache it // uses, with what they depend on, at its path, and the blob of the build // REF_BUILDS names for its manifest, or Not_Built. Documents are files of the -// project and are not listed. A manifest or a build the store does not have -// or does not read is a problem of the result, never an error; nothing is -// removed, built or fetched. Two packages that list one path take the name of +// project and are not listed. The manifests come from the store or from the +// files of root (package_load; "": the store alone). A manifest or a build +// that is not there or does not read is a problem of the result, never an +// error; nothing is removed, built or fetched. Two packages that list one path take the name of // their package after it, " ()". In allocator, an arena; error // only when memory runs out. -packages_resolve :: proc(s:^store.Store,list:^Packages,allocator:=context.allocator) -> (resolved:Packages_Resolved,error:Error) { +packages_resolve :: proc(s:^store.Store,root:string,list:^Packages,allocator:=context.allocator) -> (resolved:Packages_Resolved,error:Error) { resolved.listed=make([dynamic]Package_Listed,allocator) resolved.problems=make([dynamic]Package_Problem,allocator) names:=make([dynamic]string,context.temp_allocator) for &entry in list.packages { - p,manifest_error:=package_get(s,entry.manifest,context.temp_allocator) - if manifest_error==.Out_Of_Memory {return {},.Out_Of_Memory} - if manifest_error==.None && p.id!=entry.id {manifest_error=.Corrupt} - if manifest_error!=.None { - problem_add(&resolved,{entry.id,{},manifest_error==.Not_Found?.Manifest_Missing:.Manifest_Unreadable}) or_return - continue + p,manifest_error:=package_load(s,root,&entry,context.temp_allocator) + #partial switch manifest_error { + case .None: + case .Out_Of_Memory: return {},.Out_Of_Memory + case .Not_Found: problem_add(&resolved,{entry.id,{},.Manifest_Missing}) or_return + case .Hash: problem_add(&resolved,{entry.id,{},.Manifest_Mismatch}) or_return + case: problem_add(&resolved,{entry.id,{},.Manifest_Unreadable}) or_return } + if manifest_error!=.None {continue} used,no_used:=make([]bool,len(p.assets),context.temp_allocator) by_id:=make(map[base.Asset_Id]int,context.temp_allocator) if no_used!=nil || reserve(&by_id,len(p.assets))!=nil {return {},.Out_Of_Memory} @@ -948,6 +1002,18 @@ record_get :: proc(s:^store.Store,hash:store.Hash) -> (text:string,error:Error) return string(bytes),.None } +@(private="file") +file_error :: proc(error:serial.File_Error) -> Error { + switch error { + case .None: return .None + case .Too_Large: return .Too_Large + case .Out_Of_Memory: return .Out_Of_Memory + case .Not_Found: return .Not_Found + case .Open,.Read,.Write,.Rename,.Invalid_Alignment: return .IO + } + return .IO +} + @(private="file") store_error :: proc(error:store.Error) -> Error { switch error { diff --git a/asset/packages_root.odin b/asset/packages_root.odin index 540f1952f..1a47d705d 100644 --- a/asset/packages_root.odin +++ b/asset/packages_root.odin @@ -14,7 +14,8 @@ Root_Packages :: struct { } // packages_resolve of the list at root (PACKAGES_FILE) against the store at -// store_root, the machine's (store.default_root) when "". A root without the +// store_root, the machine's (store.default_root) when "", which takes the +// manifests it lacks from the files of the root. A root without the // list lists nothing and opens no store. A list or a store that does not read // is a problem, as is a package that is not here or not built here: none is an // error, which is only that memory ran out. In allocator, an arena. @@ -43,15 +44,17 @@ root_packages :: proc(root,store_root:string,allocator:=context.allocator) -> (r return {problems=problems[:]},.None } defer store.close(&s) - packages:=packages_resolve(&s,&list,allocator) or_return + packages:=packages_resolve(&s,root,&list,allocator) or_return unreadable:=make(map[base.Asset_Id]bool,context.temp_allocator) for problem in packages.problems { named:=package_named(&s,&list,problem.package_id) switch problem.code { case .Manifest_Missing: - problem_line(&problems,"%s is not in the store at %s: ingest its source on this machine, or fetch it from a peer that has it",named,folder) or_return + problem_line(&problems,"%s is neither in the store at %s nor in %s: restore that file from version control, or fetch the manifest from a peer that has it",named,folder,package_file(root,problem.package_id)) or_return case .Manifest_Unreadable: - problem_line(&problems,"the manifest of %s in the store at %s is not one this engine reads: ingest its source again",named,folder) or_return + problem_line(&problems,"the manifest of %s is not one this engine reads: ingest its source again",named) or_return + case .Manifest_Mismatch: + problem_line(&problems,"%s is not the manifest %s/%s pins for %s: restore both from version control, or ingest its source again",package_file(root,problem.package_id),root,PACKAGES_FILE,named) or_return case .Build_Unreadable: unreadable[problem.package_id]=true problem_line(&problems,"the build of %s on this machine does not read, so its assets are not built: refresh it on a machine with its source",named) or_return diff --git a/asset/packages_test.odin b/asset/packages_test.odin index 48420fa8f..bf38dcb0b 100644 --- a/asset/packages_test.odin +++ b/asset/packages_test.odin @@ -156,7 +156,7 @@ packages_resolve_lists_the_blobs_of_what_a_project_uses :: proc(t:^testing.T) { Project_Package{id={40,40},manifest=hash_of("absent")}, Project_Package{id=broken.id,manifest=broken_manifest}, ) - resolved,error:=packages_resolve(&s,&list,context.temp_allocator) + resolved,error:=packages_resolve(&s,"",&list,context.temp_allocator) if !testing.expect_value(t,error,Error.None) {return} mesh_path,_:=store.blob_path(&s,mesh,context.temp_allocator) texture_path,_:=store.blob_path(&s,texture,context.temp_allocator) @@ -294,3 +294,63 @@ package_records_refuse_hostile_text :: proc(t:^testing.T) { _,found,load_error:=packages_load(folder,context.temp_allocator) testing.expect(t,!found && load_error==.Corrupt,"a damaged list is an error, never an empty one") } + +// A root carries the text of each manifest it pins: a store without one takes +// it from the root's file and lists its assets not built, and root_packages +// says so by package. A file that is not the manifest its pin names, or that +// does not read, is reported by package, never listed; with neither a file +// nor the manifest in the store, the package is missing. +@(test) +a_root_carries_the_manifests_it_pins :: proc(t:^testing.T) { + folder:=base.test_folder(t,"packages-files");if folder=="" {return} + ingested,open_error:=store.open(fmt.tprintf("%s/ingested",folder),context.temp_allocator) + if !testing.expect_value(t,open_error,store.Error.None) {return} + fresh,fresh_error:=store.open(fmt.tprintf("%s/fresh",folder),context.temp_allocator) + if !testing.expect_value(t,fresh_error,store.Error.None) {return} + root:=fmt.tprintf("%s/game",folder) + p:=sample_package() + manifest,_:=package_put(&ingested,&p) + list:Packages + list.packages=make([dynamic]Project_Package,context.temp_allocator) + append(&list.packages,Project_Package{id=p.id,manifest=manifest,folder="kit"}) + testing.expect_value(t,package_file_save(&ingested,root,&list.packages[0]),Error.None) + testing.expect_value(t,packages_save(root,&list),Error.None) + wrong:=Project_Package{id={8,8},manifest=manifest} + testing.expect_value(t,package_file_save(&ingested,root,&wrong),Error.Corrupt) + testing.expect(t,!os.exists(package_file(root,wrong.id)),"a manifest of another package is not written as this one's") + + resolved,error:=packages_resolve(&fresh,root,&list,context.temp_allocator) + if !testing.expect_value(t,error,Error.None) {return} + testing.expectf(t,len(resolved.problems)==0 && len(resolved.listed)==4,"%v %v",resolved.problems,resolved.listed) + for l in resolved.listed {testing.expectf(t,l.state==.Not_Built,"%s is not built without a build",l.path)} + testing.expect(t,store.has(&fresh,manifest),"the manifest of the root goes into the store, at the address it pins") + lines,_:=root_packages(root,fresh.root,context.temp_allocator) + testing.expectf(t,len(lines.listed)==4 && len(lines.problems)==1 && strings.contains(lines.problems[0],"crate"),"not built, said by package: %v",lines.problems) + testing.expect_value(t,store.remove(&fresh,manifest),store.Error.None) + + file:=package_file(root,p.id) + text,_:=package_text(&p,context.temp_allocator) + hostile,_:=strings.replace(text,"crate","crane",1,context.temp_allocator) + garbage:="{\"version\": 1, \"id\": 7}" + Case :: struct {bytes:string,pin:store.Hash,want:Package_Problem_Code} + cases:=[]Case{ + {hostile,manifest,.Manifest_Mismatch}, + {garbage,hash_of(garbage),.Manifest_Unreadable}, + {strings.repeat(" ",RECORD_BYTES+1,context.temp_allocator),manifest,.Manifest_Unreadable}, + {"",manifest,.Manifest_Missing}, + } + for c,at in cases { + if c.bytes=="" {os.remove(file)} else {testing.expect(t,os.write_entire_file(file,transmute([]u8)c.bytes)==nil)} + list.packages[0].manifest=c.pin + got,got_error:=packages_resolve(&fresh,root,&list,context.temp_allocator) + if !testing.expect_value(t,got_error,Error.None) {continue} + testing.expectf(t,len(got.listed)==0 && len(got.problems)==1 && got.problems[0].package_id==p.id && got.problems[0].code==c.want,"case %d: %v",at,got.problems) + testing.expect_value(t,packages_save(root,&list),Error.None) + said,_:=root_packages(root,fresh.root,context.temp_allocator) + id:[base.ASSET_ID_TEXT]u8 + testing.expectf(t,len(said.problems)==1 && strings.contains(said.problems[0],base.asset_id_format(p.id,id[:])),"case %d is said by package: %v",at,said.problems) + } + testing.expect(t,!store.has(&fresh,manifest),"a file that is not its pin goes into no store") + testing.expect_value(t,package_file_remove(root,p.id),Error.None) + testing.expect_value(t,package_file_remove(root,p.id),Error.None) +} diff --git a/cook/import/identity.odin b/cook/import/identity.odin index f675b67e7..84bbecc44 100644 --- a/cook/import/identity.odin +++ b/cook/import/identity.odin @@ -260,6 +260,40 @@ path_key :: proc(path:string) -> string { return strings.clone(slashed,context.temp_allocator) } +// The project or the plugin whose assets folder has the file at path: the +// folder that holds the nearest assets folder above it. False for a file in +// no assets folder. +source_root :: proc(path:string,allocator:=context.allocator) -> (root:string,ok:bool) { + names:=project.FOLDER_NAMES + folder:=project.path_absolute(path,context.temp_allocator) + for { + parent:=filepath.dir(folder) + if parent==folder {return} + if filepath.base(parent)==names[.Assets] {return strings.clone(filepath.dir(parent),allocator),true} + folder=parent + } +} + +// Where a manifest says its source is (asset.Package_Origin.path), for the +// source at origin (path_key): relative to the root whose assets hold it +// (source_root), so every copy of that root names it alike; else origin. In +// the temp allocator. +origin_of :: proc(origin:string) -> string { + root,inside:=source_root(origin,context.temp_allocator) + under:=fmt.tprintf("%s/",path_key(root)) + if !inside || !strings.has_prefix(origin,under) {return origin} + return origin[len(under):] +} + +// The source the origin of a manifest names, as path_key gives it: a relative +// one under root, the project or plugin that pins its package. "" for a +// relative one without a root. In the temp allocator. +origin_path :: proc(origin,root:string) -> string { + if origin=="" || filepath.is_abs(origin) || strings.has_prefix(origin,"/") {return origin} + if root=="" {return ""} + return path_key(fmt.tprintf("%s/%s",root,origin)) +} + @(private) kind_name :: proc(kind:asset.Fourcc) -> string { // The scene an import writes has one root: the prefab of its model. diff --git a/cook/import/outputs_test.odin b/cook/import/outputs_test.odin index 18a17e13b..0504302c4 100644 --- a/cook/import/outputs_test.odin +++ b/cook/import/outputs_test.odin @@ -374,7 +374,7 @@ a_fast_build_is_not_published_where_peers_find_builds :: proc(t:^testing.T) { testing.expect(t,chosen==report.build,"the build this machine chose") list:=asset.Packages{packages=make([dynamic]asset.Project_Package,context.temp_allocator)} append(&list.packages,asset.Project_Package{id=p.id,manifest=report.manifest}) - resolved,resolve_error:=asset.packages_resolve(s,&list,context.temp_allocator) + resolved,resolve_error:=asset.packages_resolve(s,"",&list,context.temp_allocator) testing.expect(t,resolve_error==.None && len(resolved.problems)==0 && len(resolved.listed)==2,"a project lists the mesh and the texture of the package") for l in resolved.listed {testing.expectf(t,l.state==.Built && os.exists(l.file),"%s is built",l.path)} } diff --git a/cook/import/package.odin b/cook/import/package.odin index be08b8e6e..767c4ab47 100644 --- a/cook/import/package.odin +++ b/cook/import/package.odin @@ -193,7 +193,7 @@ package_previous :: proc(s:^store.Store,manifest:store.Hash,origin:string,alloca named,ref_error:=store.ref_get(s,asset.REF_ORIGINS,asset.origin_name(origin,&name)) if ref_error!=.None {return} read_error:asset.Error - if p,read_error=asset.package_get(s,named,allocator);read_error!=.None || p.origin.path!=origin {return {},{},false,{}} + if p,read_error=asset.package_get(s,named,allocator);read_error!=.None || p.origin.path!=origin_of(origin) {return {},{},false,{}} return p,named,true,{} } @@ -232,7 +232,7 @@ writer_publish :: proc(w:^Writer,name,importer:string) -> bool { s:=w.options.store writer_identity(w) p:=asset.Package{id=w.package_id,name=name,identity=w.identity,inputs=w.inputs,assets=w.outputs} - p.origin={path=w.origin,folder=os.is_dir(w.path),importer=importer} + p.origin={path=origin_of(w.origin),folder=os.is_dir(w.path),importer=importer} if settings:=settings_text(w.report.settings,context.temp_allocator);settings!="{}" {p.settings=settings} slice.sort(p.inputs[:]) b:=asset.Build{package_id=w.package_id,key=w.key,final=!w.fast,encoders=textures.encoders_id()} diff --git a/cook/import/sources/command.odin b/cook/import/sources/command.odin index 8384fd966..5c190e3a2 100644 --- a/cook/import/sources/command.odin +++ b/cook/import/sources/command.odin @@ -195,7 +195,7 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in origin:=importer.path_key(file) // The project it is added to, and where under its assets. if root=="" { - if inside,in_assets:=source_root(origin,context.temp_allocator);in_assets { + if inside,in_assets:=importer.source_root(origin,context.temp_allocator);in_assets { root=inside if !foldered { assets:=project.folder_path(root,.Assets,context.temp_allocator) diff --git a/cook/import/sources/library.odin b/cook/import/sources/library.odin index 4b86b360f..dbf7fa35e 100644 --- a/cook/import/sources/library.odin +++ b/cook/import/sources/library.odin @@ -26,14 +26,16 @@ import "mirai:store" // identity and build are current is not imported again. Two processes that // ingest one origin at once take turns (a lock per origin), and the second // finds the work done. -// - project_add: pins a package in the project's packages.mirai -// (project_pin, its only writer) and follows it into the project. +// - project_add: pins a package in the project's packages.mirai, with the +// text of its manifest beside it (project_pin and project_unpin, their only +// writers), and follows it into the project. // - project_refresh: for each package a project pins, the build this engine // reads, from the local refs, else from the caller's source, else by // ingesting its origin when it is on this machine (also when the source // changed); else the report says, by package name, that a machine with its -// source must ingest it. It never removes a blob, a ref or a file, and never -// builds without the source. +// source must ingest it. A manifest the store lacks comes from the +// project's file of it, else from the caller's source. It never removes a +// blob, a ref or a file, and never builds without the source. // // Following a package writes or follows its documents in the project's folder // of it: from the build's baselines (importer.documents_follow), and for a @@ -134,7 +136,7 @@ project_add :: proc(root:string,manifest:store.Hash,folder:string,assets:[]base. } append(&entry.assets,id) } - if pin_error:=project_pin(top,entry);pin_error!=.None { + if pin_error:=project_pin(top,options.store,entry);pin_error!=.None { report_line(&report.failures,"%s: %s could not be written (%v): a folder is relative to the assets, with forward slashes, and never leaves them",top,asset.PACKAGES_FILE,pin_error) return } @@ -171,7 +173,7 @@ project_ingest :: proc(root,origin,folder:string,options:Library_Options,setting for &other in list.packages { if other.id==entry.id {entry.assets=other.assets} } - if pin_error:=project_pin(top,entry);pin_error!=.None { + if pin_error:=project_pin(top,options.store,entry);pin_error!=.None { report_line(&added.failures,"%s: %s could not be written (%v): a folder is relative to the assets, with forward slashes, and never leaves them",top,asset.PACKAGES_FILE,pin_error) return } @@ -206,10 +208,14 @@ project_refresh :: proc(root:string,options:Library_Options,allocator:=context.a } // Pins entry in the packages.mirai of the project at root, in place of the -// entry of the same package: the only writer of that file. -project_pin :: proc(root:string,entry:asset.Project_Package) -> asset.Error { +// entry of the same package, and writes beside it the text of the manifest +// that s keeps under entry.manifest (asset.package_file). It and +// project_unpin are the only writers of those files. +project_pin :: proc(root:string,s:^store.Store,entry:asset.Project_Package) -> asset.Error { + entry:=entry list,_,error:=asset.packages_load(root,context.temp_allocator) if error!=.None {return error} + asset.package_file_save(s,root,&entry) or_return for &other in list.packages { if other.id==entry.id { other=entry @@ -220,6 +226,20 @@ project_pin :: proc(root:string,entry:asset.Project_Package) -> asset.Error { return asset.packages_save(root,&list) } +// Removes the package id from the packages.mirai of the project at root, and +// the file of its manifest; .None also when it does not pin it. +project_unpin :: proc(root:string,id:base.Asset_Id) -> asset.Error { + list,_,error:=asset.packages_load(root,context.temp_allocator) + if error!=.None {return error} + for &other,at in list.packages { + if other.id!=id {continue} + ordered_remove(&list.packages,at) + asset.packages_save(root,&list) or_return + break + } + return asset.package_file_remove(root,id) +} + // What ingest, project_add and project_refresh share for one call. @(private="file") Library_State :: struct { @@ -421,7 +441,7 @@ library_ingest :: proc(l:^Library_State,path:string,manifest:store.Hash,settings rebuild=l.options.rebuild,fast_textures=l.options.fast_textures,worker=l.options.worker,hashes=&hashes, } if documents!="" {options.registry,options.world=l.registry,l.plugins.world} - if options.root=="" {options.root,_=source_root(origin,l.temp)} + if options.root=="" {options.root,_=importer.source_root(origin,l.temp)} if settings!=nil {options.settings,options.ignore_stored=settings^,true} progress:importer.Progress if l.options.progress.procedure!=nil { @@ -443,24 +463,27 @@ package_refresh :: proc(l:^Library_State,top:string,entry:asset.Project_Package, entry:=entry digits,id_text:[64]u8 id_name:=base.asset_id_format(entry.id,id_text[:32]) - p,error:=asset.package_get(s,entry.manifest,l.temp) - if error==.Not_Found && fetch(l,asset.REF_PACKAGES,id_name) {p,error=asset.package_get(s,entry.manifest,l.temp)} - if error==.None && p.id!=entry.id {error=.Corrupt} + p,error:=asset.package_load(s,top,&entry,l.temp) + if error==.Not_Found && fetch(l,asset.REF_PACKAGES,id_name) {p,error=asset.package_load(s,top,&entry,l.temp)} #partial switch error { case .None: case .Not_Found: - report_line(&report.missing,"package %s: the store at %s has not its manifest %s, and no peer gave it: fetch it from a machine that ingested it",id_name,s.root,store.hex(entry.manifest,&digits)) + report_line(&report.missing,"package %s: neither the store at %s nor %s has its manifest %s, and no peer gave it: restore that file from version control",id_name,s.root,asset.package_file(top,entry.id),store.hex(entry.manifest,&digits)) + return + case .Hash: + report_line(&report.failures,"package %s: %s is not the manifest %s/%s pins: restore both from version control, or ingest its source again",id_name,asset.package_file(top,entry.id),top,asset.PACKAGES_FILE) return case: report_line(&report.failures,"package %s: its manifest %s in the store at %s is not a package of this version (%v): ingest its source again",id_name,store.hex(entry.manifest,&digits),s.root,error) return } - origin_here:=p.origin.path!="" && !importer.unsafe_path(p.origin.path) && os.exists(p.origin.path) + source:=importer.origin_path(p.origin.path,top) + origin_here:=source!="" && !importer.unsafe_path(source) && os.exists(source) documents:=l.assets if entry.folder=="" else fmt.tprintf("%s/%s",l.assets,entry.folder) prefab:=model_prefab_of(l,&p) // A source here that changed is ingested again, keeping its ids, and the project follows it. - if origin_here && source_changed(l,&p) { - reingest(l,top,&entry,&p,documents,prefab,"its source changed",report) + if origin_here && source_changed(l,&p,source) { + reingest(l,top,&entry,&p,source,documents,prefab,"its source changed",report) return } key,keyed:=package_key(l,&p) @@ -472,7 +495,7 @@ package_refresh :: proc(l:^Library_State,top:string,entry:asset.Project_Package, } if !found { if origin_here { - reingest(l,top,&entry,&p,documents,prefab,"this engine reads no build of it here",report) + reingest(l,top,&entry,&p,source,documents,prefab,"this engine reads no build of it here",report) return } switch { @@ -496,7 +519,7 @@ package_refresh :: proc(l:^Library_State,top:string,entry:asset.Project_Package, if !follow && previous==f.hash {return} // A model's prefab is written from its source in the game's types: the import does it, with the documents. if prefab && origin_here && plugins_world(l)!=nil { - reingest(l,top,&entry,&p,documents,prefab,"it is followed into the project",report) + reingest(l,top,&entry,&p,source,documents,prefab,"it is followed into the project",report) return } documents_write(l,&p,&f.build,documents,report) @@ -526,29 +549,29 @@ documents_write :: proc(l:^Library_State,p:^asset.Package,b:^asset.Build,documen } } -// Whether what the source of p is made from now is not what p was ingested -// from: its bytes, those of its inputs, or the code of the plugin that -// imports it whole. +// Whether what the source of p, here at source, is made from now is not what +// p was ingested from: its bytes, those of its inputs, or the code of the +// plugin that imports it whole. @(private="file") -source_changed :: proc(l:^Library_State,p:^asset.Package) -> bool { +source_changed :: proc(l:^Library_State,p:^asset.Package,source:string) -> bool { settings,valid:=importer.settings_parse(p.settings) if !valid {return false} code:u64 - if r,known:=reader_of(l,p.origin.path);known && r.claimed {code=r.whole.code} - hashes:=importer.source_hashes_load(l.options.store,p.origin.path,l.temp) - defer importer.source_hashes_save(l.options.store,p.origin.path,&hashes) - return importer.identity_of(p.id,p.origin.path,p.inputs[:],settings,code,&hashes)!=p.identity + if r,known:=reader_of(l,source);known && r.claimed {code=r.whole.code} + hashes:=importer.source_hashes_load(l.options.store,source,l.temp) + defer importer.source_hashes_save(l.options.store,source,&hashes) + return importer.identity_of(p.id,source,p.inputs[:],settings,code,&hashes)!=p.identity } -// Ingests the source of p again, keeping its ids, and pins the manifest it -// gives; a model is imported with the project's folder, which follows its +// Ingests the source of p again, here at source, keeping its ids, and pins +// the manifest it gives; a model is imported with the project's folder, which follows its // documents and writes its prefab, and any other follows from its build. @(private="file") -reingest :: proc(l:^Library_State,top:string,entry:^asset.Project_Package,p:^asset.Package,documents:string,prefab:bool,why:string,report:^Library_Report) { - if l.options.progress.procedure!=nil {fmt.wprintfln(l.options.progress,"importing %s: %s",p.origin.path,why)} +reingest :: proc(l:^Library_State,top:string,entry:^asset.Project_Package,p:^asset.Package,source,documents:string,prefab:bool,why:string,report:^Library_Report) { + if l.options.progress.procedure!=nil {fmt.wprintfln(l.options.progress,"importing %s: %s",source,why)} started:=time.tick_now() model:=prefab && plugins_world(l)!=nil - written,error:=library_ingest(l,p.origin.path,entry.manifest,nil,documents if model else "",l.temp) + written,error:=library_ingest(l,source,entry.manifest,nil,documents if model else "",l.temp) if error.code!=.None { report_line(&report.failures,"package %s: %s",p.name,error.message) return @@ -559,7 +582,7 @@ reingest :: proc(l:^Library_State,top:string,entry:^asset.Project_Package,p:^ass if l.options.progress.procedure!=nil {fmt.wprintfln(l.options.progress," %s: %d assets, %.1f s",p.name,len(written.assets),time.duration_seconds(time.tick_since(started)))} if written.manifest!=entry.manifest { entry.manifest=written.manifest - if pin_error:=project_pin(top,entry^);pin_error!=.None { + if pin_error:=project_pin(top,l.options.store,entry^);pin_error!=.None { report_line(&report.failures,"package %s: %s/%s could not be written: %v",p.name,top,asset.PACKAGES_FILE,pin_error) return } diff --git a/cook/import/sources/library_test.odin b/cook/import/sources/library_test.odin index 4839dbebb..1f31d1cbc 100644 --- a/cook/import/sources/library_test.odin +++ b/cook/import/sources/library_test.odin @@ -46,7 +46,7 @@ listed_of :: proc(t:^testing.T,s:^store.Store,root:string,loc:=#caller_location) listed=make(map[string]asset.Package_Listed,context.temp_allocator) list,_,error:=asset.packages_load(root,context.temp_allocator) if !testing.expect_value(t,error,asset.Error.None,loc=loc) {return} - resolved,resolve_error:=asset.packages_resolve(s,&list,context.temp_allocator) + resolved,resolve_error:=asset.packages_resolve(s,root,&list,context.temp_allocator) if !testing.expect_value(t,resolve_error,asset.Error.None,loc=loc) {return} testing.expectf(t,len(resolved.problems)==0,"%v",resolved.problems[:],loc=loc) for l in resolved.listed {listed[l.path]=l} @@ -117,11 +117,12 @@ a_peer_without_sources_lists_and_loads_and_nothing_is_removed_or_built :: proc(t lacking:=project_refresh(peer,alone,context.temp_allocator) testing.expectf(t,len(lacking.missing)==2 && len(lacking.failures)==0 && lacking.ingested==0 && lacking.fetched==0,"%v",lacking) list,_,_:=asset.packages_load(peer,context.temp_allocator) - for entry in list.packages { - id:[base.ASSET_ID_TEXT]u8 + for &entry in list.packages { + // The project carries its manifests: each package is named by its name. + p,_:=asset.package_get(here,entry.manifest,context.temp_allocator) named:=false - for line in lacking.missing {named||=strings.contains(line,base.asset_id_format(entry.id,id[:]))} - testing.expectf(t,named,"the package %v is named: %v",entry.id,lacking.missing[:]) + for line in lacking.missing {named||=strings.contains(line,fmt.tprintf("package %s:",p.name))} + testing.expectf(t,named,"the package %s is named: %v",p.name,lacking.missing[:]) } builds,_:=store.ref_list(there,asset.REF_BUILDS,context.temp_allocator) testing.expect(t,len(builds)==0,"nothing is built") @@ -264,6 +265,98 @@ an_engine_change_builds_again_where_the_source_is_and_is_reported_by_name_elsewh testing.expectf(t,len(project_refresh(game,fast,context.temp_allocator).missing)==1,"with Fast taken only the billboard, whose source is gone, is missing") } +// A root carries the manifests it pins: a copy of it elsewhere with its +// sources and an empty store refreshes into the same ids, manifests, build +// keys and blobs; a copy without its sources lists them not built and says so +// by package. Unpinning removes the manifest's file. +@(test) +a_copy_of_a_root_with_its_sources_and_an_empty_store_builds_the_same :: proc(t:^testing.T) { + top:=base.test_folder(t,"library-clone") + if top=="" {return} + context.user_ptr=prefab.import_world(t) + game:=fmt.tprintf("%s/game",top) + props:=fmt.tprintf("%s/assets/props",game) + os.make_directory_all(props) + box:=box_source(props,"box.gltf","Wood","Paint") + sign:=fmt.tprintf("%s/sign.png",props) + _=os.write_entire_file(sign,png_bytes(8,8,{200,40,40})) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + options:=Library_Options{store=here,root=game,plugins=test_library} + keys:=make([dynamic]store.Hash,context.temp_allocator) + for source in ([2]string{box,sign}) { + report,added,error:=project_ingest(game,source,"props",options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) || !report_clean(t,added) {return} + append(&keys,report.key) + } + pinned,_,_:=asset.packages_load(game,context.temp_allocator) + testing.expect_value(t,len(pinned.packages),2) + for &entry in pinned.packages { + text,read_error:=os.read_entire_file(asset.package_file(game,entry.id),context.temp_allocator) + testing.expectf(t,read_error==nil && store.address_of_bytes(text)==entry.manifest,"the root keeps the manifest it pins of %v",entry.id) + p,_:=asset.package_get(here,entry.manifest,context.temp_allocator) + testing.expectf(t,!strings.has_prefix(p.origin.path,"/") && strings.has_prefix(p.origin.path,"assets/props/"),"a source of the root is named from it: %s",p.origin.path) + } + blobs_of :: proc(t:^testing.T,s:^store.Store,root:string) -> map[string]store.Hash { + out:=make(map[string]store.Hash,context.temp_allocator) + for path,l in listed_of(t,s,root) { + hash,_,_:=store.address_of_file(l.file) + out[path]=hash + } + return out + } + before:=blobs_of(t,here,game) + + clone:=fmt.tprintf("%s/clone",top) + folder_copy(game,clone) + fresh:=test_store(t,fmt.tprintf("%s/fresh",top)) + if fresh==nil {return} + again:=options + again.store,again.root=fresh,clone + refreshed:=project_refresh(clone,again,context.temp_allocator) + testing.expectf(t,refreshed.ingested==2,"%v",refreshed) + if !report_clean(t,refreshed) {return} + cloned,_,_:=asset.packages_load(clone,context.temp_allocator) + testing.expect_value(t,len(cloned.packages),len(pinned.packages)) + for entry,at in cloned.packages { + testing.expectf(t,entry.id==pinned.packages[at].id && entry.manifest==pinned.packages[at].manifest,"the same package and manifest: %v",entry) + } + for key in keys { + digits:[64]u8 + _,ref_error:=store.ref_get(fresh,asset.REF_IMPORTS,store.hex(key,&digits)) + testing.expectf(t,ref_error==.None,"the build has the key it had where it was ingested") + } + after:=blobs_of(t,fresh,clone) + testing.expect_value(t,len(after),len(before)) + for path,hash in before {testing.expectf(t,after[path]==hash,"%s is the same blob",path)} + + // Without its sources nor a store that has them: listed, not built, said by package. + bare:=fmt.tprintf("%s/bare",top) + folder_copy(game,bare) + for file in ([]string{"box.gltf","sign.png","wood.png"}) {os.remove(fmt.tprintf("%s/assets/props/%s",bare,file))} + empty:=test_store(t,fmt.tprintf("%s/empty",top)) + if empty==nil {return} + alone:=options + alone.store,alone.root=empty,bare + lacking:=project_refresh(bare,alone,context.temp_allocator) + testing.expectf(t,len(lacking.missing)==2 && len(lacking.failures)==0 && lacking.ingested==0,"%v",lacking) + resolved,_:=asset.root_packages(bare,empty.root,context.temp_allocator) + testing.expectf(t,len(resolved.listed)==len(before) && len(resolved.problems)==2,"%v",resolved.problems) + for l in resolved.listed {testing.expectf(t,l.state==.Not_Built,"%s is not built",l.path)} + for entry in pinned.packages { + id:[base.ASSET_ID_TEXT]u8 + named:=false + for line in resolved.problems {named||=strings.contains(line,base.asset_id_format(entry.id,id[:]))} + testing.expectf(t,named,"%v is named: %v",entry.id,resolved.problems) + } + + testing.expect_value(t,project_unpin(bare,pinned.packages[0].id),asset.Error.None) + left,_,_:=asset.packages_load(bare,context.temp_allocator) + testing.expect(t,len(left.packages)==1 && left.packages[0].id==pinned.packages[1].id,"the other stays pinned") + testing.expect(t,!os.exists(asset.package_file(bare,pinned.packages[0].id)),"unpinning removes its manifest") + testing.expect(t,os.exists(asset.package_file(bare,pinned.packages[1].id))) +} + // What a project's packages.mirai or the store says that is not a record of // this version is reported, never a crash nor a default. @(test) @@ -286,6 +379,12 @@ hostile_packages_and_manifests_are_reported_without_a_crash :: proc(t:^testing.T testing.expect_value(t,asset.packages_save(game,&list),asset.Error.None) report:=project_refresh(game,options,context.temp_allocator) testing.expectf(t,report.packages==3 && len(report.failures)==2 && len(report.missing)==1 && strings.contains(report.missing[0],"package lost"),"%v",report) + // A file of the root that is not the manifest its pin names. + os.make_directory_all(fmt.tprintf("%s/%s",game,asset.PACKAGE_FILES)) + _=os.write_entire_file(asset.package_file(game,{1,1}),transmute([]u8)string("not the pinned manifest")) + mismatched:=project_refresh(game,options,context.temp_allocator) + testing.expectf(t,len(mismatched.failures)==2 && strings.contains(strings.join(mismatched.failures[:],"\n",context.temp_allocator),"is not the manifest"),"%v",mismatched.failures[:]) + os.remove(asset.package_file(game,{1,1})) text,_:=os.read_entire_file(fmt.tprintf("%s/%s",game,asset.PACKAGES_FILE),context.temp_allocator) work:=make([]u8,len(text),context.temp_allocator) diff --git a/cook/import/sources/sources.odin b/cook/import/sources/sources.odin index 04a703529..ce5ab234f 100644 --- a/cook/import/sources/sources.odin +++ b/cook/import/sources/sources.odin @@ -104,20 +104,6 @@ source_kind :: proc(path:string,formats:[]importer.Format=nil) -> Source_Kind { return .None } -// The project or the plugin whose assets folder has the file at path: the -// folder that holds the nearest assets folder above it. False for a file in -// no assets folder. -source_root :: proc(path:string,allocator:=context.allocator) -> (root:string,ok:bool) { - names:=project.FOLDER_NAMES - folder:=project.path_absolute(path,context.temp_allocator) - for { - parent:=filepath.dir(folder) - if parent==folder {return} - if filepath.base(parent)==names[.Assets] {return strings.clone(filepath.dir(parent),allocator),true} - folder=parent - } -} - // A source a plugin imports whole (editor.Source_Importer), prepared for the // assets of a root: claims says whether it is one, by its path and up to // importer.SNIFF_BYTES of its head, and run imports it as import_source does, @@ -168,7 +154,7 @@ whole_find :: proc(wholes:[]Whole_Source,path:string) -> (whole:Whole_Source,cla } // Imports a source file in the assets folder of a project or a plugin -// (options.root; "": source_root): one of wholes claims is imported whole by +// (options.root; "": importer.source_root): one of wholes claims is imported whole by // it, before the formats; a model writes its assets and, when it // has nodes, the prefab of its hierarchy, which the report lists last; an // image or a sound writes one asset. The sidecar of the source @@ -180,7 +166,7 @@ whole_find :: proc(wholes:[]Whole_Source,path:string) -> (whole:Whole_Source,cla import_source :: proc(path:string,options:importer.Import_Options={},formats:[]importer.Format=nil,wholes:[]Whole_Source=nil,allocator:=context.allocator) -> (report:importer.Import_Report,error:importer.Error) { formats:=formats if formats!=nil else FORMATS[:] options:=options - if options.root=="" {options.root,_=source_root(path,context.temp_allocator)} + if options.root=="" {options.root,_=importer.source_root(path,context.temp_allocator)} if whole,claimed:=whole_find(wholes,path);claimed { options.importer_code=whole.code scene:string diff --git a/cook/import/sources/support_test.odin b/cook/import/sources/support_test.odin index 2d2e27d16..ba860bfeb 100644 --- a/cook/import/sources/support_test.odin +++ b/cook/import/sources/support_test.odin @@ -109,7 +109,7 @@ copy_in :: proc(root,from:string,files:..string) -> string { test_options :: proc(t:^testing.T,path:string,options:importer.Import_Options={},loc:=#caller_location) -> importer.Import_Options { options:=options root:=options.root - if root=="" {root,_=source_root(path,context.temp_allocator)} + if root=="" {root,_=importer.source_root(path,context.temp_allocator)} if root=="" {root=filepath.dir(path)} os.make_directory_all(root) // A root that cannot hold a store leaves none: the import says so. @@ -146,7 +146,7 @@ registry_with :: proc(t:^testing.T,root:string,report:^importer.Import_Report,r: list:asset.Packages list.packages=make([dynamic]asset.Project_Package,context.temp_allocator) append(&list.packages,asset.Project_Package{id=report.package_id,manifest=report.manifest}) - resolved,error:=asset.packages_resolve(s,&list,context.temp_allocator) + resolved,error:=asset.packages_resolve(s,"",&list,context.temp_allocator) if !testing.expect_value(t,error,asset.Error.None,loc=loc) {return false} asset.registry_init(r,fmt.tprintf("%s/assets",root),"",context.allocator,resolved.listed[:]) return testing.expect_value(t,asset.registry_scan(r),asset.Error.None,loc=loc) diff --git a/demo/assets/characters/jelly/jelly.gltf.import b/demo/assets/characters/jelly/jelly.gltf.import deleted file mode 100644 index ec13af958..000000000 --- a/demo/assets/characters/jelly/jelly.gltf.import +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 3, - "settings": {}, - "folder": "jelly", - "inputs": ["jelly.bin"], - "outputs": [ - {"kind": "SCEN", "name": "jelly", "index": 0, "id": "107c9b8f4bf9dfe8936235eae8e042a6"}, - {"kind": "MAT", "name": "Candy", "index": 0, "id": "a18adcad4fedf21125038a819d27acd2"}, - {"kind": "SKEL", "name": "Jelly bones", "index": 0, "id": "dae452ce18d59e780ff745f7d1b43697"}, - {"kind": "MESH", "name": "Party Jelly", "index": 0, "id": "25a3e22ecb9c265a1614515f868efc52", "part": true}, - {"kind": "MESH", "name": "Party Jelly (1)", "index": 1, "id": "1c00bfb5bceb2619036b114104f42040", "part": true}, - {"kind": "ANIM", "name": "jelly", "index": 0, "id": "0d555b0da56949fb3cdbdf41410432f4"} - ] -} diff --git a/demo/assets/characters/jelly/jelly/Candy.mat b/demo/assets/characters/jelly/jelly/Candy.mat index 11d72c13927cc21dad3b4edbbc8474a781a23e57..23a6854f63681e3be5676997abed35ad57c9f376 100644 GIT binary patch delta 31 pcmV+)0O0?qBd8;gNC@bh;i=Tw($BF_&<~S15bTqX3>cFn5rhM(4-EhS delta 52 zcmZ3Xyh3?`2gh}hki~I@?LRgKT;P{TEG|whDb`QUNGwWBE=esa*3U}K$*I(ztRV1a IvWj2}06ka~`Tzg` diff --git a/demo/assets/characters/jelly/jelly/Jelly bones.skeleton b/demo/assets/characters/jelly/jelly/Jelly bones.skeleton index c1bfaf99f3124242edae27cc87273bfbdd11fae7..6e812050f9247e7b5cf18b7159fe0d38218b4151 100644 GIT binary patch delta 31 ncmew${6Tnv2Zz%BN~7LS^9(lzNN`MU;CM560<*;A9L^R1)zu8M delta 52 zcmew${6Tnv2ghq}*{XBD9yVI=EG|whDb`QUNGwWBE=esa*3U}K$*I(zoWb#C Iat>z;08z~qvj6}9 diff --git a/demo/assets/characters/jelly/jelly/jelly.anim b/demo/assets/characters/jelly/jelly/jelly.anim index 500be3dfd74489d73e6b27fbaddba2fc718c0e44..1986065d00ed91f8d442c4267f91c3261c2598b8 100644 GIT binary patch delta 31 ncmbOrHbHEH2Z!~X;!5r2{}LMmEI21G;CwUr0H?&{9wW9&8tSa3=t78j?M6zeBvBo-wmm!uXI>u06r}u}~;S5&QoAKlmyDLB4VYKolCEKD)DtM-L~H Sux{*=U_~I4UpPZ3cl$czST2!o`n^}~Y2_f~;bCM=MWPd#Q0z1d#1Xi2L5Aq!VY=O3c)stRV1a IvWj2}05IAVj{pDw diff --git a/demo/assets/characters/mirai/Mirai holograma violeta.mat b/demo/assets/characters/mirai/Mirai holograma violeta.mat index 89cfed6ab06910e75b4ad3e1273f5c6db3347dbc..6629aaf656e6063dbd5850d22cc35087904e206b 100644 GIT binary patch delta 31 ncmZ3Xyh3?`2Zz(!b)8{J!K@ntF7Qvb5O_0r0*}OG6~Pt&%S#L= delta 52 zcmZ3Xyh3?`2S;uk&m%dJfBG8(F7Qhv78j?M6zeBvBo-wmm!uXI>*r<`C1&bRRuFhI ISw*k~01&(r)Bpeg diff --git a/demo/assets/characters/mirai/mirai.anim b/demo/assets/characters/mirai/mirai.anim index 398a8e5500a4f0cafead7dd439e4424b8cf4c4ec..e990ebe6582a7e29fb4cfb6a987fcba35ea37b5e 100644 GIT binary patch delta 39 xcmV+?0NDS)^9R852arezLfh<1!nhkYfl#$j0qFOaQ1}7tmvHz27?(Wx0fddR6JG!T delta 59 zcmV-B0L1^m^9R852arezL&sk-YG;v8fl#$j0qFM_VRLh3baO9bXkl_;V{~P5b1!Xa Ra$#vNmpJ$V?3XfXV%LHwHvFDJB*drcFn5rpAD4QBuV delta 56 zcmZ3Xyh3?`2S9V3f2Pvk~y#yY$GFdQY)X&<~S15bTqX3>cFn5rp$?4jupi delta 56 zcmZ3Xyh3?`2Zz$qtty$8_oFukT;P{TEG|whDb`QUNGwWBE=esa)-TFW$}iEMtRV0P Lh*>9V3f2Pvb7>V6 diff --git a/demo/assets/characters/robot/glb/robot joints.mat b/demo/assets/characters/robot/glb/robot joints.mat index 1f89a2346d71050403407d93707dc8476ebf06a2..9248021051142a089a8c14b6edc11258b89a89dc 100644 GIT binary patch delta 31 pcmV+)0O0?qBd8;gNC@WDPU9z|l)bS~&<~S15bTqX3>cFn5rpZM4Y&XR delta 56 zcmZ3Xyh3?`2ZyEeChe(x{Vz8LT;P{TEG|whDb`QUNGwWBE=esa)-TFW$}iEMtRV0P Lh*>9V3f2PvhFKMi diff --git a/demo/assets/characters/robot/glb/robot visor.mat b/demo/assets/characters/robot/glb/robot visor.mat index b4fd85a4278e326e9ee7dbee6649c739e10d3d65..6ae9d52f15e2967c48848335084bca1f9eab765d 100644 GIT binary patch delta 31 ncmZ3Xyh3?`2gjS@AM@YmdIf9@xWGTzLg3Bh2|N;$RRmiA>5UDY delta 56 zcmZ3Xyh3?`2ggRcciW2Id|0wE-~zuyVsUY5NwI!%Mq*K7a!G1Yv3^m0Qhtg4WCejY LK+HN>Q?MQY&si58 diff --git a/demo/assets/characters/robot/gltf/13 joint robot rig.skeleton b/demo/assets/characters/robot/gltf/13 joint robot rig.skeleton index 74d5396d2faf4afc451b5640ee7d98df3ab48d90..668a3a091cfff3564dfa5ddadf6c6057aa677fe5 100644 GIT binary patch delta 33 rcmV++0N($whyk#O0gy-tET-2_x21-bu~1k-lfXjklQ0DsldMC8`wkAp delta 54 zcmdnc!MLG=ae@bj)rogCncFn5rpV-4c7nw delta 56 zcmZ3Xyh3?`2glUp>kb#$7UymZxWF%wSX`W1QmmhxkywG%pw3v*&#cFn5rpRn4Y&XR delta 56 zcmZ3Xyh3?`2gjR8)!3GXCr%p!F7Qhv78j?M6zeBvBo-wmm!uXI>lfuG<(KGBRuFgt K#H^Dw1?vHcQ5A^* diff --git a/demo/assets/characters/robot/gltf/robot joints.mat b/demo/assets/characters/robot/gltf/robot joints.mat index 12b761c94c75bc1f1965da1ec4771a5625210cd6..09d8bd977e653eefe19c5dc305958c88234c441c 100644 GIT binary patch delta 31 pcmV+)0O0?qBd8;gNC>sy?9bd@V)U_4&<~S15bTqX3>cFn5rq2_4v_!= delta 56 zcmZ3Xyh3?`2glm7N4F2mNt(1V-~zuyVsUY5NwI!%Mq*K7a!G1Yv3^m0Qhtg4WCejY LK+HN>Q?MQY!mk#{ diff --git a/demo/assets/characters/robot/gltf/robot visor.mat b/demo/assets/characters/robot/gltf/robot visor.mat index 77d7ba70cbc45c5f45a6691abeba4e1eba64d2de..cb6a53ca207f18b1bf6e64ddbd235243673ff7cf 100644 GIT binary patch delta 31 ncmZ3Xyh3?`2S?7@IY+g%h1YKkxWGTzLg3Bh2|N;$RRmiA%rp!` delta 56 zcmZ3Xyh3?`2glswjsY1=Y=<@mT;P{TEG|whDb`QUNGwWBE=esa)-TFW$}iEMtRV0P Lh*>9V3f2PvX2KM| diff --git a/demo/assets/characters/robot/robot.glb.import b/demo/assets/characters/robot/robot.glb.import deleted file mode 100644 index bb0c7e42c..000000000 --- a/demo/assets/characters/robot/robot.glb.import +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 3, - "settings": {}, - "folder": "glb", - "outputs": [ - {"kind": "SCEN", "name": "robot", "index": 0, "id": "e431738105898dd368735a6ebb1d3c19"}, - {"kind": "MAT", "name": "robot blue", "index": 0, "id": "1c5c7311b02f9c27a26158a5dfbbfe47"}, - {"kind": "MAT", "name": "robot gold", "index": 1, "id": "985941907b0208e6eaef9ca10b6db8ba"}, - {"kind": "MAT", "name": "robot joints", "index": 2, "id": "d9b762e720d363036a57efa7b616eaef"}, - {"kind": "MAT", "name": "robot visor", "index": 3, "id": "ae20beadede617fdd239a5a6ad23eb83"}, - {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "83d297850b4eccf18f4225970eff51fb"}, - {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "941beb67913839d9f4d4e05528cb0c80", "part": true} - ] -} diff --git a/demo/assets/characters/robot/robot.gltf.import b/demo/assets/characters/robot/robot.gltf.import deleted file mode 100644 index 6988a96b2..000000000 --- a/demo/assets/characters/robot/robot.gltf.import +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 3, - "settings": {}, - "folder": "gltf", - "inputs": ["robot.bin"], - "outputs": [ - {"kind": "SCEN", "name": "robot", "index": 0, "id": "b0f0241df6e014392af304b56cf59eab"}, - {"kind": "MAT", "name": "robot blue", "index": 0, "id": "c67757fb8b041b0d6b6178fe958bd91e"}, - {"kind": "MAT", "name": "robot gold", "index": 1, "id": "feebc67eaeb9081ec3d1f1edac208b2a"}, - {"kind": "MAT", "name": "robot joints", "index": 2, "id": "a453277713f405f86fd3dc27c2506977"}, - {"kind": "MAT", "name": "robot visor", "index": 3, "id": "0bf1bd3d54bcb875b34957598640c69d"}, - {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "cf5b1fe881d020461927fe9b85c42136"}, - {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "3a94f8ad990aede6566ad2af8010bb23", "part": true} - ] -} diff --git a/demo/assets/church/church_glass_color.png.import b/demo/assets/church/church_glass_color.png.import deleted file mode 100644 index c5c3d9275..000000000 --- a/demo/assets/church/church_glass_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "church_glass_color", "index": 0, "id": "2767edae7841cea315b1e40d922c9950"} - ] -} diff --git a/demo/assets/gallery/leaves_color.png.import b/demo/assets/gallery/leaves_color.png.import deleted file mode 100644 index 38ee12635..000000000 --- a/demo/assets/gallery/leaves_color.png.import +++ /dev/null @@ -1,10 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color", - "alpha_coverage": 0.5 - }, - "outputs": [ - {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "df49540f1f9dbcf6fe2e612ea427757e"} - ] -} diff --git a/demo/assets/gallery/plant.gltf.import b/demo/assets/gallery/plant.gltf.import deleted file mode 100644 index fbdd23f63..000000000 --- a/demo/assets/gallery/plant.gltf.import +++ /dev/null @@ -1,14 +0,0 @@ -{ - "version": 3, - "settings": {}, - "folder": "plant", - "inputs": ["leaves_color.png"], - "outputs": [ - {"kind": "SCEN", "name": "plant", "index": 0, "id": "fc120101f1ddb302e135784184f05931"}, - {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "dba1b6bd3e8cde629938d41dc8934d02"}, - {"kind": "MAT", "name": "Terracotta", "index": 0, "id": "e2a8b0fcb9401808520b370016df35c9"}, - {"kind": "MAT", "name": "Soil", "index": 1, "id": "9337601926ea70572a4442c984759e15"}, - {"kind": "MAT", "name": "Leaves", "index": 2, "id": "7e248f2df1c8c849ff972c9a1ce95fb5"}, - {"kind": "MESH", "name": "Plant", "index": 0, "id": "72d9220e763e42b1da3107e0e5256b90", "part": true} - ] -} diff --git a/demo/assets/gallery/plant/Leaves.mat b/demo/assets/gallery/plant/Leaves.mat index 4e33fb050b166f371370365d785ed9ac3d5e47d0..dd202f1f3fd80c70acfcb137ebd86ff9c86ac427 100644 GIT binary patch delta 31 pcmV+)0O0?)Be)}wNC@02Kq%X#`qr^f;182O5cQLg3J;Ss5rpLa4YU9N delta 43 zcmdm?yhC|{2M4>&a+byG_2V}NJmBX~EG|whDb`O<%*jbDs??usAnn~l delta 43 zcmZ3Xyh3?`2S>&o!-ogXE0}ByxWLb!SX`W1Qmmhzn3I!QRH;8%LE!6T6~Pt&kWmqY diff --git a/demo/assets/gallery/plant/Terracotta.mat b/demo/assets/gallery/plant/Terracotta.mat index 8e9385c3663dc6173968da9662944ccc5baec090..9ddc7c176a7c7fcccf52d86101f9138d29c334c4 100644 GIT binary patch delta 31 ncmZ3Xyh3?`2ZvG1!Zk6w>+&`RT;QLqFYtA80~h~f6~Pt&(sT@* delta 43 zcmZ3Xyh3?`2gmgjPtso+-I};D-~vB?VsUY5NwI!e6Tub$&w~tx delta 43 zcmdm?yhC|{2gmXBSF7$Ftcu(OV3L delta 43 zcmZ3Xyh3?`2glP6JHFaUw#FL+F7Wdw78j?M6ziua=H#RnRq9Vx5coP-MX&_`aCQ)b diff --git a/demo/assets/gallery/vase_color.png.import b/demo/assets/gallery/vase_color.png.import deleted file mode 100644 index 2089b1916..000000000 --- a/demo/assets/gallery/vase_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "vase_color", "index": 0, "id": "c94299d672a2e4371f7a5374961b418c"} - ] -} diff --git a/demo/assets/showcase/hum.wav.import b/demo/assets/showcase/hum.wav.import deleted file mode 100644 index 8bbc7bae5..000000000 --- a/demo/assets/showcase/hum.wav.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "SND", "name": "hum", "index": 0, "id": "7f27bbcfa7a4b5fac01753762f745b26"} - ] -} diff --git a/demo/assets/sky/sky_day.hdr.import b/demo/assets/sky/sky_day.hdr.import deleted file mode 100644 index 1b8d7bdf2..000000000 --- a/demo/assets/sky/sky_day.hdr.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "sky_day", "index": 0, "id": "39808e0def899c47a24a89a3fab89e64"} - ] -} diff --git a/demo/assets/sky/sky_night.jpg.import b/demo/assets/sky/sky_night.jpg.import deleted file mode 100644 index 650e7396b..000000000 --- a/demo/assets/sky/sky_night.jpg.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "sky_night", "index": 0, "id": "8a6329018eb7b53a0215bd4bdbf5a2c9"} - ] -} diff --git a/demo/assets/sky/sky_overcast.jpg.import b/demo/assets/sky/sky_overcast.jpg.import deleted file mode 100644 index 8201f0c2c..000000000 --- a/demo/assets/sky/sky_overcast.jpg.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "sky_overcast", "index": 0, "id": "88a3a1a16a759ed20966d1c3ff54ada1"} - ] -} diff --git a/demo/assets/sky/sky_sunset.jpg.import b/demo/assets/sky/sky_sunset.jpg.import deleted file mode 100644 index 294102d0a..000000000 --- a/demo/assets/sky/sky_sunset.jpg.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "sky_sunset", "index": 0, "id": "7b819d061e0487f04a17587b8ff0bce1"} - ] -} diff --git a/demo/assets/surfaces/bricks/bricks_color.png.import b/demo/assets/surfaces/bricks/bricks_color.png.import deleted file mode 100644 index 1ce539650..000000000 --- a/demo/assets/surfaces/bricks/bricks_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "bricks_color", "index": 0, "id": "4c24cf1fa5b4eeca8ac84ea3d8813cfc"} - ] -} diff --git a/demo/assets/surfaces/bricks/bricks_normal.png.import b/demo/assets/surfaces/bricks/bricks_normal.png.import deleted file mode 100644 index 221bc41d1..000000000 --- a/demo/assets/surfaces/bricks/bricks_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "bricks_normal", "index": 0, "id": "0873ef8dc2c7deca1ed9eb84f5a8d6b9"} - ] -} diff --git a/demo/assets/surfaces/bricks/bricks_orm.png.import b/demo/assets/surfaces/bricks/bricks_orm.png.import deleted file mode 100644 index 2e9261796..000000000 --- a/demo/assets/surfaces/bricks/bricks_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "bricks_orm", "index": 0, "id": "d526fc286cf3d0bb3c956200037149d5"} - ] -} diff --git a/demo/assets/surfaces/cobblestone/cobblestone_color.png.import b/demo/assets/surfaces/cobblestone/cobblestone_color.png.import deleted file mode 100644 index 07b165efe..000000000 --- a/demo/assets/surfaces/cobblestone/cobblestone_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "cobblestone_color", "index": 0, "id": "9a69320baec0a94ab9eb8d2010a8a8a4"} - ] -} diff --git a/demo/assets/surfaces/cobblestone/cobblestone_normal.png.import b/demo/assets/surfaces/cobblestone/cobblestone_normal.png.import deleted file mode 100644 index b08c832e2..000000000 --- a/demo/assets/surfaces/cobblestone/cobblestone_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "cobblestone_normal", "index": 0, "id": "1a08080be2a08b5fe300ec425923c195"} - ] -} diff --git a/demo/assets/surfaces/cobblestone/cobblestone_orm.png.import b/demo/assets/surfaces/cobblestone/cobblestone_orm.png.import deleted file mode 100644 index cebbb2ddf..000000000 --- a/demo/assets/surfaces/cobblestone/cobblestone_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "cobblestone_orm", "index": 0, "id": "94cef7c053d5afed918f83a3d6242853"} - ] -} diff --git a/demo/assets/surfaces/glass/stained_glass_color.png.import b/demo/assets/surfaces/glass/stained_glass_color.png.import deleted file mode 100644 index 3a9537dd6..000000000 --- a/demo/assets/surfaces/glass/stained_glass_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "stained_glass_color", "index": 0, "id": "601bad68a6b7e349bc2655ac13c9196f"} - ] -} diff --git a/demo/assets/surfaces/marble/marble_color.png.import b/demo/assets/surfaces/marble/marble_color.png.import deleted file mode 100644 index 5ed1dcec8..000000000 --- a/demo/assets/surfaces/marble/marble_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "marble_color", "index": 0, "id": "f3ace87b27d210d20ba711efdf73ad3a"} - ] -} diff --git a/demo/assets/surfaces/marble/marble_normal.png.import b/demo/assets/surfaces/marble/marble_normal.png.import deleted file mode 100644 index fdb8483a4..000000000 --- a/demo/assets/surfaces/marble/marble_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "marble_normal", "index": 0, "id": "6efbfe1939e3a2a35c0ccc8621cc6abb"} - ] -} diff --git a/demo/assets/surfaces/marble/marble_orm.png.import b/demo/assets/surfaces/marble/marble_orm.png.import deleted file mode 100644 index 0495873a2..000000000 --- a/demo/assets/surfaces/marble/marble_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "marble_orm", "index": 0, "id": "972fc9634ba6242d0b5f33e03e3a0741"} - ] -} diff --git a/demo/assets/surfaces/metal/brushed_normal.png.import b/demo/assets/surfaces/metal/brushed_normal.png.import deleted file mode 100644 index d263fdc4a..000000000 --- a/demo/assets/surfaces/metal/brushed_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "brushed_normal", "index": 0, "id": "b9466f1cd86f5f900d74e12980fdc702"} - ] -} diff --git a/demo/assets/surfaces/metal/brushed_orm.png.import b/demo/assets/surfaces/metal/brushed_orm.png.import deleted file mode 100644 index a183e07e5..000000000 --- a/demo/assets/surfaces/metal/brushed_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "brushed_orm", "index": 0, "id": "ec4b617cf48145349c2c76f465e19ae6"} - ] -} diff --git a/demo/assets/surfaces/metal/grate_color.png.import b/demo/assets/surfaces/metal/grate_color.png.import deleted file mode 100644 index 2400e6154..000000000 --- a/demo/assets/surfaces/metal/grate_color.png.import +++ /dev/null @@ -1,10 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color", - "alpha_coverage": 0.5 - }, - "outputs": [ - {"kind": "TEX", "name": "grate_color", "index": 0, "id": "90b31208c9dfc6a6d00ab5517f084571"} - ] -} diff --git a/demo/assets/surfaces/metal/grate_normal.png.import b/demo/assets/surfaces/metal/grate_normal.png.import deleted file mode 100644 index f556b233e..000000000 --- a/demo/assets/surfaces/metal/grate_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "grate_normal", "index": 0, "id": "c9d9c9f0f1dea3ec0ceec0f9671187eb"} - ] -} diff --git a/demo/assets/surfaces/metal/grate_orm.png.import b/demo/assets/surfaces/metal/grate_orm.png.import deleted file mode 100644 index 681d8e7ed..000000000 --- a/demo/assets/surfaces/metal/grate_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "grate_orm", "index": 0, "id": "71e4b6effc8f9e4ac6c7f1dc0f02612e"} - ] -} diff --git a/demo/assets/surfaces/neon/neon_emission.png.import b/demo/assets/surfaces/neon/neon_emission.png.import deleted file mode 100644 index 30b5d43f3..000000000 --- a/demo/assets/surfaces/neon/neon_emission.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "neon_emission", "index": 0, "id": "6d9362793631542f0b778005f9fd8dd0"} - ] -} diff --git a/demo/assets/surfaces/tiles/tiles_color.png.import b/demo/assets/surfaces/tiles/tiles_color.png.import deleted file mode 100644 index fd3feba84..000000000 --- a/demo/assets/surfaces/tiles/tiles_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "tiles_color", "index": 0, "id": "cd3d66765e834e0c508906257f888eba"} - ] -} diff --git a/demo/assets/surfaces/tiles/tiles_normal.png.import b/demo/assets/surfaces/tiles/tiles_normal.png.import deleted file mode 100644 index 9ec43a2ed..000000000 --- a/demo/assets/surfaces/tiles/tiles_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "tiles_normal", "index": 0, "id": "bd7cde48d01cd173f904fe6b00fb72dc"} - ] -} diff --git a/demo/assets/surfaces/tiles/tiles_orm.png.import b/demo/assets/surfaces/tiles/tiles_orm.png.import deleted file mode 100644 index 7478ed1fc..000000000 --- a/demo/assets/surfaces/tiles/tiles_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "tiles_orm", "index": 0, "id": "84dbfd2533a249a9716c78d548c6c654"} - ] -} diff --git a/demo/assets/surfaces/wood/planks_color.png.import b/demo/assets/surfaces/wood/planks_color.png.import deleted file mode 100644 index d71778853..000000000 --- a/demo/assets/surfaces/wood/planks_color.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Color" - }, - "outputs": [ - {"kind": "TEX", "name": "planks_color", "index": 0, "id": "79a53ef6d585bc9fd1a912a9e592f913"} - ] -} diff --git a/demo/assets/surfaces/wood/planks_normal.png.import b/demo/assets/surfaces/wood/planks_normal.png.import deleted file mode 100644 index 8d7ca3873..000000000 --- a/demo/assets/surfaces/wood/planks_normal.png.import +++ /dev/null @@ -1,9 +0,0 @@ -{ - "version": 3, - "settings": { - "image": "Normal" - }, - "outputs": [ - {"kind": "TEX", "name": "planks_normal", "index": 0, "id": "0558bb6aa0ec012cac33f59d343e99ed"} - ] -} diff --git a/demo/assets/surfaces/wood/planks_orm.png.import b/demo/assets/surfaces/wood/planks_orm.png.import deleted file mode 100644 index 6df353b68..000000000 --- a/demo/assets/surfaces/wood/planks_orm.png.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "TEX", "name": "planks_orm", "index": 0, "id": "6646d15415504809a3da8752ce1d6ca6"} - ] -} diff --git a/demo/packages.mirai b/demo/packages.mirai new file mode 100644 index 000000000..32741fe2b --- /dev/null +++ b/demo/packages.mirai @@ -0,0 +1,41 @@ +{ + "version": 1, + "packages": [ + {"id": "780bee6597fbb860e803d513bda2dee7", "manifest": "4a963eaca3eae9448050350443c6f8fd2bd347823bd95bcfbb5dcd1c5ee5370f", "folder": "characters/jelly/jelly", "assets": []}, + {"id": "0512ac6bb76639efa1aa35746972fedb", "manifest": "3917d5ddca017afdb78f7390198c987789bbf86852b07a846a96503fd3280e90", "folder": "characters/mirai", "assets": []}, + {"id": "2d26a9a0d52c3edd23a71d2cc8d6e463", "manifest": "60cb4e9c196cdad90fbf5defbdc67a6fa641338b80f3e1166064c8573acf1a00", "folder": "characters/robot/glb", "assets": []}, + {"id": "84060b2db086d874fb6a99a638a79a10", "manifest": "1c848c98e0f08a7645bc8e598dbbe72007ddbe945eaa665083f76780e08b6137", "folder": "characters/robot/gltf", "assets": []}, + {"id": "44342052da337fa065c60cf296a4466a", "manifest": "8537ae1e9dabe37f7b501e251fb39fa83d136ab825f85fd6faeeb7068f34a49a", "folder": "church", "assets": []}, + {"id": "caae575f45863b717aafbe7fe73862c0", "manifest": "7616716f5e8ef9957e7e8439c567854e015ac970d32ad5c6d61d5de288e21530", "folder": "gallery", "assets": []}, + {"id": "6498e1a94e6c4621f925d22f8384243a", "manifest": "f943fbda4e5f3e2d21db427c8edb288b619695e94bcb7044cbca54f9083bc1a1", "folder": "gallery/plant", "assets": []}, + {"id": "a3799fb21f329ae819440d1726f2aa20", "manifest": "3e577e2889305d7efd56fb9e47d11e7420b76654c6096d27663d865b36b84d1e", "folder": "gallery/vase", "assets": []}, + {"id": "ca27d206e666815335b68ee9c8900ab7", "manifest": "b3d7a22ae8f1d6a895f5398c22f91d521aeac6e34fc583f30d07f506623b7ef4", "folder": "gallery", "assets": []}, + {"id": "7572bd9ffa0dd2354005eff39b25f1bb", "manifest": "d9067a4a5673870ded5622963e9937580372fd9c4be7c60cbf1e1d7884afa80c", "folder": "showcase", "assets": []}, + {"id": "181b8168d78256877581cb3f1fda937e", "manifest": "1a337d3e6ab531f0559a93db34b3b9563d4be6ff1354a59789875c6bdd5af261", "folder": "sky", "assets": []}, + {"id": "c1a66f64c05efe53cca206765ab98117", "manifest": "a39653aff76a573ab91ac1a7c4717be9770a053b86d153a1ca083208221712d0", "folder": "sky", "assets": []}, + {"id": "4840a3f5a151cbe8c1b403b30fe8240d", "manifest": "623313b02fd3e64fec926caffcb1470c3fae02f48a29a94176b3aa42307766fd", "folder": "sky", "assets": []}, + {"id": "e022a803a1c6fb9fe4b86ab65a48bc24", "manifest": "f83c31a2d6c7929d4d5fb5b921c0c3e52c7ee06807c0967d52f150a3720623bb", "folder": "sky", "assets": []}, + {"id": "0d0874d674d00d43ab66a390641369df", "manifest": "8f31ccfb8cf203764bfa66d37fe5f6e3dbb36aa62806ca8099cf570b7902bcaf", "folder": "surfaces/bricks", "assets": []}, + {"id": "0ab7eefb80a0bb80c8cd0065d5c53f11", "manifest": "6eccf80dcb05311a5d6951e666ac5085b87a6188fb1970872abd9059030018c4", "folder": "surfaces/bricks", "assets": []}, + {"id": "26356114f7faffcfafff109308dd9a32", "manifest": "a167b042e04fa82c3cc0dc5a812b27ca4096f72ca6d54773f9dec952aa33ad9d", "folder": "surfaces/bricks", "assets": []}, + {"id": "c0d7666d947e65fc7dd68a31013c0939", "manifest": "3f538b9e6f732075b4c363c29794378657126105df9e3ea9731e73dd4735d867", "folder": "surfaces/cobblestone", "assets": []}, + {"id": "a278db7076e649a48933fc75b0dc1a4c", "manifest": "b666d4035290effaf589397c6f1db1280c115131aa95e18746990cc95c960835", "folder": "surfaces/cobblestone", "assets": []}, + {"id": "ae40765be8d60ddad02e47bb21203f8f", "manifest": "bb75ff24c6019485ca30a18ca4ceec0a85ce35e72c76365bc5b231c01b6e9974", "folder": "surfaces/cobblestone", "assets": []}, + {"id": "6cc37d40c23ded0fb9cc963a4dec2fbb", "manifest": "411e17f0d14cb6f570db4de51337b016ff7a8061180e0b648aaeced9cd6223a2", "folder": "surfaces/glass", "assets": []}, + {"id": "09648c700552c36b5b4062fd1822de01", "manifest": "3e1aada9237d0076bc8d5c548c3ee8b782a6a2de721a6d0afd492934f6f4a5fd", "folder": "surfaces/marble", "assets": []}, + {"id": "2d6b0454a36971219d185d940f836ac7", "manifest": "2b2027b23a16bbea30c4cb3ecbc2cdc8b2b16074700cd89a4780390bacaec932", "folder": "surfaces/marble", "assets": []}, + {"id": "a68982d58a94bfa0fcb588bf93dfafec", "manifest": "7b1ae691cd8969afe5a313747f792e6d3c018b6cb60a94a21e997f75bc7ac79f", "folder": "surfaces/marble", "assets": []}, + {"id": "641ffc8940c220c3b2460f7a9eaa7862", "manifest": "730eaab4050470f4db747e8eddd2a4879d8bc7e8a2a6a99b256b9df4136d28fa", "folder": "surfaces/metal", "assets": []}, + {"id": "a03a7d06fb4c433e89ba1ecd314d91ed", "manifest": "4153cc5e753ba39602ffded1fb1b353c4ae7e07387ee9d5c1a800e58f9d0299a", "folder": "surfaces/metal", "assets": []}, + {"id": "e0495081646e8954b731fc082bcd8884", "manifest": "81200390a7453dd2fbd4ee589dbd9fbc2a86da349299aa6083b7664947214e35", "folder": "surfaces/metal", "assets": []}, + {"id": "65805c17b2454730221e3d63066ce7f5", "manifest": "1f304dc9e03252dda912d5d5381f6ffd02869a7c974f0328e0bb7303499be5bd", "folder": "surfaces/metal", "assets": []}, + {"id": "81d18317bfbbf0851f30ebcb2de41b67", "manifest": "6fef9b0ccfddcf796583a20fa2837bf3a09235b4c262f28778e01d3370dc8ffd", "folder": "surfaces/metal", "assets": []}, + {"id": "856074124b2e8924607a1e79488b6abd", "manifest": "1b3c97c929e2eeca39cbe75ca5ae2a8c04e3ad5abdc22b1d90f6b8731d61a464", "folder": "surfaces/neon", "assets": []}, + {"id": "a270e82878cb164498ba98398060ccb8", "manifest": "990aa1e56c54010dcdbfb641b030bfe274a72281753e8f0a567bcadb53ec4028", "folder": "surfaces/tiles", "assets": []}, + {"id": "0b5202b0a19887317347ddcbfdcad5f6", "manifest": "1d9f92898ed6e4775e013790a7ccb27755b33b881047c8eed6be2a41b6ec63c0", "folder": "surfaces/tiles", "assets": []}, + {"id": "de7548cbbad3494b21280744cd16ed68", "manifest": "e6a7c04950a4f843283e188eeca9538d0f98035ff07ddf8e2216edc44f76ab90", "folder": "surfaces/tiles", "assets": []}, + {"id": "e96915c067e6fe50d517b59bf3b17d61", "manifest": "79982691c9fd719b5f3e34331a1cbeb88676be5f5fd8c630453b5860d0fd59f9", "folder": "surfaces/wood", "assets": []}, + {"id": "f4ccf188ab1bf19a62c53faaef829fba", "manifest": "a129e3f38b1008609560167f132b43076b087fce75805e78f41f0d2cd09623c2", "folder": "surfaces/wood", "assets": []}, + {"id": "0964d5141215cea8462119f07a904e49", "manifest": "c5c4fb4a315b60117f7f6564c7f39b7e4efa7d2db6caa5afcf0cb48f30715cc9", "folder": "surfaces/wood", "assets": []} + ] +} diff --git a/demo/packages/0512ac6bb76639efa1aa35746972fedb.mirai b/demo/packages/0512ac6bb76639efa1aa35746972fedb.mirai new file mode 100644 index 000000000..3f6ca2aa4 --- /dev/null +++ b/demo/packages/0512ac6bb76639efa1aa35746972fedb.mirai @@ -0,0 +1,21 @@ +{ + "version": 1, + "id": "0512ac6bb76639efa1aa35746972fedb", + "name": "mirai", + "origin": {"path": "assets/characters/mirai/mirai.glb", "folder": false, "importer": "glTF"}, + "identity": "339c12d886baefcd14f96e1c9cb6501e6b85183d0b5edbba6b290d3853c2937c", + "inputs": [], + "assets": [ + {"kind": "SCEN", "name": "mirai", "index": 0, "id": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "TEX", "name": "texture_0", "index": 0, "id": "a3d324f2a5454495463d13c1ebad2c84", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "TEX", "name": "texture_0_metallic_roughness", "index": 1, "id": "08abe809e118473bb530ac037696309e", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "MAT", "name": "BakedMaterial", "index": 0, "id": "f1d4dfad902a16a7afc56bb3202b1d94", "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["08abe809e118473bb530ac037696309e", "a3d324f2a5454495463d13c1ebad2c84"]}, + {"kind": "MAT", "name": "Mirai holograma violeta", "index": 1, "id": "dbb521d6ca206ef049b42c4fad14b7d5"}, + {"kind": "MAT", "name": "Mirai borde luminoso", "index": 2, "id": "bbfe2f2406e2d1a5aaf297d5d2c8a990"}, + {"kind": "SKEL", "name": "Armature", "index": 0, "id": "0208471f893327bdcf8bbaa108a26121", "owner": "94f640750cf5246f1c0728cc4cd2a07c"}, + {"kind": "MESH", "name": "output_unwrapped", "index": 0, "id": "f0c947702bb8141dc1d27444a341d52c", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["0208471f893327bdcf8bbaa108a26121", "f1d4dfad902a16a7afc56bb3202b1d94"]}, + {"kind": "MESH", "name": "Cinta orbital Left", "index": 1, "id": "8d6c8aa43c816332bca9f38c1dd0d17f", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["0208471f893327bdcf8bbaa108a26121", "bbfe2f2406e2d1a5aaf297d5d2c8a990", "dbb521d6ca206ef049b42c4fad14b7d5"]}, + {"kind": "MESH", "name": "Cinta orbital Right", "index": 2, "id": "78744c9faee4ebd82e0f9ca443b72656", "part": true, "owner": "94f640750cf5246f1c0728cc4cd2a07c", "dependencies": ["0208471f893327bdcf8bbaa108a26121", "bbfe2f2406e2d1a5aaf297d5d2c8a990", "dbb521d6ca206ef049b42c4fad14b7d5"]}, + {"kind": "ANIM", "name": "mirai", "index": 0, "id": "f3016e0fca41fca429b049307bde6f9f", "dependencies": ["0208471f893327bdcf8bbaa108a26121"]} + ] +} diff --git a/demo/packages/09648c700552c36b5b4062fd1822de01.mirai b/demo/packages/09648c700552c36b5b4062fd1822de01.mirai new file mode 100644 index 000000000..908889266 --- /dev/null +++ b/demo/packages/09648c700552c36b5b4062fd1822de01.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "09648c700552c36b5b4062fd1822de01", + "name": "marble_color", + "origin": {"path": "assets/surfaces/marble/marble_color.png", "folder": false, "importer": "image"}, + "identity": "2bbad925aaaa5a3c690d3e9061e0cee2f6d89ed230c0a78735a74b97bd707b66", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "marble_color", "index": 0, "id": "f3ace87b27d210d20ba711efdf73ad3a"} + ] +} diff --git a/demo/packages/0964d5141215cea8462119f07a904e49.mirai b/demo/packages/0964d5141215cea8462119f07a904e49.mirai new file mode 100644 index 000000000..467ab477d --- /dev/null +++ b/demo/packages/0964d5141215cea8462119f07a904e49.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "0964d5141215cea8462119f07a904e49", + "name": "planks_orm", + "origin": {"path": "assets/surfaces/wood/planks_orm.png", "folder": false, "importer": "image"}, + "identity": "a2dcb1c391aa037e1f7886bec3d3e17360d588365e162544f1aac4c6cb4532c1", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "planks_orm", "index": 0, "id": "6646d15415504809a3da8752ce1d6ca6"} + ] +} diff --git a/demo/packages/0ab7eefb80a0bb80c8cd0065d5c53f11.mirai b/demo/packages/0ab7eefb80a0bb80c8cd0065d5c53f11.mirai new file mode 100644 index 000000000..d47109d81 --- /dev/null +++ b/demo/packages/0ab7eefb80a0bb80c8cd0065d5c53f11.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "0ab7eefb80a0bb80c8cd0065d5c53f11", + "name": "bricks_normal", + "origin": {"path": "assets/surfaces/bricks/bricks_normal.png", "folder": false, "importer": "image"}, + "identity": "368a6ec83712603a49d698bec592fc764a4c289b9ffacc76273700cb4eaca77b", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "bricks_normal", "index": 0, "id": "0873ef8dc2c7deca1ed9eb84f5a8d6b9"} + ] +} diff --git a/demo/packages/0b5202b0a19887317347ddcbfdcad5f6.mirai b/demo/packages/0b5202b0a19887317347ddcbfdcad5f6.mirai new file mode 100644 index 000000000..1bda6238d --- /dev/null +++ b/demo/packages/0b5202b0a19887317347ddcbfdcad5f6.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "0b5202b0a19887317347ddcbfdcad5f6", + "name": "tiles_normal", + "origin": {"path": "assets/surfaces/tiles/tiles_normal.png", "folder": false, "importer": "image"}, + "identity": "fab044b54421b971441925ac3cd28572d405557ff481d49e448e1604c2842e2c", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "tiles_normal", "index": 0, "id": "bd7cde48d01cd173f904fe6b00fb72dc"} + ] +} diff --git a/demo/packages/0d0874d674d00d43ab66a390641369df.mirai b/demo/packages/0d0874d674d00d43ab66a390641369df.mirai new file mode 100644 index 000000000..83d3a5406 --- /dev/null +++ b/demo/packages/0d0874d674d00d43ab66a390641369df.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "0d0874d674d00d43ab66a390641369df", + "name": "bricks_color", + "origin": {"path": "assets/surfaces/bricks/bricks_color.png", "folder": false, "importer": "image"}, + "identity": "5161142e9629e908d0b4834890974b3c71806bdc8c2ef3f4dac651326c87609b", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "bricks_color", "index": 0, "id": "4c24cf1fa5b4eeca8ac84ea3d8813cfc"} + ] +} diff --git a/demo/packages/181b8168d78256877581cb3f1fda937e.mirai b/demo/packages/181b8168d78256877581cb3f1fda937e.mirai new file mode 100644 index 000000000..d38b1f80a --- /dev/null +++ b/demo/packages/181b8168d78256877581cb3f1fda937e.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "181b8168d78256877581cb3f1fda937e", + "name": "sky_day", + "origin": {"path": "assets/sky/sky_day.hdr", "folder": false, "importer": "image"}, + "identity": "69fe28de049e33d362a88e29fb8cd207eb86ece14f820cd2495f4ce6ab5e1a63", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_day", "index": 0, "id": "39808e0def899c47a24a89a3fab89e64"} + ] +} diff --git a/demo/packages/26356114f7faffcfafff109308dd9a32.mirai b/demo/packages/26356114f7faffcfafff109308dd9a32.mirai new file mode 100644 index 000000000..d4873aa32 --- /dev/null +++ b/demo/packages/26356114f7faffcfafff109308dd9a32.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "26356114f7faffcfafff109308dd9a32", + "name": "bricks_orm", + "origin": {"path": "assets/surfaces/bricks/bricks_orm.png", "folder": false, "importer": "image"}, + "identity": "b5b53e22299c99493478359f1e2936970ca5191915b380c995ea296bd95458fe", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "bricks_orm", "index": 0, "id": "d526fc286cf3d0bb3c956200037149d5"} + ] +} diff --git a/demo/packages/2d26a9a0d52c3edd23a71d2cc8d6e463.mirai b/demo/packages/2d26a9a0d52c3edd23a71d2cc8d6e463.mirai new file mode 100644 index 000000000..5b9c86c74 --- /dev/null +++ b/demo/packages/2d26a9a0d52c3edd23a71d2cc8d6e463.mirai @@ -0,0 +1,17 @@ +{ + "version": 1, + "id": "2d26a9a0d52c3edd23a71d2cc8d6e463", + "name": "robot", + "origin": {"path": "assets/characters/robot/robot.glb", "folder": false, "importer": "glTF"}, + "identity": "7afe8a1a9c863a77b6a70bb570e9e678e59d01af0c432e3c93d4d922f5425235", + "inputs": [], + "assets": [ + {"kind": "SCEN", "name": "robot", "index": 0, "id": "e431738105898dd368735a6ebb1d3c19"}, + {"kind": "MAT", "name": "robot blue", "index": 0, "id": "1c5c7311b02f9c27a26158a5dfbbfe47"}, + {"kind": "MAT", "name": "robot gold", "index": 1, "id": "985941907b0208e6eaef9ca10b6db8ba"}, + {"kind": "MAT", "name": "robot joints", "index": 2, "id": "d9b762e720d363036a57efa7b616eaef"}, + {"kind": "MAT", "name": "robot visor", "index": 3, "id": "ae20beadede617fdd239a5a6ad23eb83"}, + {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "83d297850b4eccf18f4225970eff51fb", "owner": "e431738105898dd368735a6ebb1d3c19"}, + {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "941beb67913839d9f4d4e05528cb0c80", "part": true, "owner": "e431738105898dd368735a6ebb1d3c19", "dependencies": ["1c5c7311b02f9c27a26158a5dfbbfe47", "83d297850b4eccf18f4225970eff51fb", "985941907b0208e6eaef9ca10b6db8ba", "ae20beadede617fdd239a5a6ad23eb83", "d9b762e720d363036a57efa7b616eaef"]} + ] +} diff --git a/demo/packages/2d6b0454a36971219d185d940f836ac7.mirai b/demo/packages/2d6b0454a36971219d185d940f836ac7.mirai new file mode 100644 index 000000000..de397c2ba --- /dev/null +++ b/demo/packages/2d6b0454a36971219d185d940f836ac7.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "2d6b0454a36971219d185d940f836ac7", + "name": "marble_normal", + "origin": {"path": "assets/surfaces/marble/marble_normal.png", "folder": false, "importer": "image"}, + "identity": "bc6f7c7783c0abc4aa24aa5951e9358ffcce1a75eabeba4a844305b51a82fecd", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "marble_normal", "index": 0, "id": "6efbfe1939e3a2a35c0ccc8621cc6abb"} + ] +} diff --git a/demo/packages/44342052da337fa065c60cf296a4466a.mirai b/demo/packages/44342052da337fa065c60cf296a4466a.mirai new file mode 100644 index 000000000..d6565cc96 --- /dev/null +++ b/demo/packages/44342052da337fa065c60cf296a4466a.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "44342052da337fa065c60cf296a4466a", + "name": "church_glass_color", + "origin": {"path": "assets/church/church_glass_color.png", "folder": false, "importer": "image"}, + "identity": "7a992dadb99729c9515c80e3b85b47c43e13b44bdfbedd0ff0815211b10596b7", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "church_glass_color", "index": 0, "id": "2767edae7841cea315b1e40d922c9950"} + ] +} diff --git a/demo/packages/4840a3f5a151cbe8c1b403b30fe8240d.mirai b/demo/packages/4840a3f5a151cbe8c1b403b30fe8240d.mirai new file mode 100644 index 000000000..22666e6dc --- /dev/null +++ b/demo/packages/4840a3f5a151cbe8c1b403b30fe8240d.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "4840a3f5a151cbe8c1b403b30fe8240d", + "name": "sky_overcast", + "origin": {"path": "assets/sky/sky_overcast.jpg", "folder": false, "importer": "image"}, + "identity": "badb56de863cc885d6f6554fa75d6f109b40416b073e2a3ae1e798674bfa030d", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_overcast", "index": 0, "id": "88a3a1a16a759ed20966d1c3ff54ada1"} + ] +} diff --git a/demo/packages/641ffc8940c220c3b2460f7a9eaa7862.mirai b/demo/packages/641ffc8940c220c3b2460f7a9eaa7862.mirai new file mode 100644 index 000000000..8cd88cc53 --- /dev/null +++ b/demo/packages/641ffc8940c220c3b2460f7a9eaa7862.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "641ffc8940c220c3b2460f7a9eaa7862", + "name": "brushed_normal", + "origin": {"path": "assets/surfaces/metal/brushed_normal.png", "folder": false, "importer": "image"}, + "identity": "ccde0d801f6b101dc0a1ee36b07dad836fe33412f22d3ac32723076196627c90", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "brushed_normal", "index": 0, "id": "b9466f1cd86f5f900d74e12980fdc702"} + ] +} diff --git a/demo/packages/6498e1a94e6c4621f925d22f8384243a.mirai b/demo/packages/6498e1a94e6c4621f925d22f8384243a.mirai new file mode 100644 index 000000000..5b41c9903 --- /dev/null +++ b/demo/packages/6498e1a94e6c4621f925d22f8384243a.mirai @@ -0,0 +1,16 @@ +{ + "version": 1, + "id": "6498e1a94e6c4621f925d22f8384243a", + "name": "plant", + "origin": {"path": "assets/gallery/plant.gltf", "folder": false, "importer": "glTF"}, + "identity": "5d76a337b138857552efbbc0ff81c70d8e38cf597e18c05eaa8f03ac8b63b278", + "inputs": ["leaves_color.png"], + "assets": [ + {"kind": "SCEN", "name": "plant", "index": 0, "id": "fc120101f1ddb302e135784184f05931"}, + {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "dba1b6bd3e8cde629938d41dc8934d02"}, + {"kind": "MAT", "name": "Terracotta", "index": 0, "id": "e2a8b0fcb9401808520b370016df35c9"}, + {"kind": "MAT", "name": "Soil", "index": 1, "id": "9337601926ea70572a4442c984759e15"}, + {"kind": "MAT", "name": "Leaves", "index": 2, "id": "7e248f2df1c8c849ff972c9a1ce95fb5", "dependencies": ["dba1b6bd3e8cde629938d41dc8934d02"]}, + {"kind": "MESH", "name": "Plant", "index": 0, "id": "72d9220e763e42b1da3107e0e5256b90", "part": true, "owner": "fc120101f1ddb302e135784184f05931", "dependencies": ["7e248f2df1c8c849ff972c9a1ce95fb5", "9337601926ea70572a4442c984759e15", "e2a8b0fcb9401808520b370016df35c9"]} + ] +} diff --git a/demo/packages/65805c17b2454730221e3d63066ce7f5.mirai b/demo/packages/65805c17b2454730221e3d63066ce7f5.mirai new file mode 100644 index 000000000..39cd3441b --- /dev/null +++ b/demo/packages/65805c17b2454730221e3d63066ce7f5.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "65805c17b2454730221e3d63066ce7f5", + "name": "grate_normal", + "origin": {"path": "assets/surfaces/metal/grate_normal.png", "folder": false, "importer": "image"}, + "identity": "84317f85bc995fe3ab6ae9962f1cd05bfd423bd05d87a2183232ecbececa02ca", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "grate_normal", "index": 0, "id": "c9d9c9f0f1dea3ec0ceec0f9671187eb"} + ] +} diff --git a/demo/packages/6cc37d40c23ded0fb9cc963a4dec2fbb.mirai b/demo/packages/6cc37d40c23ded0fb9cc963a4dec2fbb.mirai new file mode 100644 index 000000000..8256ca8d2 --- /dev/null +++ b/demo/packages/6cc37d40c23ded0fb9cc963a4dec2fbb.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "6cc37d40c23ded0fb9cc963a4dec2fbb", + "name": "stained_glass_color", + "origin": {"path": "assets/surfaces/glass/stained_glass_color.png", "folder": false, "importer": "image"}, + "identity": "d8b7a96453dd5e34e53b1e0870031149ef2eb57824041ee561e645cf333fb6c5", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "stained_glass_color", "index": 0, "id": "601bad68a6b7e349bc2655ac13c9196f"} + ] +} diff --git a/demo/packages/7572bd9ffa0dd2354005eff39b25f1bb.mirai b/demo/packages/7572bd9ffa0dd2354005eff39b25f1bb.mirai new file mode 100644 index 000000000..045fc563a --- /dev/null +++ b/demo/packages/7572bd9ffa0dd2354005eff39b25f1bb.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "7572bd9ffa0dd2354005eff39b25f1bb", + "name": "hum", + "origin": {"path": "assets/showcase/hum.wav", "folder": false, "importer": "sound"}, + "identity": "a136b93fa9f126831d67ac5f21d3453fd3c404372406ad7cbbca848e5f8cf9db", + "inputs": [], + "assets": [ + {"kind": "SND", "name": "hum", "index": 0, "id": "7f27bbcfa7a4b5fac01753762f745b26"} + ] +} diff --git a/demo/packages/780bee6597fbb860e803d513bda2dee7.mirai b/demo/packages/780bee6597fbb860e803d513bda2dee7.mirai new file mode 100644 index 000000000..288895207 --- /dev/null +++ b/demo/packages/780bee6597fbb860e803d513bda2dee7.mirai @@ -0,0 +1,16 @@ +{ + "version": 1, + "id": "780bee6597fbb860e803d513bda2dee7", + "name": "jelly", + "origin": {"path": "assets/characters/jelly/jelly.gltf", "folder": false, "importer": "glTF"}, + "identity": "a8aa4182d7481e139c7925b475ece930212b1ddf640d0b4da7342ca25c2d26af", + "inputs": ["jelly.bin"], + "assets": [ + {"kind": "SCEN", "name": "jelly", "index": 0, "id": "107c9b8f4bf9dfe8936235eae8e042a6"}, + {"kind": "MAT", "name": "Candy", "index": 0, "id": "a18adcad4fedf21125038a819d27acd2"}, + {"kind": "SKEL", "name": "Jelly bones", "index": 0, "id": "dae452ce18d59e780ff745f7d1b43697", "owner": "107c9b8f4bf9dfe8936235eae8e042a6"}, + {"kind": "MESH", "name": "Party Jelly", "index": 0, "id": "25a3e22ecb9c265a1614515f868efc52", "part": true, "owner": "107c9b8f4bf9dfe8936235eae8e042a6", "dependencies": ["a18adcad4fedf21125038a819d27acd2"]}, + {"kind": "MESH", "name": "Party Jelly (1)", "index": 1, "id": "1c00bfb5bceb2619036b114104f42040", "part": true, "owner": "107c9b8f4bf9dfe8936235eae8e042a6", "dependencies": ["a18adcad4fedf21125038a819d27acd2", "dae452ce18d59e780ff745f7d1b43697"]}, + {"kind": "ANIM", "name": "jelly", "index": 0, "id": "0d555b0da56949fb3cdbdf41410432f4", "dependencies": ["dae452ce18d59e780ff745f7d1b43697"]} + ] +} diff --git a/demo/packages/81d18317bfbbf0851f30ebcb2de41b67.mirai b/demo/packages/81d18317bfbbf0851f30ebcb2de41b67.mirai new file mode 100644 index 000000000..4ba5c9d0e --- /dev/null +++ b/demo/packages/81d18317bfbbf0851f30ebcb2de41b67.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "81d18317bfbbf0851f30ebcb2de41b67", + "name": "grate_orm", + "origin": {"path": "assets/surfaces/metal/grate_orm.png", "folder": false, "importer": "image"}, + "identity": "a135326e4f00d4d96a7c4e6b64637be8940ab66db13f2d4747db018ce7c753f3", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "grate_orm", "index": 0, "id": "71e4b6effc8f9e4ac6c7f1dc0f02612e"} + ] +} diff --git a/demo/packages/84060b2db086d874fb6a99a638a79a10.mirai b/demo/packages/84060b2db086d874fb6a99a638a79a10.mirai new file mode 100644 index 000000000..dae073294 --- /dev/null +++ b/demo/packages/84060b2db086d874fb6a99a638a79a10.mirai @@ -0,0 +1,17 @@ +{ + "version": 1, + "id": "84060b2db086d874fb6a99a638a79a10", + "name": "robot", + "origin": {"path": "assets/characters/robot/robot.gltf", "folder": false, "importer": "glTF"}, + "identity": "4e42171c4b4f4eeb1e541f249c84ef5c3c9e2d15c0224e9dc3371ebbe5d37c53", + "inputs": ["robot.bin"], + "assets": [ + {"kind": "SCEN", "name": "robot", "index": 0, "id": "b0f0241df6e014392af304b56cf59eab"}, + {"kind": "MAT", "name": "robot blue", "index": 0, "id": "c67757fb8b041b0d6b6178fe958bd91e"}, + {"kind": "MAT", "name": "robot gold", "index": 1, "id": "feebc67eaeb9081ec3d1f1edac208b2a"}, + {"kind": "MAT", "name": "robot joints", "index": 2, "id": "a453277713f405f86fd3dc27c2506977"}, + {"kind": "MAT", "name": "robot visor", "index": 3, "id": "0bf1bd3d54bcb875b34957598640c69d"}, + {"kind": "SKEL", "name": "13 joint robot rig", "index": 0, "id": "cf5b1fe881d020461927fe9b85c42136", "owner": "b0f0241df6e014392af304b56cf59eab"}, + {"kind": "MESH", "name": "Mirai robot", "index": 0, "id": "3a94f8ad990aede6566ad2af8010bb23", "part": true, "owner": "b0f0241df6e014392af304b56cf59eab", "dependencies": ["0bf1bd3d54bcb875b34957598640c69d", "a453277713f405f86fd3dc27c2506977", "c67757fb8b041b0d6b6178fe958bd91e", "cf5b1fe881d020461927fe9b85c42136", "feebc67eaeb9081ec3d1f1edac208b2a"]} + ] +} diff --git a/demo/packages/856074124b2e8924607a1e79488b6abd.mirai b/demo/packages/856074124b2e8924607a1e79488b6abd.mirai new file mode 100644 index 000000000..0e67bb0e6 --- /dev/null +++ b/demo/packages/856074124b2e8924607a1e79488b6abd.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "856074124b2e8924607a1e79488b6abd", + "name": "neon_emission", + "origin": {"path": "assets/surfaces/neon/neon_emission.png", "folder": false, "importer": "image"}, + "identity": "e7133ec38468f5414b4e8f8a65e57e4e065b176b21049819ef50ca0a852eed76", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "neon_emission", "index": 0, "id": "6d9362793631542f0b778005f9fd8dd0"} + ] +} diff --git a/demo/packages/a03a7d06fb4c433e89ba1ecd314d91ed.mirai b/demo/packages/a03a7d06fb4c433e89ba1ecd314d91ed.mirai new file mode 100644 index 000000000..ac0bdfa3b --- /dev/null +++ b/demo/packages/a03a7d06fb4c433e89ba1ecd314d91ed.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "a03a7d06fb4c433e89ba1ecd314d91ed", + "name": "brushed_orm", + "origin": {"path": "assets/surfaces/metal/brushed_orm.png", "folder": false, "importer": "image"}, + "identity": "1ec644f5675f0f31c435f1531200733cd7404a1dfc65c6ff16e37b9ac6040feb", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "brushed_orm", "index": 0, "id": "ec4b617cf48145349c2c76f465e19ae6"} + ] +} diff --git a/demo/packages/a270e82878cb164498ba98398060ccb8.mirai b/demo/packages/a270e82878cb164498ba98398060ccb8.mirai new file mode 100644 index 000000000..45812d8f0 --- /dev/null +++ b/demo/packages/a270e82878cb164498ba98398060ccb8.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "a270e82878cb164498ba98398060ccb8", + "name": "tiles_color", + "origin": {"path": "assets/surfaces/tiles/tiles_color.png", "folder": false, "importer": "image"}, + "identity": "3915769e8b5d9cc9af39584990a4fb69ac1f58764adc66d3fb4094005497f149", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "tiles_color", "index": 0, "id": "cd3d66765e834e0c508906257f888eba"} + ] +} diff --git a/demo/packages/a278db7076e649a48933fc75b0dc1a4c.mirai b/demo/packages/a278db7076e649a48933fc75b0dc1a4c.mirai new file mode 100644 index 000000000..1cf0235e5 --- /dev/null +++ b/demo/packages/a278db7076e649a48933fc75b0dc1a4c.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "a278db7076e649a48933fc75b0dc1a4c", + "name": "cobblestone_normal", + "origin": {"path": "assets/surfaces/cobblestone/cobblestone_normal.png", "folder": false, "importer": "image"}, + "identity": "c96c505d447b06db5058734fe8e034334fba141d57eaeca7f85ae25802a45f68", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "cobblestone_normal", "index": 0, "id": "1a08080be2a08b5fe300ec425923c195"} + ] +} diff --git a/demo/packages/a3799fb21f329ae819440d1726f2aa20.mirai b/demo/packages/a3799fb21f329ae819440d1726f2aa20.mirai new file mode 100644 index 000000000..8926a2be4 --- /dev/null +++ b/demo/packages/a3799fb21f329ae819440d1726f2aa20.mirai @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "a3799fb21f329ae819440d1726f2aa20", + "name": "vase", + "origin": {"path": "assets/gallery/vase.obj", "folder": false, "importer": "OBJ"}, + "identity": "5875aea8a04578f3158c7f275449d9b33f78c1b058906552335a81afd01f7fe4", + "inputs": ["vase.mtl", "vase_color.png"], + "assets": [ + {"kind": "SCEN", "name": "vase", "index": 0, "id": "f430e11663e1ae229bf1511ac0c7f82d"}, + {"kind": "TEX", "name": "vase_color", "index": 0, "id": "f8b0959094ef4d6757a87f85ec92a493"}, + {"kind": "MAT", "name": "Ceramic", "index": 0, "id": "757da5a655983f642167f358f7add279", "dependencies": ["f8b0959094ef4d6757a87f85ec92a493"]}, + {"kind": "MAT", "name": "Gilding", "index": 1, "id": "be0bbaf3f4f7c9fc87a15c7d08b0bb18"}, + {"kind": "MESH", "name": "Vase", "index": 0, "id": "0d6f52247150edd52ac17287d54e7bf9", "part": true, "owner": "f430e11663e1ae229bf1511ac0c7f82d", "dependencies": ["757da5a655983f642167f358f7add279", "be0bbaf3f4f7c9fc87a15c7d08b0bb18"]} + ] +} diff --git a/demo/packages/a68982d58a94bfa0fcb588bf93dfafec.mirai b/demo/packages/a68982d58a94bfa0fcb588bf93dfafec.mirai new file mode 100644 index 000000000..4c5f5539f --- /dev/null +++ b/demo/packages/a68982d58a94bfa0fcb588bf93dfafec.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "a68982d58a94bfa0fcb588bf93dfafec", + "name": "marble_orm", + "origin": {"path": "assets/surfaces/marble/marble_orm.png", "folder": false, "importer": "image"}, + "identity": "318157572d9864452082d14787cc9ac8eb7cf3ce56cf81609d20996ef34a273b", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "marble_orm", "index": 0, "id": "972fc9634ba6242d0b5f33e03e3a0741"} + ] +} diff --git a/demo/packages/ae40765be8d60ddad02e47bb21203f8f.mirai b/demo/packages/ae40765be8d60ddad02e47bb21203f8f.mirai new file mode 100644 index 000000000..d748f76a4 --- /dev/null +++ b/demo/packages/ae40765be8d60ddad02e47bb21203f8f.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "ae40765be8d60ddad02e47bb21203f8f", + "name": "cobblestone_orm", + "origin": {"path": "assets/surfaces/cobblestone/cobblestone_orm.png", "folder": false, "importer": "image"}, + "identity": "2a95d7c8e5402ec5bdc2f7e01aed3c0ed96d37d9ed066c08c7c1e944ac88a630", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "cobblestone_orm", "index": 0, "id": "94cef7c053d5afed918f83a3d6242853"} + ] +} diff --git a/demo/packages/c0d7666d947e65fc7dd68a31013c0939.mirai b/demo/packages/c0d7666d947e65fc7dd68a31013c0939.mirai new file mode 100644 index 000000000..7069a3086 --- /dev/null +++ b/demo/packages/c0d7666d947e65fc7dd68a31013c0939.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "c0d7666d947e65fc7dd68a31013c0939", + "name": "cobblestone_color", + "origin": {"path": "assets/surfaces/cobblestone/cobblestone_color.png", "folder": false, "importer": "image"}, + "identity": "4bf20aa0c9be138d32c781598a709118b64335c81d7d9b86aec7b3ddf9efbf07", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "cobblestone_color", "index": 0, "id": "9a69320baec0a94ab9eb8d2010a8a8a4"} + ] +} diff --git a/demo/packages/c1a66f64c05efe53cca206765ab98117.mirai b/demo/packages/c1a66f64c05efe53cca206765ab98117.mirai new file mode 100644 index 000000000..f5ab52504 --- /dev/null +++ b/demo/packages/c1a66f64c05efe53cca206765ab98117.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "c1a66f64c05efe53cca206765ab98117", + "name": "sky_night", + "origin": {"path": "assets/sky/sky_night.jpg", "folder": false, "importer": "image"}, + "identity": "a6bc49bf6d7489954dafb9d55d279df060250cc8f6d8366cbd48b1501a6ee048", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_night", "index": 0, "id": "8a6329018eb7b53a0215bd4bdbf5a2c9"} + ] +} diff --git a/demo/packages/ca27d206e666815335b68ee9c8900ab7.mirai b/demo/packages/ca27d206e666815335b68ee9c8900ab7.mirai new file mode 100644 index 000000000..0e1970ea2 --- /dev/null +++ b/demo/packages/ca27d206e666815335b68ee9c8900ab7.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "ca27d206e666815335b68ee9c8900ab7", + "name": "vase_color", + "origin": {"path": "assets/gallery/vase_color.png", "folder": false, "importer": "image"}, + "identity": "f2fb83e9ce3a4a2aeba751a76a796edb4ee13ec9450b9011befd02e0348d1c30", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "vase_color", "index": 0, "id": "c94299d672a2e4371f7a5374961b418c"} + ] +} diff --git a/demo/packages/caae575f45863b717aafbe7fe73862c0.mirai b/demo/packages/caae575f45863b717aafbe7fe73862c0.mirai new file mode 100644 index 000000000..ea539fcd0 --- /dev/null +++ b/demo/packages/caae575f45863b717aafbe7fe73862c0.mirai @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "caae575f45863b717aafbe7fe73862c0", + "name": "leaves_color", + "origin": {"path": "assets/gallery/leaves_color.png", "folder": false, "importer": "image"}, + "identity": "30b62e5a3a6b818103845cb1db9d509622a31605dd8f4cdc740053311340fa93", + "settings": { + "image": "Color", + "alpha_coverage": 0.5 + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "leaves_color", "index": 0, "id": "df49540f1f9dbcf6fe2e612ea427757e"} + ] +} diff --git a/demo/packages/de7548cbbad3494b21280744cd16ed68.mirai b/demo/packages/de7548cbbad3494b21280744cd16ed68.mirai new file mode 100644 index 000000000..2099e9dad --- /dev/null +++ b/demo/packages/de7548cbbad3494b21280744cd16ed68.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "de7548cbbad3494b21280744cd16ed68", + "name": "tiles_orm", + "origin": {"path": "assets/surfaces/tiles/tiles_orm.png", "folder": false, "importer": "image"}, + "identity": "032ec4ed33e3897d8890d072eaf44ae99ea73024adfe597e062e583e61c06b88", + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "tiles_orm", "index": 0, "id": "84dbfd2533a249a9716c78d548c6c654"} + ] +} diff --git a/demo/packages/e022a803a1c6fb9fe4b86ab65a48bc24.mirai b/demo/packages/e022a803a1c6fb9fe4b86ab65a48bc24.mirai new file mode 100644 index 000000000..1973c3a07 --- /dev/null +++ b/demo/packages/e022a803a1c6fb9fe4b86ab65a48bc24.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "e022a803a1c6fb9fe4b86ab65a48bc24", + "name": "sky_sunset", + "origin": {"path": "assets/sky/sky_sunset.jpg", "folder": false, "importer": "image"}, + "identity": "0b4e80818fedc88b32a0b6550eec6158e2727bfd1d99253c232e50cb828668cb", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "sky_sunset", "index": 0, "id": "7b819d061e0487f04a17587b8ff0bce1"} + ] +} diff --git a/demo/packages/e0495081646e8954b731fc082bcd8884.mirai b/demo/packages/e0495081646e8954b731fc082bcd8884.mirai new file mode 100644 index 000000000..3b211455b --- /dev/null +++ b/demo/packages/e0495081646e8954b731fc082bcd8884.mirai @@ -0,0 +1,15 @@ +{ + "version": 1, + "id": "e0495081646e8954b731fc082bcd8884", + "name": "grate_color", + "origin": {"path": "assets/surfaces/metal/grate_color.png", "folder": false, "importer": "image"}, + "identity": "f65a1b168299988cb3046b4003612437577bb217f7afbbeae8dada3c085b60c2", + "settings": { + "image": "Color", + "alpha_coverage": 0.5 + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "grate_color", "index": 0, "id": "90b31208c9dfc6a6d00ab5517f084571"} + ] +} diff --git a/demo/packages/e96915c067e6fe50d517b59bf3b17d61.mirai b/demo/packages/e96915c067e6fe50d517b59bf3b17d61.mirai new file mode 100644 index 000000000..5e25a58e7 --- /dev/null +++ b/demo/packages/e96915c067e6fe50d517b59bf3b17d61.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "e96915c067e6fe50d517b59bf3b17d61", + "name": "planks_color", + "origin": {"path": "assets/surfaces/wood/planks_color.png", "folder": false, "importer": "image"}, + "identity": "49264e2dfeefc900bf103a7d93f6c80ec28d75a409eb86d103cdf0c1313bf953", + "settings": { + "image": "Color" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "planks_color", "index": 0, "id": "79a53ef6d585bc9fd1a912a9e592f913"} + ] +} diff --git a/demo/packages/f4ccf188ab1bf19a62c53faaef829fba.mirai b/demo/packages/f4ccf188ab1bf19a62c53faaef829fba.mirai new file mode 100644 index 000000000..a9e7b0588 --- /dev/null +++ b/demo/packages/f4ccf188ab1bf19a62c53faaef829fba.mirai @@ -0,0 +1,14 @@ +{ + "version": 1, + "id": "f4ccf188ab1bf19a62c53faaef829fba", + "name": "planks_normal", + "origin": {"path": "assets/surfaces/wood/planks_normal.png", "folder": false, "importer": "image"}, + "identity": "be4569af461f13cc92440f370b2e097d6c0e4fd8119656525b343b7695c98dcc", + "settings": { + "image": "Normal" + }, + "inputs": [], + "assets": [ + {"kind": "TEX", "name": "planks_normal", "index": 0, "id": "0558bb6aa0ec012cac33f59d343e99ed"} + ] +} diff --git a/docs/game/assets.md b/docs/game/assets.md index 461d419f9..b2abb723f 100644 --- a/docs/game/assets.md +++ b/docs/game/assets.md @@ -32,10 +32,10 @@ mirai import assets/player/step.wav # assets/player/step.sound - A model writes meshes, materials, textures, skeletons, clips and a prefab: instantiate that. - An image is color unless its name says otherwise (`normal`, `_n`, `rough`, `metal`, `orm`, `_ao`, `height`, `mask`); force it with `--srgb`, `--data`, `--normal`, `--normal-green-down`. -- `.import` beside the source keeps the flags, the folder of its assets and the id of each - asset, so importing again updates in place, keeping ids and material edits: commit it, and move - it with its source. A part renamed in the source is a new asset. -- Meshes, textures, sounds and fonts are built into `.mirai/imported/`, never committed: `mirai +- The import adds the source's package to `packages.mirai`, with its manifest in `packages/`: + the flags, the folder of its assets and the id of each asset, so importing again updates in + place, keeping ids and material edits. Commit both. A part renamed in the source is a new asset. +- Meshes, textures, sounds and fonts are built into the machine's store, never committed: `mirai build` makes them again on a fresh clone. Edit their source, not them. - A mesh keeps only the colliders its source brings; placed, it collides only with them. An `Auto` `Collider` without them is its triangles when still, convex pieces of them when it moves. diff --git a/docs/game/assets/content-tool.md b/docs/game/assets/content-tool.md index d4ff92090..7c19d9d0b 100644 --- a/docs/game/assets/content-tool.md +++ b/docs/game/assets/content-tool.md @@ -55,9 +55,9 @@ hum:=report.assets[0].id // random the first time, kept after - Write the WAV only when its bytes differ from the file on disk, so an unchanged source is imported as the same file. -- The id an import returns is random the first time and the one `.import` keeps from then - on: read it with `asset.list`; never in a constant. Commit the WAV and its `.import`; the - `.sound` is built again from them. +- The id an import returns is random the first time and the one the package of the source keeps + from then on: read it with `asset.list`; never in a constant. Commit the WAV, `packages.mirai` + and `packages/`; the `.sound` is built again from them. ## Sound set @@ -79,8 +79,8 @@ A tool can import art too, with the function `mirai import` calls: report,error:=sources.import_source("assets/arena/crate.glb") ``` -`sources` is `mirai:cook/import/sources`. It keeps the settings and the ids in `.import` -beside the source, so a tool and `mirai import` give the same assets. +`sources` is `mirai:cook/import/sources`. It keeps the settings and the ids in the package of the +source, so a tool and `mirai import` give the same assets. ## Pitfalls that fail silently diff --git a/docs/game/ship.md b/docs/game/ship.md index 2d1fe6596..6b2f1264c 100644 --- a/docs/game/ship.md +++ b/docs/game/ship.md @@ -53,6 +53,6 @@ says how to fix each finding. - An asset only code names works in `mirai play` and is missing from `build game` unless it is a [root](assets.md#reference). Build before delivering. -- Art sources and their sidecars never ship, and a game package that imports an editor package - fails the build with `Engine_Time_Code`. +- Art sources and the manifests of their packages never ship, and a game package that imports + an editor package fails the build with `Engine_Time_Code`. - The shipped game on Windows is a console program. diff --git a/editor/assets/presence/source/pup.glb.import b/editor/assets/presence/source/pup.glb.import deleted file mode 100644 index 04cae8dbd..000000000 --- a/editor/assets/presence/source/pup.glb.import +++ /dev/null @@ -1,12 +0,0 @@ -{ - "textures": "Blocks", - "rdo_lambda": 0, - "image": "Data", - "alpha_coverage": 0, - "lods": "Auto", - "parts": "All", - "front": "Auto", - "rig": "Source", - "rig_crotch": 0, - "rig_neck": 0 -} diff --git a/editor/import.odin b/editor/import.odin index ce17cf6c7..8be87d6ce 100644 --- a/editor/import.odin +++ b/editor/import.odin @@ -162,7 +162,7 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb source:=source // A source a plugin imports whole goes before the formats, which may read its bytes too. whole,code,claimed:=import_whole(s,source) - if _,in_assets:=sources.source_root(source,context.temp_allocator);!in_assets { + if _,in_assets:=importer.source_root(source,context.temp_allocator);!in_assets { for at:=filepath.dir(project.path_absolute(source,context.temp_allocator));filepath.dir(at)!=at;at=filepath.dir(at) { if !project.path_same(at,s.project_root) {continue} import_say(s,"%s is in the project but in no assets folder: move it under assets/, beside what it makes, and import it there",name) @@ -175,7 +175,7 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb import_say(s,"%s: %s",name,bring_error.message) return false } - if _,inside:=sources.source_root(brought,context.temp_allocator);!inside { + if _,inside:=importer.source_root(brought,context.temp_allocator);!inside { import_say(s,"%s: %s brought it to %s, which is in no assets folder",name,whole.name,brought) return false } @@ -193,7 +193,7 @@ import_file :: proc(s:^Session,source:string,folder:="",profiled:=false,rig:Mayb source=copied } } - root,_:=sources.source_root(source,context.temp_allocator) + root,_:=importer.source_root(source,context.temp_allocator) allocator:=runtime.heap_allocator() options:=importer.Import_Options{ root=strings.clone(root,allocator), @@ -426,7 +426,7 @@ import_root :: proc(s:^Session,r:^asset.Registry,id:base.Asset_Id) -> (sidecar:s _,info,imported,error:=asset.import_peek(full,&types) if error!=.None || !imported {return} source:=base.text_get(info.source[:]) - root,in_assets:=sources.source_root(full,context.temp_allocator) + root,in_assets:=importer.source_root(full,context.temp_allocator) if file.listed || !in_assets {root=s.project_root} for mount in r.mounts { if file.listed && strings.has_prefix(file.path,mount.prefix) {root=filepath.dir(mount.folder)} diff --git a/plugins/prompts_kenney/assets/kenney.prompts b/plugins/prompts_kenney/assets/kenney.prompts index 9cb997c1f37ed0dfd4a8465763ad6f3ccb0af74f..48133b6415de4b72973e8b216c676cb1d8942603 100644 GIT binary patch delta 31 ncmZ3$zkq*&2S@1@!DUKUdZ%v;IKetup6&1C05(Z delta 35 rcmZ3$zkq*&2S@0OIT9tzwZR($PO!2k78j?M6zflxVEa2+hP?#<>%0v1 diff --git a/plugins/prompts_kenney/packages.mirai b/plugins/prompts_kenney/packages.mirai new file mode 100644 index 000000000..6b2f7d15b --- /dev/null +++ b/plugins/prompts_kenney/packages.mirai @@ -0,0 +1,6 @@ +{ + "version": 1, + "packages": [ + {"id": "18e1ec191e6b7a7386eba31ba4c8e829", "manifest": "f268bd4d578161613590410938762579f4ab68f33dfab2be955f84500b07c098", "folder": "", "assets": []} + ] +} diff --git a/plugins/prompts_kenney/assets/kenney.prompts.json.import b/plugins/prompts_kenney/packages/18e1ec191e6b7a7386eba31ba4c8e829.mirai similarity index 88% rename from plugins/prompts_kenney/assets/kenney.prompts.json.import rename to plugins/prompts_kenney/packages/18e1ec191e6b7a7386eba31ba4c8e829.mirai index 91d7cc194..6e75758d4 100644 --- a/plugins/prompts_kenney/assets/kenney.prompts.json.import +++ b/plugins/prompts_kenney/packages/18e1ec191e6b7a7386eba31ba4c8e829.mirai @@ -1,10 +1,13 @@ { - "version": 3, - "settings": {}, + "version": 1, + "id": "18e1ec191e6b7a7386eba31ba4c8e829", + "name": "kenney", + "origin": {"path": "assets/kenney.prompts.json", "folder": false, "importer": "prompts"}, + "identity": "e03872ef8c90780fb08f7d102e9c319228e918ad57c2989736406be967112549", "inputs": ["keyboard_mouse/keyboard_0.png", "keyboard_mouse/keyboard_1.png", "keyboard_mouse/keyboard_2.png", "keyboard_mouse/keyboard_3.png", "keyboard_mouse/keyboard_4.png", "keyboard_mouse/keyboard_5.png", "keyboard_mouse/keyboard_6.png", "keyboard_mouse/keyboard_7.png", "keyboard_mouse/keyboard_8.png", "keyboard_mouse/keyboard_9.png", "keyboard_mouse/keyboard_a.png", "keyboard_mouse/keyboard_alt.png", "keyboard_mouse/keyboard_apostrophe.png", "keyboard_mouse/keyboard_arrow_down.png", "keyboard_mouse/keyboard_arrow_left.png", "keyboard_mouse/keyboard_arrow_right.png", "keyboard_mouse/keyboard_arrow_up.png", "keyboard_mouse/keyboard_b.png", "keyboard_mouse/keyboard_backspace.png", "keyboard_mouse/keyboard_bracket_close.png", "keyboard_mouse/keyboard_bracket_open.png", "keyboard_mouse/keyboard_c.png", "keyboard_mouse/keyboard_comma.png", "keyboard_mouse/keyboard_ctrl.png", "keyboard_mouse/keyboard_d.png", "keyboard_mouse/keyboard_delete.png", "keyboard_mouse/keyboard_e.png", "keyboard_mouse/keyboard_end.png", "keyboard_mouse/keyboard_enter.png", "keyboard_mouse/keyboard_equals.png", "keyboard_mouse/keyboard_escape.png", "keyboard_mouse/keyboard_f.png", "keyboard_mouse/keyboard_f1.png", "keyboard_mouse/keyboard_f10.png", "keyboard_mouse/keyboard_f11.png", "keyboard_mouse/keyboard_f12.png", "keyboard_mouse/keyboard_f2.png", "keyboard_mouse/keyboard_f3.png", "keyboard_mouse/keyboard_f4.png", "keyboard_mouse/keyboard_f5.png", "keyboard_mouse/keyboard_f6.png", "keyboard_mouse/keyboard_f7.png", "keyboard_mouse/keyboard_f8.png", "keyboard_mouse/keyboard_f9.png", "keyboard_mouse/keyboard_g.png", "keyboard_mouse/keyboard_h.png", "keyboard_mouse/keyboard_home.png", "keyboard_mouse/keyboard_i.png", "keyboard_mouse/keyboard_insert.png", "keyboard_mouse/keyboard_j.png", "keyboard_mouse/keyboard_k.png", "keyboard_mouse/keyboard_l.png", "keyboard_mouse/keyboard_m.png", "keyboard_mouse/keyboard_minus.png", "keyboard_mouse/keyboard_n.png", "keyboard_mouse/keyboard_o.png", "keyboard_mouse/keyboard_p.png", "keyboard_mouse/keyboard_page_down.png", "keyboard_mouse/keyboard_page_up.png", "keyboard_mouse/keyboard_period.png", "keyboard_mouse/keyboard_q.png", "keyboard_mouse/keyboard_r.png", "keyboard_mouse/keyboard_s.png", "keyboard_mouse/keyboard_semicolon.png", "keyboard_mouse/keyboard_shift.png", "keyboard_mouse/keyboard_slash_back.png", "keyboard_mouse/keyboard_slash_forward.png", "keyboard_mouse/keyboard_space.png", "keyboard_mouse/keyboard_t.png", "keyboard_mouse/keyboard_tab.png", "keyboard_mouse/keyboard_tilde.png", "keyboard_mouse/keyboard_u.png", "keyboard_mouse/keyboard_v.png", "keyboard_mouse/keyboard_w.png", "keyboard_mouse/keyboard_win.png", "keyboard_mouse/keyboard_x.png", "keyboard_mouse/keyboard_y.png", "keyboard_mouse/keyboard_z.png", "keyboard_mouse/mouse_left.png", "keyboard_mouse/mouse_move.png", "keyboard_mouse/mouse_right.png", "keyboard_mouse/mouse_scroll.png", "keyboard_mouse/mouse_scroll_vertical.png", "playstation/playstation5_button_create.png", "playstation/playstation5_button_options.png", "playstation/playstation_button_color_circle.png", "playstation/playstation_button_color_cross.png", "playstation/playstation_button_color_square.png", "playstation/playstation_button_color_triangle.png", "playstation/playstation_dpad.png", "playstation/playstation_dpad_down.png", "playstation/playstation_dpad_left.png", "playstation/playstation_dpad_right.png", "playstation/playstation_dpad_up.png", "playstation/playstation_stick_l.png", "playstation/playstation_stick_l_press.png", "playstation/playstation_stick_r.png", "playstation/playstation_stick_r_press.png", "playstation/playstation_trigger_l1.png", "playstation/playstation_trigger_l2.png", "playstation/playstation_trigger_r1.png", "playstation/playstation_trigger_r2.png", "xbox/xbox_button_color_a.png", "xbox/xbox_button_color_b.png", "xbox/xbox_button_color_x.png", "xbox/xbox_button_color_y.png", "xbox/xbox_button_menu.png", "xbox/xbox_button_view.png", "xbox/xbox_dpad.png", "xbox/xbox_dpad_down.png", "xbox/xbox_dpad_left.png", "xbox/xbox_dpad_right.png", "xbox/xbox_dpad_up.png", "xbox/xbox_lb.png", "xbox/xbox_lt.png", "xbox/xbox_rb.png", "xbox/xbox_rt.png", "xbox/xbox_stick_l.png", "xbox/xbox_stick_l_press.png", "xbox/xbox_stick_r.png", "xbox/xbox_stick_r_press.png"], - "outputs": [ - {"kind": "PRMT", "name": "kenney", "index": 0, "id": "81aa127badfbdbeb988b5d3a1c12ab30"}, - {"kind": "TEX", "name": "kenney_atlas_0", "index": 0, "id": "53a2b283de1b99ca9e8a22525430f426", "part": true}, - {"kind": "TEX", "name": "kenney_atlas_1", "index": 1, "id": "818deac4a87429281666b4b0ec524d02", "part": true} + "assets": [ + {"kind": "PRMT", "name": "kenney", "index": 0, "id": "81aa127badfbdbeb988b5d3a1c12ab30", "dependencies": ["53a2b283de1b99ca9e8a22525430f426", "818deac4a87429281666b4b0ec524d02"]}, + {"kind": "TEX", "name": "kenney_atlas_0", "index": 0, "id": "53a2b283de1b99ca9e8a22525430f426", "part": true, "owner": "81aa127badfbdbeb988b5d3a1c12ab30"}, + {"kind": "TEX", "name": "kenney_atlas_1", "index": 1, "id": "818deac4a87429281666b4b0ec524d02", "part": true, "owner": "81aa127badfbdbeb988b5d3a1c12ab30"} ] } diff --git a/templates/game/assets/arena/hum.wav.import b/templates/game/assets/arena/hum.wav.import deleted file mode 100644 index 8bbc7bae5..000000000 --- a/templates/game/assets/arena/hum.wav.import +++ /dev/null @@ -1,7 +0,0 @@ -{ - "version": 3, - "settings": {}, - "outputs": [ - {"kind": "SND", "name": "hum", "index": 0, "id": "7f27bbcfa7a4b5fac01753762f745b26"} - ] -} diff --git a/templates/game/packages.mirai b/templates/game/packages.mirai new file mode 100644 index 000000000..3e00c14de --- /dev/null +++ b/templates/game/packages.mirai @@ -0,0 +1,6 @@ +{ + "version": 1, + "packages": [ + {"id": "ea1313ae2ad84fe489d5bf4061c5fda9", "manifest": "9f95b82ed704973291889f56bedd45b35f1631a3736280baf4b15e0736c760dd", "folder": "arena", "assets": []} + ] +} diff --git a/templates/game/packages/ea1313ae2ad84fe489d5bf4061c5fda9.mirai b/templates/game/packages/ea1313ae2ad84fe489d5bf4061c5fda9.mirai new file mode 100644 index 000000000..1a3a02195 --- /dev/null +++ b/templates/game/packages/ea1313ae2ad84fe489d5bf4061c5fda9.mirai @@ -0,0 +1,11 @@ +{ + "version": 1, + "id": "ea1313ae2ad84fe489d5bf4061c5fda9", + "name": "hum", + "origin": {"path": "assets/arena/hum.wav", "folder": false, "importer": "sound"}, + "identity": "e309f8873a2e4ee2e258c6d26d667da121324047c403dcbe0b87ee4376231c53", + "inputs": [], + "assets": [ + {"kind": "SND", "name": "hum", "index": 0, "id": "7f27bbcfa7a4b5fac01753762f745b26"} + ] +} From 1f99bdf372050906e3cf243547e1aa70ba3e39bf Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 20:11:32 -0300 Subject: [PATCH 022/123] asset: say how to fix a store of another layout Co-Authored-By: Claude Fable 5.2 --- asset/packages_root.odin | 5 ++++- docs/game/assets.md | 5 ++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/asset/packages_root.odin b/asset/packages_root.odin index 1a47d705d..5d2e306f0 100644 --- a/asset/packages_root.odin +++ b/asset/packages_root.odin @@ -39,7 +39,10 @@ root_packages :: proc(root,store_root:string,allocator:=context.allocator) -> (r switch open_error { case .None: case .Out_Of_Memory: return {},.Out_Of_Memory - case .Missing,.Not_Here_Yet,.Corrupt,.Invalid_Argument,.Io,.Unsupported_Layout: + case .Unsupported_Layout: + problem_line(&problems,"the store at %s was written by another version of Mirai, so the packages of %s are not listed: remove that folder and build again, which fills it",folder,root) or_return + return {problems=problems[:]},.None + case .Missing,.Not_Here_Yet,.Corrupt,.Wrong_Size,.Invalid_Argument,.Io: problem_line(&problems,"the store at %s does not open (%v), so the packages of %s are not listed",folder,open_error,root) or_return return {problems=problems[:]},.None } diff --git a/docs/game/assets.md b/docs/game/assets.md index f426fde7d..8bd897826 100644 --- a/docs/game/assets.md +++ b/docs/game/assets.md @@ -33,9 +33,8 @@ mirai import assets/player/step.wav # assets/player/step.sound - An image is color unless its name says otherwise (`normal`, `_n`, `rough`, `metal`, `orm`, `_ao`, `height`, `mask`); force it with `--srgb`, `--data`, `--normal`, `--normal-green-down`. - An import leaves the source where it is and adds its package to `packages.mirai`, with its - manifest in `packages/`: the flags, the folder of its assets and the id of each asset, so - importing again updates in place, keeping ids and material edits. Commit both. A part renamed in - the source is a new asset. + manifest in `packages/` (flags, folder, the id of each asset), so importing again updates in + place, keeping ids and material edits. Commit both. A part renamed in the source is a new asset. - Meshes, textures, sounds and fonts are built into the machine's store, never committed: `mirai build` makes them again on a fresh clone. Edit their source, not them. - A mesh keeps only the colliders its source brings; placed, it collides only with them. An `Auto` From ba97dd6cf9e93bb7e597bac392b0be489fe3c381 Mon Sep 17 00:00:00 2001 From: Tomas Zbk Date: Wed, 18 Nov 2026 20:45:37 -0300 Subject: [PATCH 023/123] build: refresh and cook from the library, ship plugin builds in the installed engine Every build refreshes the roots with a packages.mirai in the store it is given, requiring Final for a cook; a machine whose store holds the builds builds and ships without sources, and a missing build fails by package name. The cook lists the packages of the project and its plugins from the store, leaves the sources of packages inside a root out of the game and names the package of a texture still Fast. --store reaches build, play, play test, edit, generate, session, import, asset and package; mirai import --refresh takes --from, a second store copied through cache.store_source, which the installed engine uses for builds/, the store of its plugins' builds that package engine fills. asset/sidecar.odin goes. Co-Authored-By: Claude Opus 5.5 --- .../skills/mirai-assets/references/formats.md | 2 +- .../skills/mirai-gameplay/references/demo.md | 10 +- .agents/skills/mirai-imports/SKILL.md | 15 +- .../mirai-imports/references/imports.md | 14 +- .../mirai-plugins/references/plugins.md | 4 +- .../skills/mirai-projects/references/build.md | 53 ++-- .../mirai-projects/references/projects.md | 19 +- asset/packages.odin | 62 +++++ asset/packages_root.odin | 57 +++-- asset/registry.odin | 2 +- asset/registry_test.odin | 4 +- asset/sidecar.odin | 226 ------------------ cook/build/build.odin | 22 +- cook/build/cook.odin | 119 +++++++-- cook/build/cook_test.odin | 2 - cook/build/imports.odin | 35 ++- cook/cache/cache.odin | 30 +++ cook/import/sound_test.odin | 2 +- cook/import/sources/cache_test.odin | 6 +- cook/import/sources/command.odin | 27 ++- cook/import/sources/cook_test.odin | 109 +++++++++ cook/import/sources/library_test.odin | 5 +- cook/import/sources/sources.odin | 11 +- cook/import/sources/support_test.odin | 28 --- cook/import/sources/write_test.odin | 2 +- editor/build.odin | 4 +- editor/player_processes.odin | 1 + editor/projects.odin | 2 +- host/assets_loading_test.odin | 1 - host/player.odin | 7 +- project/engine.odin | 5 + tools/mirai/main.odin | 2 +- tools/mirai/package.odin | 37 ++- tools/mirai/play_test.odin | 6 +- tools/mirai/project.odin | 59 ++++- tools/mirai/session.odin | 20 +- tools/mirai/session_test.odin | 2 +- 37 files changed, 595 insertions(+), 417 deletions(-) delete mode 100644 asset/sidecar.odin diff --git a/.agents/skills/mirai-assets/references/formats.md b/.agents/skills/mirai-assets/references/formats.md index 9db227c1b..a72818a83 100644 --- a/.agents/skills/mirai-assets/references/formats.md +++ b/.agents/skills/mirai-assets/references/formats.md @@ -230,7 +230,7 @@ scan opens every file and writes it again. (the machine's unless it names another); a root without the list opens no store, and what it cannot list is a problem that names the package, reported where that program reports asset problems, never a failure to start. A scan lists each like any asset (`Registry_File.listed`), - with no source, sidecar or imported folder anywhere; the registry reads no sidecar. An entry whose + with no source anywhere. An entry whose build is not on this machine, or whose blob is gone, stays listed as `Not_Built` (its id and kind, no file, not found by id), never removed and never an error of the scan; a blob that is not the asset its entry names is `Broken`. A path a file, a folder or another entry has takes diff --git a/.agents/skills/mirai-gameplay/references/demo.md b/.agents/skills/mirai-gameplay/references/demo.md index d6da99384..5e790e612 100644 --- a/.agents/skills/mirai-gameplay/references/demo.md +++ b/.agents/skills/mirai-gameplay/references/demo.md @@ -279,11 +279,11 @@ Content: others, the light meter alone: `lights` used to be clipped and `skeletal`, without light. No demo camera carries `Exposure`: the viewport and the game expose the same. - **Imports**: each image is a source beside the materials that show it (`assets/surfaces/bricks`, - `assets/sky`, `assets/gallery`), read as its sidecar says; each model is a source beside a - folder with its materials and the prefab of its nodes (`assets/gallery/plant.gltf` and - `assets/gallery/plant/`). The textures and the meshes are not versioned: they are in - `demo/.mirai/imported`, which `mirai build` makes again from the sources. - Re-importing keeps the ids, which the sidecars list + `assets/sky`, `assets/gallery`), and each model a source beside a folder with its materials and + the prefab of its nodes (`assets/gallery/plant.gltf` and `assets/gallery/plant/`), each a + package `demo/packages.mirai` pins. The textures and the meshes are not versioned: they are + blobs of the store, which `mirai build` refreshes from the sources or from a peer. + Re-importing keeps the ids, which the manifests of the packages list ([identity](../../mirai-imports/references/imports.md#identity-and-reimport)), and what the demo generates over what was imported reads them from there (`imported.odin`); what it generates on its own uses `asset.content_id`. What did not change is kept diff --git a/.agents/skills/mirai-imports/SKILL.md b/.agents/skills/mirai-imports/SKILL.md index 38aad5161..66d9cf4ab 100644 --- a/.agents/skills/mirai-imports/SKILL.md +++ b/.agents/skills/mirai-imports/SKILL.md @@ -112,12 +112,11 @@ the assets. - **An import never touches another.** Two models that embed the same image get a texture each, encoded once in the content cache; no import edits the documents or the package of another source. Decided by the owner. -- **Every output is an immutable blob of the store, the same on every machine.** Nothing under - `.mirai/imported/` and no sidecar: the build lists the blobs, documents among them as their - baseline. No blob is rewritten in place (a changed owner is a new blob), and the source an asset - names is relative to its package. Materials, prefabs, skeletons, clips, texts and prompts, which - the editor edits, are also documents of a project, followed from the baseline. Decided by the - owner. +- **Every output is an immutable blob of the store, the same on every machine.** The build lists + the blobs, documents among them as their baseline. No blob is rewritten in place (a changed + owner is a new blob), and the source an asset names is relative to its package. Materials, + prefabs, skeletons, clips, texts and prompts, which the editor edits, are also documents of a + project, followed from the baseline. Decided by the owner. - **The identity holds no engine version; the key holds the identity and the versions.** The identity (package, source, inputs, settings, plugin code: two packages of the same bytes never share a build, since its blobs hold their ids) is in the manifest; the key @@ -213,7 +212,9 @@ the assets. encoders or another builder is built again); `cook/build/cook_cache_test.odin` (a cook takes the blocks of a texture from the cache of its content) and `cook/import/sources/cook_test.odin` (the refresh of a cook builds a fast texture - final from its source; that of a build that does not cook keeps either). + final from its source; that of a build that does not cook keeps either; a peer with only + fetched builds cooks what the sources cook, sources inside a root stay behind and a missing + build fails by package; a refresh takes builds from a second store). - Parallel: import a map with `mirai import --rebuild` over a copy of an import made without the job system (or before the change) and compare the trees with `diff -r`: the same bytes. - Also test `cook/prefab` if the imported hierarchy changes, and `asset` if an asset format or its diff --git a/.agents/skills/mirai-imports/references/imports.md b/.agents/skills/mirai-imports/references/imports.md index 2aa1f4fef..e74451d17 100644 --- a/.agents/skills/mirai-imports/references/imports.md +++ b/.agents/skills/mirai-imports/references/imports.md @@ -332,10 +332,12 @@ checkout, worktree and project of the machine, keeps manifests and builds. the assets written every `importer.PROGRESS_EVERY`. - **Commands.** `mirai import ` ingests and adds to the project of `--project`, else the one whose assets hold the file, under `--folder` (else the folder of the file); `mirai import - --refresh ` refreshes a project or a plugin, `--fast` taking Fast. Every build of a - project refreshes it and its plugins once its target stands (`build.job_imports`, - `imports_refresh`) with the engine's import program, which reads the formats of plugins from - the editor library just built. + --refresh ` refreshes a project or a plugin, `--fast` taking Fast and `--from ` + naming a second store as its source (`cache.store_source`, which copies a ref with its blobs, + as a peer gives them); an installed engine takes its `builds/` store as that source. Every + build of a project refreshes it and its plugins with a `packages.mirai` once its target stands + (`build.job_imports`, `imports_refresh`) with the engine's import program, which reads the + formats of plugins from the editor library just built. ## In parallel @@ -375,8 +377,8 @@ machine that shares blobs with another finds what that one built. (`cache-`, `-`) (`asset.cache_namespace`). The store reaches it as a value (`Import_Options.store`, `Cook_Desc.store`; nil, none): the editor's imports (the session's: `--store`, else the machine's), `mirai import` (the machine's store, or - `--store`) and the cook of a build (the machine's, `store.default_root`). Tests make one in a - temporary folder. + `--store`) and the cook of a build (that of the build: `--store`, else the machine's). Tests + make one in a temporary folder. - A miss asks `Import_Options.cache_source` (a `cache.Source`) once before building: it makes the blob present in the store and sets the ref, or says it has none. - A texture that is not kept looks for its `textures.cache_key_build` (the bytes of the image, or diff --git a/.agents/skills/mirai-plugins/references/plugins.md b/.agents/skills/mirai-plugins/references/plugins.md index 38aecbf0f..88fbfff2d 100644 --- a/.agents/skills/mirai-plugins/references/plugins.md +++ b/.agents/skills/mirai-plugins/references/plugins.md @@ -216,8 +216,8 @@ demo enables. How an `editor/` adds steps, panels and tools: which gives an image to each `core.Key`, to the mouse buttons and movement and to the faces, shoulders, triggers, sticks and their press, d-pad and its directions, Start and Back of each gamepad (the colored buttons on the faces), and beside it `kenney.prompts`, imported with - `mirai import plugins/prompts_kenney/assets/kenney.prompts.json`; its atlases are of the - imported folder of the plugin ([prompts](../../mirai-imports/references/imports.md#prompts)). + `mirai import plugins/prompts_kenney/assets/kenney.prompts.json`; its atlases are blobs of the + package the plugin pins ([prompts](../../mirai-imports/references/imports.md#prompts)). - **`playtest`** (`Editor`, development): `play.observe`, the monitors of `play.qa.*` with their repros and the bot of `play.explore`, over the ticks the API runs ([`Run_Provider`](../../mirai-editor/references/extensions.md#runs-the-api-drives)). The bot diff --git a/.agents/skills/mirai-projects/references/build.md b/.agents/skills/mirai-projects/references/build.md index 8a0b5f365..168c41201 100644 --- a/.agents/skills/mirai-projects/references/build.md +++ b/.agents/skills/mirai-projects/references/build.md @@ -227,30 +227,46 @@ Settings > Input changed, and SDL3, which the executable links. ## Imported sources -The meshes, textures, sounds and fonts an import builds are not versioned: they are blobs of the -machine store, built again from the sources and the package manifests a project or plugin pins +The meshes, textures, sounds and fonts an import builds are not versioned: they are blobs of a +store that the package manifests a project or plugin pins name ([library](../../mirai-imports/references/imports.md#library)). -- **A build imports, and says so.** Every build of a project runs the refresh of the project and of - its active plugins once its target stands (`job_imports`, `build.imports_refresh`): the editor, - the game, the server, `mirai play` and the builds the editor starts. It uses the engine's import - program, since the tool builds without the importers, which reads the formats the plugins of - the project add from its editor library; the game, compiled from the engine's sources, - builds those binaries too when its project has assets. A source that does not import fails - the build naming it (`Import_Failed`). There is no other step: a checkout that was just cloned - runs after any build. Only a build that cooks (and `mirai package engine`) encodes textures - Final; the others build them Fast, as the editor imports them, and take Final where it is. -- **What ships.** A game has no imported folder and no sources: the cook writes every asset it - reaches at its path in the assets of the game, whichever of the two folders it came from. -- **The installed engine** carries the imported folder of each of its plugins, built when it is - packaged (`mirai package engine`): nothing is imported on the machine that installs it. +- **The store is a value.** The machine's (`store.default_root`), or the one `--store ` + names on `build`, `play`, `play test`, `edit`, `generate`, `session`, `import`, `asset ls|deps` + and `package`: it reaches the build (`Desc.store_root`), the import program, the cook, the + editor and the player alike. +- **A build refreshes, and says so.** Every build of a project runs the refresh of the project and + of each active plugin with a `packages.mirai` once its target stands (`job_imports`, + `build.imports_refresh`): the editor, the game, the server, `mirai play` and the builds the + editor starts. It uses the engine's import program, since the tool builds without the + importers, which reads the formats the plugins of the project add from its editor library; the + game, compiled from the engine's sources, builds those binaries too when it has packages. A + build is taken from the store, then from a second store (`--from` of `mirai import`; an + installed engine's own), then ingested from its source when the source is on this machine; a + package none of them gives fails the build by its name, saying what gives it + (`Import_Failed`). There is no other step: a checkout that was just cloned runs after any build. + A build that cooks requires Final textures; the others take Fast, as the editor imports them. +- **Without sources.** A machine whose store holds the builds (fetched from a peer, or a copy of + another store) builds and ships with no source: it imports nothing. +- **What ships.** The cook copies what the registry resolves, documents of the project and blobs of + builds alike, at its path in the assets of the game, with the index: a shipped game never opens + a store. +- **The installed engine** carries `builds/` (`project.ENGINE_STORE`), a store with the Final + builds of the packages its plugins pin, made when it is packaged: a refresh on the machine that + installs it copies them into that machine's store (`cache.store_source`, as a peer gives them), + so nothing is imported there. ## Asset cooking `cook_assets` (`cook/build/cook.odin`) leaves in a folder the assets the game can reach. The `Game` job calls it with the project assets, `cook.mirai`, the asset folders of the active plugins and the -machine store, whose texture cache it reads and fills; a failure is `Cook_Failed` with -`Result.cook.error` and `detail`. +store of the build, which lists their packages (`asset.root_packages_in`) and whose texture cache it +reads and fills; a failure is `Cook_Failed` with `Result.cook.error` and `detail`. + +- **Packages.** The registry of the cook lists the outputs of the packages of the project and of + each plugin beside their files. A package the store does not list whole is `Package_Missing`, + by its name, and a texture still Fast is `Source_Failed`, naming its package: nothing is + cooked. - **Roots.** Every `.scene` (code reaches a prefab only by instantiating it) and `.records` of the folder and what `cook.mirai` (`COOK_FILE`) names: the other assets only code refers to (a @@ -267,7 +283,8 @@ machine store, whose texture cache it reads and fills; a failure is `Cook_Failed or an asset id. A root that names nothing is `Root_Missing`; a malformed `cook.mirai` or one with another key, `Manifest_Invalid`; in both cases nothing is cooked. - **Plugins.** The folder of each plugin is mounted under `plugins/` and everything in it is - a root (its code uses it). + a root (its code uses it), except what a package of a root was imported from (its origin and + inputs inside the root), which stays behind. - **Parallel.** Files are cooked on a job system (`Cook_Desc.worker`, else one of the cook with a thread per core), the largest first; what is written and the report are those of a cook on one thread, and an error is the first by path. Prefab resolution and texture cache writes take a lock. diff --git a/.agents/skills/mirai-projects/references/projects.md b/.agents/skills/mirai-projects/references/projects.md index 2467dcad2..e8e0c1312 100644 --- a/.agents/skills/mirai-projects/references/projects.md +++ b/.agents/skills/mirai-projects/references/projects.md @@ -436,6 +436,7 @@ release of the tag; run by hand, it leaves the packages as artifacts. .mirai/engine/release/ libraries, programs, SDL3, decl/, engine.declared and engine.stamp .mirai/cache/plugins/ libraries of the engine plugins, minimal and speed .deps/ Odin, Slang and spirv-val/opt/cross; SDL3 on Linux and macOS + builds/ a store with the builds of the packages its plugins pin core/ engine/ ... Odin sources without tests, plugins/, third_party/freetype already built shaders/ templates/ assets/fonts/ editor/assets/ .agents/skills/ THIRD_PARTY_NOTICES.md ``` @@ -500,21 +501,21 @@ by its absolute path, which hands it to Git Bash ([launcher](#tool-and-machine)) | Operation | Command | | --- | --- | | Create a worktree ready to build | `mirai worktree new [--from ]` | -| Build and open the editor | `mirai edit [project] [--release] [editor arguments]` | -| Build and run the game | `mirai play [project] [--release] [player arguments]` | -| Build the editor without opening it | `mirai build editor [project] [--release]` | -| Build the shippable game | `mirai build game [project] [--out folder]` | +| Build and open the editor | `mirai edit [project] [--release] [--store folder] [editor arguments]` | +| Build and run the game | `mirai play [project] [--release] [--store folder] [player arguments]` | +| Build the editor without opening it | `mirai build editor [project] [--release] [--store folder]` | +| Build the shippable game | `mirai build game [project] [--out folder] [--store folder]` | | Build its dedicated server | `mirai build server [project] [--out folder]` | -| Build only the engine | `mirai build engine [--release] [--force]`|--release] [--force]` | -| Generate the world without building | `mirai generate [project]` | +| Build only the engine | `mirai build engine [--release] [--force]` | +| Generate the world without building | `mirai generate [project] [--store folder]` | | Run the content tool of a project | `mirai content [project] [-- arguments]` | | Ingest a source from anywhere and add it to a project | `mirai import [--project P] [--folder F] [settings]` | -| Bring the packages a project pins to builds of this engine | `mirai import --refresh [--fast]` | +| Bring the packages a project pins to builds of this engine | `mirai import --refresh [--fast] [--store folder] [--from folder]` | | Documents of interfaces and scenes, no editor | `mirai ui\|scene read\|write\|check ... [--project P]` | | Git plugins of a project | `mirai plugin add\|update\|remove\|sync\|list\|catalog\|check ...` | | Check types | `mirai check [ [-flag...]]`: a package relative to the working folder or to the engine, or absolute; none inside a project: its game, editor and plugins. In a runtime package it also fails on an allocation whose error nothing receives (`tools/mirai/alloc_lint.odin` says which calls and which packages), and in the runtime or editor part of a plugin on an import of `render` or `rhi` outside its tests or of the physics solver (`tools/mirai/solver_lint.odin`) | | Run tests | `mirai test [] [test\|file.odin...] [--list\|--failed\|--no-gpu\|--quiet] [-flag...]`: the package as for `check`; none inside a project: its game | -| Headless background editor | `mirai session start\|restart\|stop\|status [project] [--release] [--copy]` | +| Headless background editor | `mirai session start\|restart\|stop\|status [project] [--release] [--copy] [--store folder]` | | Call the editor API | `mirai rpc [key=value...\|json] [--select c] [--wait-until c=v] [--project P]` | | Capture without UI | `mirai capture [project] --out x.png [--reference r.png] [--repeat]` | | Validate demo image and cost | `mirai render check [project] [--views a,b] [--quality] [--release]` | @@ -525,7 +526,7 @@ by its absolute path, which hands it to Git Bash ([launcher](#tool-and-machine)) | Search the code | `rg [folder]` | | Check the machine and a project | `mirai doctor [project]` | | Merge and diff assets with git, once per clone | `mirai setup git [folder]` | -| Assemble the installed engine (binary) | `mirai package engine --out ` | +| Assemble the installed engine (binary) | `mirai package engine --out [--store folder]` | - Every command that builds does it in dev unless `--release` is asked for ([AGENTS.md](../../../../AGENTS.md#common-commands)). diff --git a/asset/packages.odin b/asset/packages.odin index c9abb07b2..0c539fd6f 100644 --- a/asset/packages.odin +++ b/asset/packages.odin @@ -565,6 +565,15 @@ Package_Problem :: struct { Packages_Resolved :: struct { listed:[dynamic]Package_Listed, problems:[dynamic]Package_Problem, + packages:[dynamic]Resolved_Package, // each one whose manifest read, in the order of the list +} + +// A package a root uses, by what its manifest says of it. +Resolved_Package :: struct { + id:base.Asset_Id, + name:string, + origin:Package_Origin, + inputs:[]string, // the files the import read besides the source, from its folder } // What a project lists from the packages it uses: each asset of the cache it @@ -579,6 +588,7 @@ Packages_Resolved :: struct { packages_resolve :: proc(s:^store.Store,root:string,list:^Packages,allocator:=context.allocator) -> (resolved:Packages_Resolved,error:Error) { resolved.listed=make([dynamic]Package_Listed,allocator) resolved.problems=make([dynamic]Package_Problem,allocator) + resolved.packages=make([dynamic]Resolved_Package,allocator) names:=make([dynamic]string,context.temp_allocator) for &entry in list.packages { p,manifest_error:=package_load(s,root,&entry,context.temp_allocator) @@ -590,6 +600,15 @@ packages_resolve :: proc(s:^store.Store,root:string,list:^Packages,allocator:=co case: problem_add(&resolved,{entry.id,{},.Manifest_Unreadable}) or_return } if manifest_error!=.None {continue} + named:=Resolved_Package{id=p.id,origin={folder=p.origin.folder}} + named.name=clone(p.name,allocator) or_return + named.origin.path=clone(p.origin.path,allocator) or_return + named.origin.importer=clone(p.origin.importer,allocator) or_return + inputs,no_inputs:=make([]string,len(p.inputs),allocator) + if no_inputs!=nil {return {},.Out_Of_Memory} + for input,at in p.inputs {inputs[at]=clone(input,allocator) or_return} + named.inputs=inputs + if _,no_room:=append(&resolved.packages,named);no_room!=nil {return {},.Out_Of_Memory} used,no_used:=make([]bool,len(p.assets),context.temp_allocator) by_id:=make(map[base.Asset_Id]int,context.temp_allocator) if no_used!=nil || reserve(&by_id,len(p.assets))!=nil {return {},.Out_Of_Memory} @@ -1032,3 +1051,46 @@ problem_add :: proc(resolved:^Packages_Resolved,problem:Package_Problem) -> Erro if _,no_room:=append(&resolved.problems,problem);no_room!=nil {return .Out_Of_Memory} return .None } + +// Whether what an import builds of a kind is of the cache: nothing edits it +// after the import, so its build gives it back whole. +import_cached :: proc "contextless" (kind:Fourcc) -> bool { + return kind==KIND_MESH || kind==KIND_TEXTURE || kind==KIND_SOUND || kind==KIND_FONT +} + +// A name a file can have: what no file system takes becomes _, and the +// spaces and dots of its ends go. "unnamed" when nothing is left. In the +// temp allocator. +safe_name :: proc(name:string) -> string { + cleaned:=name_clean(name) + return cleaned if cleaned!="" else "unnamed" +} + +// Where under the folder of an import the output named name goes: each part +// between '/' as safe_name makes it, without the parts that are empty, "." +// or "..". "unnamed" when no part is left. In the temp allocator. +safe_path :: proc(name:string) -> string { + b:=strings.builder_make(context.temp_allocator) + rest:=name + for part in strings.split_iterator(&rest,"/") { + cleaned:=name_clean(part) + if cleaned=="" {continue} + if strings.builder_len(b)>0 {strings.write_byte(&b,'/')} + strings.write_string(&b,cleaned) + } + return strings.to_string(b) if strings.builder_len(b)>0 else "unnamed" +} + +@(private="file") +name_clean :: proc(name:string) -> string { + b:=strings.builder_make(context.temp_allocator) + for r in name { + switch { + case r<32 || r==127,r=='<',r=='>',r==':',r=='"',r=='/',r=='\\',r=='|',r=='?',r=='*',r=='#': + strings.write_byte(&b,'_') + case: + strings.write_rune(&b,r) + } + } + return strings.trim(strings.to_string(b)," .") +} diff --git a/asset/packages_root.odin b/asset/packages_root.odin index 5d2e306f0..a8c9335eb 100644 --- a/asset/packages_root.odin +++ b/asset/packages_root.odin @@ -9,8 +9,9 @@ import "mirai:store" // keeps its packages as a project does. Every program that sets a registry up // for a root that is not shipped gives it these entries. Root_Packages :: struct { - listed:[]Package_Listed, // for registry_init or registry_mount of the root - problems:[]string, // one line per package it cannot list whole: which one, and how to fix it + listed:[]Package_Listed, // for registry_init or registry_mount of the root + packages:[]Resolved_Package, // whose manifest read: their names and origins + problems:[]string, // one line per package it cannot list whole: which one, and how to fix it } // packages_resolve of the list at root (PACKAGES_FILE) against the store at @@ -20,16 +21,8 @@ Root_Packages :: struct { // is a problem, as is a package that is not here or not built here: none is an // error, which is only that memory ran out. In allocator, an arena. root_packages :: proc(root,store_root:string,allocator:=context.allocator) -> (resolved:Root_Packages,error:Error) { - problems:=make([dynamic]string,allocator) - list,found,load_error:=packages_load(root,context.temp_allocator) - switch load_error { - case .None: - case .Out_Of_Memory: return {},.Out_Of_Memory - case .Truncated,.Not_An_Asset,.Version,.Corrupt,.Hash,.Too_Large,.Schema,.Not_Found,.IO,.Invalid,.Codec_Missing: - problem_line(&problems,"%s/%s does not read (%v), so its packages are not listed: restore it from version control",root,PACKAGES_FILE,load_error) or_return - return {problems=problems[:]},.None - } - if !found || len(list.packages)==0 {return {problems=problems[:]},.None} + list,problems,listed:=packages_list(root,allocator) or_return + if !listed {return {problems=problems[:]},.None} folder:=store_root!=""?store_root:store.default_root(context.temp_allocator) if folder=="" { problem_line(&problems,"this machine has no folder for the store of Mirai, so the packages of %s are not listed",root) or_return @@ -47,10 +40,42 @@ root_packages :: proc(root,store_root:string,allocator:=context.allocator) -> (r return {problems=problems[:]},.None } defer store.close(&s) - packages:=packages_resolve(&s,root,&list,allocator) or_return + return packages_listed(&s,root,&list,problems,allocator) +} + +// root_packages against the store s, already open; a root without the list +// lists nothing. In allocator, an arena. +root_packages_in :: proc(s:^store.Store,root:string,allocator:=context.allocator) -> (resolved:Root_Packages,error:Error) { + list,problems,listed:=packages_list(root,allocator) or_return + if !listed {return {problems=problems[:]},.None} + return packages_listed(s,root,&list,problems,allocator) +} + +// The list at root, with a problem when it does not read; listed: it names packages. +@(private="file") +packages_list :: proc(root:string,allocator:=context.allocator) -> (list:Packages,problems:[dynamic]string,listed:bool,error:Error) { + problems=make([dynamic]string,allocator) + load_error:Error + found:bool + list,found,load_error=packages_load(root,context.temp_allocator) + switch load_error { + case .None: + case .Out_Of_Memory: return {},problems,false,.Out_Of_Memory + case .Truncated,.Not_An_Asset,.Version,.Corrupt,.Hash,.Too_Large,.Schema,.Not_Found,.IO,.Invalid,.Codec_Missing: + problem_line(&problems,"%s/%s does not read (%v), so its packages are not listed: restore it from version control",root,PACKAGES_FILE,load_error) or_return + return + } + return list,problems,found && len(list.packages)>0,.None +} + +@(private="file") +packages_listed :: proc(s:^store.Store,root:string,list:^Packages,problems:[dynamic]string,allocator:=context.allocator) -> (resolved:Root_Packages,error:Error) { + problems:=problems + folder:=s.root + packages:=packages_resolve(s,root,list,allocator) or_return unreadable:=make(map[base.Asset_Id]bool,context.temp_allocator) for problem in packages.problems { - named:=package_named(&s,&list,problem.package_id) + named:=package_named(s,list,problem.package_id) switch problem.code { case .Manifest_Missing: problem_line(&problems,"%s is neither in the store at %s nor in %s: restore that file from version control, or fetch the manifest from a peer that has it",named,folder,package_file(root,problem.package_id)) or_return @@ -69,9 +94,9 @@ root_packages :: proc(root,store_root:string,allocator:=context.allocator) -> (r for &l in packages.listed { if l.state!=.Not_Built || unreadable[l.package_id] {continue} unreadable[l.package_id]=true - problem_line(&problems,"%s has no build on this machine that this engine reads, so its assets do not load: refresh it on a machine with its source, or fetch it from a peer that built it",package_named(&s,&list,l.package_id)) or_return + problem_line(&problems,"%s has no build on this machine that this engine reads, so its assets do not load: refresh it on a machine with its source, or fetch it from a peer that built it",package_named(s,list,l.package_id)) or_return } - return {listed=packages.listed[:],problems=problems[:]},.None + return {listed=packages.listed[:],packages=packages.packages[:],problems=problems[:]},.None } // "the package () of /": by its manifest when the store diff --git a/asset/registry.odin b/asset/registry.odin index a7996cbec..efc544048 100644 --- a/asset/registry.odin +++ b/asset/registry.odin @@ -129,7 +129,7 @@ registry_slashed :: proc(path:string,allocator:runtime.Allocator) -> (slashed:st } // listed: the outputs of the packages of the root (packages_resolve), which -// scans list beside its files without a source or a sidecar. False, with a +// scans list beside its files. False, with a // registry of no folder that registry_destroy takes, without memory for it. registry_init :: proc(r:^Registry,root:string,cache:="",allocator:=context.allocator,listed:[]Package_Listed=nil) -> bool { r^={allocator=allocator} diff --git a/asset/registry_test.odin b/asset/registry_test.odin index ea77e5eca..2a2e0393c 100644 --- a/asset/registry_test.odin +++ b/asset/registry_test.odin @@ -560,8 +560,8 @@ registry_mounts_a_guest_folder_that_never_takes_an_id_the_registry_has :: proc(t } // What the packages a project and a plugin pin list of the cache is listed -// from the store through root_packages of each root, with no source, sidecar -// or imported folder anywhere: a built output is its blob and loads, one whose +// from the store through root_packages of each root, with no source +// anywhere: a built output is its blob and loads, one whose // build is not here is listed Not_Built and named as a problem, a path a file of the folder has takes the id of the output after // its name, a refresh keeps every listed output and memory never holds one. @(test) diff --git a/asset/sidecar.odin b/asset/sidecar.odin deleted file mode 100644 index fde0a177a..000000000 --- a/asset/sidecar.odin +++ /dev/null @@ -1,226 +0,0 @@ -package asset - -import "base:runtime" -import "core:fmt" -import "core:strings" -import "mirai:std/base" -import "mirai:std/serial" - -// .import, the sidecar of a source of a project or a plugin, beside -// it in its assets folder: the settings its import applies (read by -// cook/import with the schema of its Import_Settings), the folder of the -// source's folder its outputs go to, the files the import reads besides the -// source and one output per asset the source gives, with its id. What nobody -// edits (import_cached) is no file of the assets but of the imported folder -// that goes with them, -, never versioned. The registry -// reads neither: it lists the outputs of packages (packages.odin). - -SIDECAR_EXTENSION :: ".import" -SIDECAR_VERSION :: 3 - -// One asset an import gives, by what it is in the source. -Import_Output :: struct { - kind:Fourcc, - name:string, // in the source: the mesh, the material, the image - index:int, // among those of its kind in the source - id:base.Asset_Id, - part:bool, // of the cache: it is a part of the root of the import, decided when it was first imported -} - -Sidecar :: struct { - settings:string, // the text of the settings object, "" without one - folder:string, // under the folder of the source, where its outputs go; "": beside it - inputs:[dynamic]string, // the files the import read besides the source, from the folder of the source - outputs:[dynamic]Import_Output, -} - -// Whether what an import builds of a kind is of the cache: nothing edits it -// after the import, so the source and its sidecar give it back whole. -import_cached :: proc "contextless" (kind:Fourcc) -> bool { - return kind==KIND_MESH || kind==KIND_TEXTURE || kind==KIND_SOUND || kind==KIND_FONT -} - -// The file of an imported asset: in folder, by its id and what it was built from. -imported_file :: proc(folder:string,id:base.Asset_Id,key:u64,extension:string,allocator:=context.allocator) -> string { - buffer:[base.ASSET_ID_TEXT]u8 - digits:[16]u8 - file,error:=strings.concatenate({folder,"/",base.asset_id_format(id,buffer[:]),"-",fmt.bprintf(digits[:],"%016x",key),extension},allocator) - if error!=nil {return ""} - return file -} - -// The id and the key a file name of an imported folder says. -imported_name :: proc(name:string) -> (id:base.Asset_Id,key:u64,ok:bool) { - if len(name)<49 || name[32]!='-' {return} - id=serial.text_parse_asset_id(name[:32]) or_return - key=serial.text_parse_hex(name[33:49]) or_return - return id,key,true -} - -// The text of a sidecar whose settings are already the text of their object. -sidecar_text :: proc(settings:string,sidecar:Sidecar,allocator:=context.allocator) -> string { - w:serial.Writer - serial.writer_init(&w,1024,allocator) - serial.text_write_raw(&w,"{\n \"version\": ") - serial.text_write_int(&w,SIDECAR_VERSION) - serial.text_write_raw(&w,",\n \"settings\": ") - serial.text_write_raw(&w,settings) - if sidecar.folder!="" { - serial.text_write_raw(&w,",\n \"folder\": ") - serial.text_write_string(&w,sidecar.folder) - } - if len(sidecar.inputs)>0 { - serial.text_write_raw(&w,",\n \"inputs\": [") - for input,at in sidecar.inputs { - if at>0 {serial.text_write_raw(&w,", ")} - serial.text_write_string(&w,input) - } - serial.text_write_raw(&w,"]") - } - serial.text_write_raw(&w,",\n \"outputs\": [") - for output,at in sidecar.outputs { - buffer:[4]u8 - serial.text_write_raw(&w,at>0?",\n {\"kind\": ":"\n {\"kind\": ") - serial.text_write_string(&w,strings.trim_right(fourcc_text(output.kind,&buffer)," ")) - serial.text_write_raw(&w,", \"name\": ") - serial.text_write_string(&w,output.name) - serial.text_write_raw(&w,", \"index\": ") - serial.text_write_int(&w,i64(output.index)) - serial.text_write_raw(&w,", \"id\": \"") - serial.text_write_asset_id(&w,output.id) - serial.text_write_raw(&w,output.part?"\", \"part\": true}":"\"}") - } - serial.text_write_raw(&w,len(sidecar.outputs)>0?"\n ]\n}\n":"]\n}\n") - return string(serial.writer_take(&w)) -} - -// False for a text that is not a sidecar of this version: a field or a value -// it does not know, a folder that leaves the folder of the source, an output -// without an id or kind, twice the same id or the same output. What it holds -// lives in allocator, an arena: nothing frees it piece by piece; settings is -// a view of text. -sidecar_parse :: proc(text:string,allocator:=context.allocator) -> (sidecar:Sidecar,ok:bool) { - r:serial.Text_Reader - serial.text_reader_init(&r,text,context.temp_allocator) - defer serial.text_reader_destroy(&r) - sidecar.inputs=make([dynamic]string,allocator) - sidecar.outputs=make([dynamic]Import_Output,allocator) - version:=u64(0) - serial.text_expect(&r,'{') or_return - for first:=true;!serial.text_accept(&r,'}');first=false { - if !first {serial.text_expect(&r,',') or_return} - // The key is a view the next string replaces. - field:=strings.clone(serial.text_string(&r) or_return,context.temp_allocator) - serial.text_expect(&r,':') or_return - switch field { - case "version": - negative,magnitude:=serial.text_integer(&r) or_return - if negative {return} - version=magnitude - case "settings": - serial.text_space(&r) - start:=r.offset - if serial.text_peek(&r)!='{' {return} - serial.text_skip(&r) or_return - sidecar.settings=text[start:r.offset] - case "folder": - no_room:runtime.Allocator_Error - if sidecar.folder,no_room=strings.clone(serial.text_string(&r) or_return,allocator);no_room!=nil {return} - case "inputs": - serial.text_expect(&r,'[') or_return - for begun:=false;!serial.text_accept(&r,']');begun=true { - if begun {serial.text_expect(&r,',') or_return} - input,no_room:=strings.clone(serial.text_string(&r) or_return,allocator) - if no_room!=nil {return} - append(&sidecar.inputs,input) - } - case "outputs": - serial.text_expect(&r,'[') or_return - for begun:=false;!serial.text_accept(&r,']');begun=true { - if begun {serial.text_expect(&r,',') or_return} - append(&sidecar.outputs,output_read(&r,allocator) or_return) - } - case: - return - } - } - serial.text_end(&r) or_return - if version!=SIDECAR_VERSION || !folder_valid(sidecar.folder) {return} - for output,at in sidecar.outputs { - if output.id=={} || output.kind==0 || output.part && !import_cached(output.kind) {return} - for other in sidecar.outputs[:at] { - if other.id==output.id || other.kind==output.kind && other.index==output.index && other.name==output.name {return} - } - } - return sidecar,true -} - -@(private="file") -output_read :: proc(r:^serial.Text_Reader,allocator:=context.allocator) -> (output:Import_Output,ok:bool) { - serial.text_expect(r,'{') or_return - for first:=true;!serial.text_accept(r,'}');first=false { - if !first {serial.text_expect(r,',') or_return} - field:=strings.clone(serial.text_string(r) or_return,context.temp_allocator) - serial.text_expect(r,':') or_return - switch field { - case "kind": - text:=serial.text_string(r) or_return - if len(text)==0 || len(text)>4 {return} - bytes:=[4]u8{' ',' ',' ',' '} - copy(bytes[:],text) - output.kind=Fourcc(u32(bytes[0])|u32(bytes[1])<<8|u32(bytes[2])<<16|u32(bytes[3])<<24) - case "name": - no_room:runtime.Allocator_Error - if output.name,no_room=strings.clone(serial.text_string(r) or_return,allocator);no_room!=nil {return} - case "index": - negative,magnitude:=serial.text_integer(r) or_return - if negative || magnitude>u64(max(i32)) {return} - output.index=int(magnitude) - case "id": - output.id=serial.text_parse_asset_id(serial.text_string(r) or_return) or_return - case "part": - output.part=serial.text_keyword(r,"true") - if !output.part && !serial.text_keyword(r,"false") {return} - case: - return - } - } - return output,true -} - -// A name a file can have: what no file system takes becomes _, and the -// spaces and dots of its ends go. "unnamed" when nothing is left. In the -// temp allocator. -safe_name :: proc(name:string) -> string { - cleaned:=name_clean(name) - return cleaned if cleaned!="" else "unnamed" -} - -// Where under the folder of an import the output named name goes: each part -// between '/' as safe_name makes it, without the parts that are empty, "." -// or "..". "unnamed" when no part is left. In the temp allocator. -safe_path :: proc(name:string) -> string { - b:=strings.builder_make(context.temp_allocator) - rest:=name - for part in strings.split_iterator(&rest,"/") { - cleaned:=name_clean(part) - if cleaned=="" {continue} - if strings.builder_len(b)>0 {strings.write_byte(&b,'/')} - strings.write_string(&b,cleaned) - } - return strings.to_string(b) if strings.builder_len(b)>0 else "unnamed" -} - -@(private="file") -name_clean :: proc(name:string) -> string { - b:=strings.builder_make(context.temp_allocator) - for r in name { - switch { - case r<32 || r==127,r=='<',r=='>',r==':',r=='"',r=='/',r=='\\',r=='|',r=='?',r=='*',r=='#': - strings.write_byte(&b,'_') - case: - strings.write_rune(&b,r) - } - } - return strings.trim(strings.to_string(b)," .") -} diff --git a/cook/build/build.odin b/cook/build/build.odin index 201872522..87c34f214 100644 --- a/cook/build/build.odin +++ b/cook/build/build.odin @@ -78,7 +78,7 @@ Desc :: struct { action:Action, // Check and Test: of the Editor target test_package:string, // Test: the folder of the package, absolute test:Test_Options, // Test: how its executable runs - store_root:string, // the machine store the packages of the project list from, for the paths of assets its code writes; "": the machine's + store_root:string, // the store the packages of the project and its plugins are refreshed in and listed from; "": the machine's (store.default_root) allocator:runtime.Allocator, // of the job. Unset: context.allocator } @@ -115,6 +115,7 @@ Job :: struct { engine:Engine_Mode, imports:[]string, // Build: the roots whose sources it imports once the target stands (imports_roots) import_program:string, // Build: that of the engine of the project's defines, which imports them + store_root:string, // Desc.store_root, or the machine's compiler:string, flags:[]string, // what identifies the build besides its sources staged:string, // what the compiler writes @@ -126,7 +127,7 @@ Job :: struct { copies:[]Copy, // Game: what goes next to the executable settings:string, // Game: its settings file: those of the engine and of the project merged, since it ships without the engine cook:Cook_Desc, // Game: the assets it ships; without assets, a project that has none - store:store.Store, // Game: the machine store, whose caches cook.store names; unopened: none + store:store.Store, // Game: the store of store_root, which cook.store names; unopened: none plugin_assets:[dynamic]project.Asset_Folder, // Game: the folders of assets of its active plugins, in cook.plugins development:[dynamic]string, // Game: the folders of the active development plugins, which it must not reach plugins:[]Plugin_Library, // the plugin libraries the target links (plugins.odin) @@ -222,13 +223,14 @@ job_start :: proc(desc:Desc) -> ^Job { } } job.engine=desc.engine + job.store_root=strings.clone(desc.store_root!=""?desc.store_root:store.default_root(context.temp_allocator),a) if desc.action==.Build { roots:=imports_roots(p) job.imports=make([]string,len(roots),a) for root,i in roots {job.imports[i]=strings.clone(root,a)} job.import_program=project_engine(p,desc.engine,a).import_program } - generated,written:=generated_write(p,desc.target,desc.store_root) + generated,written:=generated_write(p,desc.target,job.store_root) if !written { if len(generated.errors)>0 { job.result.text=strings.join(generated.errors,"\n",a) @@ -265,8 +267,8 @@ job_start :: proc(desc:Desc) -> ^Job { if assets:=project.folder_path(p.root,.Assets,a);os.is_dir(assets) { job.cook={assets=assets,output=project.folder_path(folder,.Assets,a),manifest=project.path(p.root,COOK_FILE,allocator=a),collision_cache=project.content_cache_path(p.root,project.COLLISION_CACHE,allocator=a),root=p.root,server=desc.server,quick=desc.dev,keys=project.path(job.work,fmt.tprintf("%s.cook",name),allocator=a),allocator=a} job.cook.codecs=asset.project_codecs(p.root,p.engine_root) - if store_root:=store.default_root(context.temp_allocator);store_root!="" { - if opened,error:=store.open(store_root,a);error==.None { + if job.store_root!="" { + if opened,error:=store.open(job.store_root,a);error==.None { job.store=opened job.cook.store=&job.store } @@ -887,14 +889,14 @@ job_run :: proc(desc:Desc) -> ^Job { return job } -// What the imports of the project keep outside version control, from its -// sources, once the target stands: the library of its editor, built, has the -// formats its plugins add. False, with the job failed, when a source did not -// import. +// The packages of the project and its plugins brought to builds this engine +// reads, once the target stands: the library of its editor, built, has the +// formats its plugins add. A build that cooks requires Final textures. False, +// with the job failed, when a package has no build and could not get one. @(private) job_imports :: proc(job:^Job) -> bool { if len(job.imports)==0 {return true} - ok,message:=imports_refresh(job.import_program,job.imports,fast=job.cook.assets=="") + ok,message:=imports_refresh(job.import_program,job.imports,job.cook.assets=="",job.store_root) if !ok { job.result.text=strings.clone(message,virtual.arena_allocator(&job.arena)) job_fail(job,.Import_Failed) diff --git a/cook/build/cook.odin b/cook/build/cook.odin index 57e6b9307..abe5e8fe3 100644 --- a/cook/build/cook.odin +++ b/cook/build/cook.odin @@ -33,10 +33,14 @@ import "mirai:store" // interfaces, rows and code show) of the project and what cook.mirai // names, the other assets only code refers to. From them, everything an // asset depends on, and so on. The rest stays behind and is listed in the -// report. A texture that stores RGBA8 is written in the blocks of its use, -// unless only interfaces show it, and the blocks are kept in the texture -// cache for the next cook; a texture the editor imported Fast is encoded -// again as Final from its source, which the cook imports again; a scene or prefab with a stale prefab +// report. The assets of the project and of each plugin are their files and +// the outputs of the packages each root pins, listed from the store of the +// build (asset.root_packages_in), which its refresh brought to Final builds: +// a package not listed whole fails the cook by its name, and a texture still +// Fast is not shipped. What a package was imported from, inside a root, is a +// source and stays behind. A texture that stores RGBA8 is written in the +// blocks of its use, unless only interfaces show it, and the blocks are kept +// in the texture cache for the next cook; a scene or prefab with a stale prefab // instance is written with its instances brought up to date, so a shipped // game loads them as they are; every other file is copied byte for byte. // With far, every scene a section references gets its far mesh beside it @@ -46,7 +50,8 @@ import "mirai:store" // (asset.mesh_collision_pieces, kept in the collision cache), which a // shipped game reads and never makes. The // folder gets a packed index, so the game does not look at its files to -// know what they are, and everything written is read back and checked. +// know what they are, and never opens a store; everything written is read +// back and checked. COOK_FILE :: "cook.mirai" @@ -59,12 +64,12 @@ Cook_Desc :: struct { assets:string, // the assets folder of the project output:string, // the assets folder of the shipped game: it ends up holding what was cooked and nothing else manifest:string, // the cook.mirai of the project. "", or a file that does not exist: the scenes are the only roots - plugins:[]project.Asset_Folder, // the assets of active plugins: every file a root, cooked under its prefix - store:^store.Store, // whose texture cache keeps compressed textures by content (textures.cache_read); nil: none + plugins:[]project.Asset_Folder, // the assets of active plugins: every file but the sources of its packages a root, cooked under its prefix + store:^store.Store, // that lists the packages of the roots and whose texture cache keeps compressed textures by content (textures.cache_read); nil: none, for roots without packages cache_source:cache.Source, // asked on a miss of that cache before compressing collision_cache:string, // folder of the convex pieces of meshes kept by content (asset.mesh_collision_pieces); "": none codecs:asset.Codecs, // what the project ships with (build.compression) - root:string, // of the project: the sources of imported assets are named relative to it. "": the folder of assets + root:string, // of the project, where its packages.mirai is. "": a project without packages worker:jobs.Worker_Ref, // where the files are cooked, in parallel on its system. No system: a system of the cook, a thread per core server:bool, // for a dedicated server: what is only shown or heard (asset.Kind_Info.presentation) stays behind, and is not missing quick:bool, // a cook that checks a change (a dev game): the profile Balanced in place of Ultra @@ -86,7 +91,8 @@ Cook_Error :: enum u8 { Write_Failed, // detail: the file Verify_Failed, // detail: the file that does not read back as what was written Resolve_Failed, // detail: the scene or prefab whose prefab instances could not be brought up to date - Source_Failed, // detail: the texture that is still Fast, which the import of its source by a build makes Final + Source_Failed, // detail: the texture that is still Fast and the package it is of, which a refresh that requires Final brings + Package_Missing, // detail: one line per package the store does not list whole, with how to fix it Duplicate_Id, // detail: the two files with one id and how to tell them apart } @@ -171,6 +177,57 @@ manifest_read :: proc(path:string,allocator:runtime.Allocator) -> (manifest:Cook return } +// A package a root pins, with its outputs as the registry lists them. +@(private="file") +Cook_Package :: struct { + name:string, + source:string, // absolute, when it is inside the root: what it was imported from + folder:bool, // the source is a folder + inputs:[]string, // absolute, with the source: the other files the import read + listed:[]asset.Package_Listed, +} + +// The outputs of the packages root pins, from the store of the cook, with +// each package added to packages; a package not listed whole adds a line to +// missing. In the temp allocator. +@(private="file") +cook_listed :: proc(desc:Cook_Desc,root:string,packages:^[dynamic]Cook_Package,missing:^[dynamic]string) -> []asset.Package_Listed { + if root=="" {return nil} + if desc.store==nil { + if list,found,_:=asset.packages_load(root,context.temp_allocator);found && len(list.packages)>0 { + append(missing,fmt.tprintf("the packages of %s are listed from a store, and the cook has none",root)) + } + return nil + } + resolved,error:=asset.root_packages_in(desc.store,root,context.temp_allocator) + if error!=.None { + append(missing,fmt.tprintf("the packages of %s could not be listed (%v)",root,error)) + return nil + } + append(missing,..resolved.problems) + for &found in resolved.packages { + entry:=Cook_Package{name=found.name,folder=found.origin.folder} + if found.origin.path!="" && !filepath.is_abs(found.origin.path) && !strings.has_prefix(found.origin.path,"/") { + entry.source=fmt.tprintf("%s/%s",root,found.origin.path) + beside:=entry.source if found.origin.folder else entry.source[:strings.last_index_byte(entry.source,'/')] + inputs:=make([]string,len(found.inputs),context.temp_allocator) + for input,at in found.inputs {inputs[at]=fmt.tprintf("%s/%s",beside,input)} + entry.inputs=inputs + } + first:=-1 + count:=0 + for &l,at in resolved.listed { + if l.package_id!=found.id {continue} + if first<0 {first=at} + count+=1 + } + // packages_resolve lists the outputs of each package together. + if first>=0 {entry.listed=resolved.listed[first:first+count]} + append(packages,entry) + } + return resolved.listed +} + @(private="file") cook_fail :: proc(report:^Cook_Report,error:Cook_Error,detail:string,allocator:runtime.Allocator) { report.error,report.detail=error,strings.clone(detail,allocator) @@ -194,11 +251,33 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { if problem!="" {cook_fail(&report,.Manifest_Invalid,problem,allocator);return} registry:asset.Registry - asset.registry_init(®istry,desc.assets,"",temp) - for folder in desc.plugins {asset.registry_mount(®istry,folder.path,folder.prefix)} + packages:=make([dynamic]Cook_Package,temp) + missing_packages:=make([dynamic]string,temp) + asset.registry_init(®istry,desc.assets,"",temp,cook_listed(desc,desc.root,&packages,&missing_packages)) + for folder in desc.plugins {asset.registry_mount(®istry,folder.path,folder.prefix,cook_listed(desc,folder.root,&packages,&missing_packages))} defer asset.registry_destroy(®istry) + if len(missing_packages)>0 {cook_fail(&report,.Package_Missing,strings.join(missing_packages[:],"\n",temp),allocator);return} if asset.registry_scan(®istry)!=.None {cook_fail(&report,.Assets_Missing,desc.assets,allocator);return} files:=registry.files[:] + // What a package was imported from, inside a root, is not for the game. + source:=make([]bool,len(files),temp) + owners:=make(map[base.Asset_Id]string,temp) + for &found in packages { + for &l in found.listed {owners[l.id]=found.name} + if found.source=="" {continue} + path:=asset.registry_relative(®istry,found.source) or_continue + for input in found.inputs { + if read,inside:=asset.registry_relative(®istry,input);inside { + if at,listed:=registry.by_path[read];listed {source[at]=true} + } + } + if !found.folder { + if at,listed:=registry.by_path[path];listed {source[at]=true} + continue + } + under:=strings.concatenate({path,"/"},temp) + for file,at in files {source[at]||=strings.has_prefix(file.path,under)} + } // Which of two files with one id is the asset depends on the machine: the cook takes neither. if len(registry.duplicates)>0 { twins:=registry.duplicates[0] @@ -222,11 +301,9 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { } for file,at in files { if file.state==.Asset && (file.kind==asset.KIND_SCENE || file.kind==asset.KIND_RECORDS || file.kind==asset.KIND_STRINGS) {root_add(reached,&pending,&report,u32(at))} - // What a plugin brings is for its code, which no scene names; its sources - // (a file with a sidecar beside it) and their sidecars stay behind. - source:=strings.has_suffix(file.path,asset.SIDECAR_EXTENSION) || strings.concatenate({file.path,asset.SIDECAR_EXTENSION},temp) in registry.by_path + // What a plugin brings is for its code, which no scene names; its sources stay behind. for folder in desc.plugins { - if !source && strings.has_prefix(file.path,folder.prefix) && len(file.path)>len(folder.prefix) && file.path[len(folder.prefix)]=='/' {root_add(reached,&pending,&report,u32(at))} + if !source[at] && strings.has_prefix(file.path,folder.prefix) && len(file.path)>len(folder.prefix) && file.path[len(folder.prefix)]=='/' {root_add(reached,&pending,&report,u32(at))} } } for root in manifest.roots { @@ -285,7 +362,7 @@ cook_run :: proc(desc:Cook_Desc) -> (report:Cook_Report) { } defer if on.system==&own {jobs.system_destroy(&own)} work:=Cook_Work{ - desc=desc,registry=®istry,files=files,textures=manifest.textures,drawn=drawn,shown=shown,pieces=pieces_wanted(®istry,reached,left,temp), + desc=desc,registry=®istry,files=files,owners=owners,textures=manifest.textures,drawn=drawn,shown=shown,pieces=pieces_wanted(®istry,reached,left,temp), on=on,allocator=on.system.allocator,first_error=len(files), results=make([]Cook_Result,len(files),temp), types=make([]asset.Types,jobs.worker_count(on.system),temp), @@ -367,6 +444,7 @@ Cook_Work :: struct { desc:Cook_Desc, registry:^asset.Registry, files:[]asset.Registry_File, + owners:map[base.Asset_Id]string, // the name of the package each listed output is of textures:Cook_Textures, drawn,shown:[]bool, pieces:[]bool, // a mesh written with the convex pieces a moving body collides as @@ -430,7 +508,7 @@ work_types :: proc(work:^Cook_Work,worker:int) -> ^asset.Types { } // A texture still built Fast is not shipped: the imports of the editor build one so until its -// Final is there, and the sync before the cook builds every one Final. +// Final is there, and the refresh before the cook requires Final. @(private="file") texture_probe :: proc(work:^Cook_Work,first,last:int,worker:int) { context.allocator=work.allocator @@ -446,8 +524,11 @@ texture_probe :: proc(work:^Cook_Work,first,last:int,worker:int) { defer asset.asset_close(&file) view,error:=asset.texture_view(&file,types) if error!=.None || .Fast not_in view.info.flags {continue} - named:=view.source!=nil?base.text_get(view.source.source[:]):"a source it does not name" - result.problem=work_text(work,"it was imported Fast and %s was not imported again as Final: `mirai build game` does it before it cooks",named) + if name,listed:=work.owners[work.files[at].id];listed { + result.problem=work_text(work,"the package %s has only a build with Fast textures here, and a game ships Final: `mirai build game` refreshes it requiring Final, which a machine with its source must ingest with this engine, or a peer that has that build must give",name) + } else { + result.problem=work_text(work,"it was written Fast, and a game ships Final: write it again Final") + } } } diff --git a/cook/build/cook_test.odin b/cook/build/cook_test.odin index ce7e4e8be..7d948d0b2 100644 --- a/cook/build/cook_test.odin +++ b/cook/build/cook_test.odin @@ -268,8 +268,6 @@ a_cook_ships_every_asset_of_an_active_plugin_under_its_prefix :: proc(t:^testing BRUSH :: base.Asset_Id{7,1} material_write(t,&types,fmt.tprintf("%s/brushes/soft.mat",plugin),BRUSH,WOOD_COLOR,{}) testing.expect(t,os.write_entire_file(fmt.tprintf("%s/readme.txt",plugin),"for the code of the plugin")==nil) - // A source and its sidecar stay behind: what its import made ships, not what it was made from. - testing.expect(t,os.write_entire_file(fmt.tprintf("%s/brushes/soft.png",plugin),"a source")==nil && os.write_entire_file(fmt.tprintf("%s/brushes/soft.png.import",plugin),"{}")==nil) report:=cook_assets({assets=assets,output=output,plugins={{path=plugin,prefix="plugins/probe"}},allocator=context.temp_allocator}) testing.expect_value(t,report.error,Cook_Error.None) diff --git a/cook/build/imports.odin b/cook/build/imports.odin index 81c10cbe6..7470350ec 100644 --- a/cook/build/imports.odin +++ b/cook/build/imports.odin @@ -3,24 +3,31 @@ package build import "core:fmt" import "core:os" import "core:strings" +import "mirai:asset" import "mirai:project" import "mirai:std/subprocess" // Brings the packages each root (a project, a plugin) pins in its -// packages.mirai to builds this engine reads (cook/import/sources.project_refresh), with the -// import program of the engine (Engine_Paths.import_program, of the defines -// the engine was built with), which reads the formats the plugins of the -// project add from the library of its editor: the tool builds without the -// importers. Textures are built Final for a build that cooks them and Fast, -// as the editor imports them, for one that does not (fast). What the program -// says of each source it imports goes to the output of this process as it -// says it: a large source imports for minutes. message, in the temp -// allocator: why it failed. -imports_refresh :: proc(program:string,roots:[]string,fast:bool) -> (ok:bool,message:string) { +// packages.mirai to builds this engine reads (cook/import/sources.project_refresh) +// in the store at store_root ("": the machine's), with the import program of +// the engine (Engine_Paths.import_program, of the defines the engine was +// built with), which reads the formats the plugins of the project add from +// the library of its editor: the tool builds without the importers. A build +// is taken from the store, then from the store at from ("": none; an +// installed engine adds the store of the builds of its plugins), then built +// from its source when the source is on this machine; a package none of them +// gives fails by name. Textures are required Final for a build that cooks +// them and taken Fast, as the editor imports them, for one that does not +// (fast). What the program says of each source it imports goes to the output +// of this process as it says it: a large source imports for minutes. +// message, in the temp allocator: why it failed. +imports_refresh :: proc(program:string,roots:[]string,fast:bool,store_root:string,from:="") -> (ok:bool,message:string) { if len(roots)==0 {return true,""} command:=make([dynamic]string,context.temp_allocator) append(&command,program) if fast {append(&command,"--fast")} + if store_root!="" {append(&command,"--store",store_root)} + if from!="" {append(&command,"--from",from)} for root in roots {append(&command,"--refresh",root)} state,_,errors,error:=subprocess.exec({command=command[:]},context.temp_allocator,echo=os.to_writer(os.stdout)) if error!=nil {return false,fmt.tprintf("the import program could not run: %v",error)} @@ -28,10 +35,12 @@ imports_refresh :: proc(program:string,roots:[]string,fast:bool) -> (ok:bool,mes return true,"" } -// The roots of a project whose sources its builds import: the project and -// its active plugins, those with an assets folder. In the temp allocator. +// The roots of a project whose packages its builds refresh: the project and +// its active plugins, those with a packages.mirai. In the temp allocator. imports_roots :: proc(p:^project.Project) -> []string { roots:=make([dynamic]string,context.temp_allocator) - for folder in project.asset_folders(p,context.temp_allocator) {append(&roots,folder.root)} + for folder in project.asset_folders(p,context.temp_allocator) { + if os.is_file(project.path(folder.root,asset.PACKAGES_FILE,allocator=context.temp_allocator)) {append(&roots,folder.root)} + } return roots[:] } diff --git a/cook/cache/cache.odin b/cook/cache/cache.odin index 132d961d1..dd4c0d802 100644 --- a/cook/cache/cache.odin +++ b/cook/cache/cache.odin @@ -104,3 +104,33 @@ load_here :: proc(s:^store.Store,namespace,ref:string,key,version:u64,file:^asse if store.verify(s,hash)==.Corrupt {_=store.remove(s,hash)} return false } + +// A source that copies from the store at from, a second root of blobs and +// refs: the blob of the ref and, for a build (asset.REF_IMPORTS), every blob +// it lists. What a peer gives, and what an installed engine gives from the +// store of the builds of its plugins. +store_source :: proc(from:^store.Store) -> Source { + fetch :: proc(user:rawptr,into:^store.Store,namespace,name:string) -> bool { + from:=(^store.Store)(user) + hash,error:=store.ref_get(from,namespace,name) + if error!=.None || !blob_copy(from,into,hash) {return false} + if namespace==asset.REF_IMPORTS { + b,build_error:=asset.build_get(from,hash,context.temp_allocator) + if build_error!=.None {return false} + for o in b.outputs { + if !blob_copy(from,into,o.blob) {return false} + } + } + return store.ref_set(into,namespace,name,hash)==.None + } + return {fetch=fetch,user=from} +} + +@(private="file") +blob_copy :: proc(from,into:^store.Store,hash:store.Hash) -> bool { + if store.has(into,hash) {return true} + path,path_error:=store.blob_path(from,hash,context.temp_allocator) + if path_error!=.None {return false} + put,put_error:=store.put_file(into,path) + return put_error==.None && put==hash +} diff --git a/cook/import/sound_test.odin b/cook/import/sound_test.odin index 6ba0fc330..f13cea832 100644 --- a/cook/import/sound_test.odin +++ b/cook/import/sound_test.odin @@ -235,7 +235,7 @@ a_wav_and_an_ogg_become_sounds_decoded_or_streamed_by_their_length :: proc(t:^te if bytes!=nil {_=os.write_entire_file(path,bytes)} report,error:=import_sound(path,options={store=s}) testing.expect_value(t,error.code,code) - testing.expect(t,len(report.assets)==0 && !os.exists(fmt.tprintf("%s.import",path)),name) + testing.expect(t,len(report.assets)==0,name) } refuse(t,s,folder,"noise.wav",{1,2,3,4,5,6,7,8,9,10,11,12,13,14},.Malformed) refuse(t,s,folder,"surround.ogg",THREE_OGG,.Unsupported) diff --git a/cook/import/sources/cache_test.odin b/cook/import/sources/cache_test.odin index b0a58d86e..da0755fd9 100644 --- a/cook/import/sources/cache_test.odin +++ b/cook/import/sources/cache_test.odin @@ -43,7 +43,7 @@ sources_are_known_by_their_content :: proc(t:^testing.T) { testing.expect_value(t,missing.code,importer.Error_Code.Not_Found) // An image is known by its bytes, and nothing is written beside it. picture,picture_error:=import_test(t,fmt.tprintf("%s/picture.dat",folder)) - testing.expect(t,picture_error.code==.None && len(picture.assets)==1 && picture.assets[0].kind==asset.KIND_TEXTURE && !os.exists(fmt.tprintf("%s/picture.dat.import",folder))) + testing.expect(t,picture_error.code==.None && len(picture.assets)==1 && picture.assets[0].kind==asset.KIND_TEXTURE) importer.report_destroy(&picture) _,collection:=import_test(t,fmt.tprintf("%s/family.ttc",folder)) testing.expect(t,collection.code==.Unsupported && strings.contains(collection.message,"collection"),collection.message) @@ -207,7 +207,7 @@ an_import_takes_a_texture_that_another_import_built_from_the_texture_cache :: pr // A store that misses asks its source before building: what a peer built is taken as it is. peer:=options peer.store=test_store(t,fmt.tprintf("%s/peer",root)) - peer.cache_source=copy_source(kept) + peer.cache_source=cache.store_source(kept) format,mips,_,_=imported(t,&types,folder,"p",bytes,peer) testing.expect(t,format==.RGBA8 && mips==1,"the levels of the source, not built") testing.expect_value(t,len(cached_refs(peer.store,textures.CACHE)),1) @@ -285,7 +285,7 @@ an_import_takes_a_mesh_that_another_import_built_from_the_mesh_cache :: proc(t:^ // A store that misses asks its source before building: what a peer built is taken as it is. peer:=options peer.store=test_store(t,fmt.tprintf("%s/peer",root)) - peer.cache_source=copy_source(kept) + peer.cache_source=cache.store_source(kept) peer_path:string if peer_path,ok=imported(t,source,peer);!ok {return} testing.expect_value(t,colliders(t,&types,peer_path),1) diff --git a/cook/import/sources/command.odin b/cook/import/sources/command.odin index b1c2aa365..5301116b2 100644 --- a/cook/import/sources/command.odin +++ b/cook/import/sources/command.odin @@ -3,11 +3,13 @@ package sources import "base:runtime" import "core:fmt" import "core:io" +import "core:os" import "core:path/filepath" import "core:strconv" import "core:strings" import "core:time" import "mirai:asset" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/textures" import "mirai:project" @@ -38,7 +40,11 @@ import "mirai:store" // source on this machine; a package nothing here gives is a failure that names // it. --fast builds textures Fast, as the editor imports them, unless the // texture cache keeps them Final, and takes a Fast build: what a build that -// does not cook takes. The store is that of the machine, or the one at --store. +// does not cook takes. --from names a second store a build this +// store lacks is copied from before it is built (cache.store_source), as a +// peer gives it; an installed engine takes the store of the builds of its +// plugins (project.ENGINE_STORE) unless another is given. The store is that +// of the machine, or the one at --store. // The report goes to output, one asset per line: id, kind, bytes and path, and // under it, indented, what the asset holds; then the package and the // warnings. How far a long import is goes to errors while it runs, and a @@ -51,7 +57,7 @@ import "mirai:store" // Exit codes: 0 done, 1 failed, 2 misuse. @(private="file") -IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--project ] [--folder ] [--rebuild] [--fast] [--store ] [--profile] [--release] | mirai import --refresh [--refresh ...] [--fast] [--store ] [--release]" +IMPORT_USAGE :: "Usage: mirai import [--uncompressed|--blocks] [--rdo-lambda L] [--srgb|--normal|--normal-green-down|--data] [--alpha-coverage C] [--lods auto|none] [--parts auto|all|none] [--front auto|negative_z|z|x|negative_x] [--rig source|humanoid|fox|horse|bird|dragon|kaiju|shark|snake|spider] [--rig-crotch M] [--rig-neck M] [--skeleton ] [--project ] [--folder ] [--rebuild] [--fast] [--store ] [--profile] [--release] | mirai import --refresh [--refresh ...] [--fast] [--store ] [--from ] [--release]" // library reads the formats and the importers of whole sources the plugins of // a project add and the types of its game (in the import program of the @@ -65,6 +71,7 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in foldered:bool profiled:bool store_root:string // of the store whose caches keep what imports build; "": that of the machine + from_root:string // of --from // What the options change of the settings the source has. Change :: enum {Textures,Rdo_Lambda,Image,Alpha_Coverage,Lods,Parts,Front,Rig,Rig_Crotch,Rig_Neck,Skeleton} given:importer.Import_Settings @@ -109,6 +116,7 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in changed+={.Alpha_Coverage} } case "--store": store_root,ok=value(args,&at) + case "--from": from_root,ok=value(args,&at) case "--profile": profiled=true case "--rebuild": options.rebuild=true case "--fast": options.fast_textures=true @@ -189,8 +197,23 @@ command :: proc(args:[]string,output,errors:io.Writer,library:Library=nil) -> in if options.store==nil {fmt.wprintln(errors,"mirai import: this machine has no folder of user data for its store: pass one with --store");return 1} if len(roots)>0 { if file!="" || root!="" || foldered || changed!={} {fmt.wprintln(errors,IMPORT_USAGE);return 2} + if from_root=="" { + if engine:=project.engine_root_find("",context.temp_allocator);project.engine_installed(engine) { + if bundled:=project.path(engine,project.ENGINE_STORE,allocator=context.temp_allocator);os.is_dir(bundled) {from_root=bundled} + } + } + from:store.Store + if from_root!="" { + opened:store.Error + if from,opened=store.open(from_root,context.temp_allocator);opened!=.None { + fmt.wprintfln(errors,"mirai import: the store at %s could not be opened (%v): pass another with --from",from_root,opened) + return 1 + } + library_options.source=cache.store_source(&from) + } return refresh_command(roots[:],library_options,output,errors) } + if from_root!="" {fmt.wprintln(errors,IMPORT_USAGE);return 2} if file=="" {fmt.wprintln(errors,IMPORT_USAGE);return 2} origin:=importer.path_key(file) // The project it is added to, and where under its assets. diff --git a/cook/import/sources/cook_test.odin b/cook/import/sources/cook_test.odin index 633febe19..ef694c64a 100644 --- a/cook/import/sources/cook_test.odin +++ b/cook/import/sources/cook_test.odin @@ -6,8 +6,11 @@ import "core:slice" import "core:strings" import "core:testing" import "mirai:asset" +import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/build" +import "mirai:cook/prefab" +import "mirai:project" import "mirai:std/base" import "mirai:store" @@ -74,3 +77,109 @@ a_cook_ships_a_fast_texture_final_from_its_source :: proc(t:^testing.T) { shipped_id,shipped_flags,shipped_blocks:=level_of(t,&types,fmt.tprintf("%s/game/noise.tex",folder)) testing.expect(t,shipped_id==fast_id && .Fast not_in shipped_flags && slice.equal(shipped_blocks,final_blocks),"Final, with the id it had") } + +// A game cooks what its project and its plugins pin from the store, and what a +// package was imported from inside a root stays behind. A peer with the +// project and none of its sources fails the cook by the name of each package +// it has no build of; once it fetched the builds alone, it ships the same +// files, which load as the runtime loads them, from the index of the cook. +@(test) +a_peer_with_only_fetched_builds_cooks_the_game_its_sources_cook :: proc(t:^testing.T) { + top:=base.test_folder(t,"cook-library") + if top=="" {return} + defer free_all(context.temp_allocator) + context.user_ptr=prefab.import_world(t) + game:=fmt.tprintf("%s/game",top) + plugin:=fmt.tprintf("%s/plugins/brushes",game) + os.make_directory_all(fmt.tprintf("%s/assets/props",game)) + os.make_directory_all(fmt.tprintf("%s/assets",plugin)) + box:=box_source(fmt.tprintf("%s/assets/props",game),"box.gltf","Wood","Paint") + brush:=fmt.tprintf("%s/assets/soft.png",plugin) + testing.expect(t,os.write_entire_file(brush,png_bytes(8,8,{200,40,40}))==nil) + here:=test_store(t,fmt.tprintf("%s/here",top)) + if here==nil {return} + names:=make([dynamic]string,context.temp_allocator) + for pinned in ([2][3]string{{game,box,"props"},{plugin,brush,""}}) { + options:=Library_Options{store=here,root=pinned[0],plugins=test_library} + report,error:=ingest(pinned[1],options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s: %s",pinned[1],error.message) {return} + p,_:=asset.package_get(here,report.manifest,context.temp_allocator) + append(&names,p.name) + added:=project_add(pinned[0],report.manifest,pinned[2],nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + } + cook :: proc(root:string,s:^store.Store,output:string) -> build.Cook_Report { + plugins:=[1]project.Asset_Folder{{path=fmt.tprintf("%s/plugins/brushes/assets",root),prefix="plugins/brushes",root=fmt.tprintf("%s/plugins/brushes",root)}} + return build.cook_assets({assets=fmt.tprintf("%s/assets",root),output=output,root=root,store=s,plugins=plugins[:],allocator=context.temp_allocator}) + } + shipped:=cook(game,here,fmt.tprintf("%s/shipped",top)) + if !testing.expectf(t,shipped.error==.None,"%v %s",shipped.error,shipped.detail) {return} + paths:=make(map[string]bool,context.temp_allocator) + for file in shipped.files {paths[file.path]=true} + testing.expectf(t,paths["plugins/brushes/soft.tex"] && paths["props/Box.mesh"] && paths["props/wood.tex"],"the outputs of the packages ship: %v",paths) + testing.expectf(t,!paths["plugins/brushes/soft.png"] && !paths["props/box.gltf"] && !paths["props/wood.png"],"their sources stay behind: %v",paths) + + // A peer: the project and its plugin without their sources, and a store of its own. + peer:=fmt.tprintf("%s/peer",top) + folder_copy(game,peer) + for source in ([3]string{"assets/props/box.gltf","assets/props/wood.png","plugins/brushes/assets/soft.png"}) {testing.expect(t,os.remove(fmt.tprintf("%s/%s",peer,source))==nil)} + there:=test_store(t,fmt.tprintf("%s/there",top)) + if there==nil {return} + lacking:=cook(peer,there,fmt.tprintf("%s/lacking",top)) + testing.expect_value(t,lacking.error,build.Cook_Error.Package_Missing) + for name in names {testing.expectf(t,strings.contains(lacking.detail,name),"the package %s is named: %s",name,lacking.detail)} + for root in ([2]string{peer,fmt.tprintf("%s/plugins/brushes",peer)}) { + fetched:=project_refresh(root,{store=there,root=root,source=cache.store_source(here)},context.temp_allocator) + testing.expectf(t,fetched.fetched==1 && fetched.ingested==0 && len(fetched.failures)==0 && len(fetched.missing)==0,"%v",fetched) + } + output:=fmt.tprintf("%s/again",top) + again:=cook(peer,there,output) + if !testing.expectf(t,again.error==.None,"%v %s",again.error,again.detail) {return} + testing.expect_value(t,len(again.files),len(shipped.files)) + indexed:asset.Registry + asset.registry_init(&indexed,output);defer asset.registry_destroy(&indexed) + testing.expect_value(t,asset.registry_load(&indexed,fmt.tprintf("%s/%s",output,asset.REGISTRY_INDEX)),asset.Error.None) + for file,at in again.files { + if !testing.expect_value(t,file.path,shipped.files[at].path) {continue} + first,_:=os.read_entire_file(fmt.tprintf("%s/shipped/%s",top,file.path),context.temp_allocator) + second,_:=os.read_entire_file(fmt.tprintf("%s/%s",output,file.path),context.temp_allocator) + testing.expectf(t,len(first)>0 && slice.equal(first,second),"%s ships as the machine with its source ships it",file.path) + if file.id=={} {continue} + listed,found:=asset.registry_find(&indexed,file.id) + testing.expectf(t,found && listed.path==file.path,"the index has %s",file.path) + a:asset.Asset + if testing.expectf(t,asset.asset_load(&a,fmt.tprintf("%s/%s",output,file.path),context.temp_allocator)==.None,"%s loads",file.path) {asset.asset_close(&a)} + } +} + +// A refresh takes the builds its store lacks from a second store, as the +// installed engine does from the store of the builds of its plugins: with +// the source gone, mirai import --refresh --from copies them and imports nothing. +@(test) +a_refresh_takes_the_builds_of_a_second_store_without_their_sources :: proc(t:^testing.T) { + top:=base.test_folder(t,"refresh-from") + if top=="" {return} + defer free_all(context.temp_allocator) + plugin:=fmt.tprintf("%s/plugin",top) + os.make_directory_all(fmt.tprintf("%s/assets",plugin)) + brush:=fmt.tprintf("%s/assets/soft.png",plugin) + testing.expect(t,os.write_entire_file(brush,png_bytes(8,8,{40,200,40}))==nil) + bundled_folder,machine_folder:=fmt.tprintf("%s/engine/builds",top),fmt.tprintf("%s/machine",top) + bundled:=test_store(t,bundled_folder) + if bundled==nil {return} + options:=Library_Options{store=bundled,root=plugin} + report,error:=ingest(brush,options,allocator=context.temp_allocator) + if !testing.expectf(t,error.code==.None,"%s",error.message) {return} + added:=project_add(plugin,report.manifest,"",nil,options,context.temp_allocator) + testing.expectf(t,len(added.failures)==0 && len(added.missing)==0,"%v",added) + testing.expect(t,os.remove(brush)==nil) + + out,errors:=strings.builder_make(context.temp_allocator),strings.builder_make(context.temp_allocator) + code:=command({"--refresh",plugin,"--store",machine_folder,"--from",bundled_folder},strings.to_writer(&out),strings.to_writer(&errors)) + testing.expectf(t,code==0 && strings.contains(strings.to_string(out),"1 builds fetched"),"%d\n%s%s",code,strings.to_string(out),strings.to_string(errors)) + machine:=test_store(t,machine_folder) + if machine==nil {return} + resolved,resolve_error:=asset.root_packages_in(machine,plugin,context.temp_allocator) + testing.expectf(t,resolve_error==.None && len(resolved.problems)==0 && len(resolved.listed)==1 && resolved.listed[0].state==.Built,"%v",resolved) + testing.expect(t,!os.exists(brush),"nothing is written where the source was") +} diff --git a/cook/import/sources/library_test.odin b/cook/import/sources/library_test.odin index 170aefccc..f71bf5d69 100644 --- a/cook/import/sources/library_test.odin +++ b/cook/import/sources/library_test.odin @@ -8,13 +8,13 @@ import "core:sync" import "core:testing" import "core:thread" import "mirai:asset" +import "mirai:cook/cache" import "mirai:cook/prefab" import importer "mirai:cook/import" import "mirai:std/base" import "mirai:store" // The files under folder, by their paths under it, in the temp allocator. -@(private="file") files_of :: proc(folder:string) -> map[string]bool { files:=make(map[string]bool,context.temp_allocator) todo:=make([dynamic]string,context.temp_allocator) @@ -30,7 +30,6 @@ files_of :: proc(folder:string) -> map[string]bool { } // Copies the files under from to the same paths under to. -@(private="file") folder_copy :: proc(from,to:string) { for file in files_of(from) { target:=fmt.tprintf("%s%s",to,file) @@ -132,7 +131,7 @@ a_peer_without_sources_lists_and_loads_and_nothing_is_removed_or_built :: proc(t // With a store that has them, the builds are fetched: listed where the project lists them, loaded. fetching:=alone - fetching.source=copy_source(here) + fetching.source=cache.store_source(here) fetched:=project_refresh(peer,fetching,context.temp_allocator) testing.expectf(t,fetched.fetched==2 && fetched.ingested==0,"%v",fetched) if !report_clean(t,fetched) {return} diff --git a/cook/import/sources/sources.odin b/cook/import/sources/sources.odin index ce5ab234f..b98f38b58 100644 --- a/cook/import/sources/sources.odin +++ b/cook/import/sources/sources.odin @@ -157,12 +157,11 @@ whole_find :: proc(wholes:[]Whole_Source,path:string) -> (whole:Whole_Source,cla // (options.root; "": importer.source_root): one of wholes claims is imported whole by // it, before the formats; a model writes its assets and, when it // has nodes, the prefab of its hierarchy, which the report lists last; an -// image or a sound writes one asset. The sidecar of the source -// (.import, Import_Report.saved) keeps the settings it was imported -// with, the folder of its assets and the id of each of them: importing the -// source again gives the same assets. What the import builds that nobody -// edits goes to the imported folder of the root (importer.Sidecar), whatever -// the format that reads it. +// image or a sound writes one asset. The package of the source in +// options.store keeps the settings it was imported with and the id of each +// asset: importing the source again gives the same assets. What the import +// builds that nobody edits is a blob of its build, whatever the format that +// reads it, and its documents go to options.documents. import_source :: proc(path:string,options:importer.Import_Options={},formats:[]importer.Format=nil,wholes:[]Whole_Source=nil,allocator:=context.allocator) -> (report:importer.Import_Report,error:importer.Error) { formats:=formats if formats!=nil else FORMATS[:] options:=options diff --git a/cook/import/sources/support_test.odin b/cook/import/sources/support_test.odin index edb9690db..e4f409c34 100644 --- a/cook/import/sources/support_test.odin +++ b/cook/import/sources/support_test.odin @@ -10,7 +10,6 @@ import "core:strings" import "core:testing" import "mirai:asset" import "mirai:core" -import "mirai:cook/cache" import importer "mirai:cook/import" import "mirai:cook/import/gltf" import "mirai:cook/import/obj" @@ -165,33 +164,6 @@ cached_key :: proc(ref:store.Ref) -> u64 { return key } -// A source that copies what the store at user keeps, as a peer would give -// it: the blob of the ref and, for a build, every blob it lists. -copy_source :: proc(from:^store.Store) -> cache.Source { - fetch :: proc(user:rawptr,into:^store.Store,namespace,name:string) -> bool { - from:=(^store.Store)(user) - hash,error:=store.ref_get(from,namespace,name) - if error!=.None || !blob_copy(from,into,hash) {return false} - if namespace==asset.REF_IMPORTS { - b,build_error:=asset.build_get(from,hash,context.temp_allocator) - if build_error!=.None {return false} - for o in b.outputs { - if !blob_copy(from,into,o.blob) {return false} - } - } - return store.ref_set(into,namespace,name,hash)==.None - } - return {fetch=fetch,user=from} -} - -// Copies the blob hash of from into into. -blob_copy :: proc(from,into:^store.Store,hash:store.Hash) -> bool { - path,path_error:=store.blob_path(from,hash,context.temp_allocator) - if path_error!=.None {return false} - put,put_error:=store.put_file(into,path) - return put_error==.None && put==hash -} - // A folder of a test as the root of a project, with its assets folder, where // the sources of its imports are. "" when there is none. test_project :: proc(t:^testing.T,name:string) -> (root,assets:string) { diff --git a/cook/import/sources/write_test.odin b/cook/import/sources/write_test.odin index 72d834b92..787ae90d9 100644 --- a/cook/import/sources/write_test.odin +++ b/cook/import/sources/write_test.odin @@ -496,7 +496,7 @@ importing_again_keeps_the_identity_of_every_asset :: proc(t:^testing.T) { p,_:=asset.package_get(s,first.manifest,context.temp_allocator) cached:=0 for a in p.assets {cached+=int(asset.import_cached(a.kind))} - testing.expectf(t,len(documents)+cached==len(first.assets) && len(p.assets)==len(first.assets)+int(first.prefab!={}) && !os.exists(fmt.tprintf("%s/one/assets/plant.gltf.import",folder)),"%d documents, %d of the cache, %d listed, %d written",len(documents),cached,len(p.assets),len(first.assets)) + testing.expectf(t,len(documents)+cached==len(first.assets) && len(p.assets)==len(first.assets)+int(first.prefab!={}),"%d documents, %d of the cache, %d listed, %d written",len(documents),cached,len(p.assets),len(first.assets)) // A copy of a project, its sources and documents, imported with the manifest the project pins, keeps the ids of its assets. shared:=test_store(t,fmt.tprintf("%s/shared",folder)) diff --git a/editor/build.odin b/editor/build.odin index 028a41563..04643e125 100644 --- a/editor/build.odin +++ b/editor/build.odin @@ -90,6 +90,7 @@ Build_Run :: struct { thread:^thread.Thread, project_root,engine_root,folder,test_package:string, runs:string, // the folder of the binaries of the engine this editor runs + store_root:string, // of the store of the session names:[]string, action:build.Action, target:build.Target, @@ -118,6 +119,7 @@ build_run_start :: proc(s:^Session,action:build.Action,target:build.Target,folde r:=new(Build_Run,a) r.project_root,r.engine_root,r.runs=strings.clone(s.project_root,a),strings.clone(s.engine_root,a),strings.clone(engine_folder_here(s),a) r.folder,r.test_package=strings.clone(folder,a),strings.clone(test_package,a) + r.store_root=strings.clone(s.store.root,a) r.names=make([]string,len(names),a) for name,i in names {r.names[i]=strings.clone(name,a)} r.action,r.target,r.seconds,r.started=action,target,seconds,time.tick_now() @@ -161,7 +163,7 @@ build_run :: proc(r:^Build_Run) { if r.shaders.result.status==.Failed {return} } } - r.job=build.job_start({project=&p,target=r.target,action=r.action,output=r.folder,test_package=r.test_package,test={names=r.names,quiet=true},dev=r.target==.Editor && build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=r.runs}) + r.job=build.job_start({project=&p,target=r.target,action=r.action,output=r.folder,test_package=r.test_package,test={names=r.names,quiet=true},dev=r.target==.Editor && build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=r.runs,store_root=r.store_root}) for { ended:=build.job_poll(r.job) if r.shaders!=nil {ended=build.shaders_poll(r.shaders) && ended} diff --git a/editor/player_processes.odin b/editor/player_processes.odin index eee44b7c7..82c908a53 100644 --- a/editor/player_processes.odin +++ b/editor/player_processes.odin @@ -109,6 +109,7 @@ player_process_command :: proc(s:^Session,player:string,args:[]string,k:int,disp tiles:=p.first+p.count command:=make([dynamic]string,context.temp_allocator) append(&command,player,"--attached") + if s.store.root!="" {append(&command,"--store",s.store.root)} append(&command,..args) if p.embedded { pixels:=player_pixels(s,player_tile(player_region(s),tiles,p.first+k)) diff --git a/editor/projects.odin b/editor/projects.odin index 20ad83df9..cd5ed02cf 100644 --- a/editor/projects.odin +++ b/editor/projects.odin @@ -350,7 +350,7 @@ projects_build_start :: proc(s:^Session) { built^=opened p.built=built p.shaders=build.shaders_start(built) - p.job=build.job_start({project=built,target=.Editor,dev=build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=engine_folder_here(s)}) + p.job=build.job_start({project=built,target=.Editor,dev=build.ENGINE_MODE_HERE==.Dev,engine=build.ENGINE_MODE_HERE,runs=engine_folder_here(s),store_root=s.store.root}) log.infof("building %s to open it",absolute) } diff --git a/host/assets_loading_test.odin b/host/assets_loading_test.odin index 2b54e1b91..2cb848350 100644 --- a/host/assets_loading_test.odin +++ b/host/assets_loading_test.odin @@ -429,5 +429,4 @@ the_assets_of_a_project_load_its_packages_from_the_store :: proc(t:^testing.T) { testing.expect_value(t,asset.registry_file_path(&a.registry,"props/crate.mesh",context.temp_allocator),found.listed[0].file) pending,listed:=asset.registry_find_path(&a.registry,"plugins/pack/props/atlas.tex") testing.expect(t,listed && pending.state==.Not_Built) - testing.expect(t,!os.exists(fmt.tprintf("%s/.mirai/imported",root))) } diff --git a/host/player.odin b/host/player.odin index d57337a5c..3eebffe68 100644 --- a/host/player.odin +++ b/host/player.odin @@ -42,6 +42,7 @@ Player_Options :: struct { scene:string, capture:string, input:string, + store:string, // the store the packages of the project list from; "": the machine's frames:int, players:int, // local players; 0: one window:[4]i32, // where the window goes, in screen coordinates; zero: centered @@ -75,10 +76,11 @@ Player_Script :: struct { PLAYER_MEMORY_TEXT :: "mirai: out of memory to start the game" @(private) -PLAYER_USAGE_TEXT :: `Usage: [--project ] [--scene ] [--headless] [--no-gpu] [--dedicated] [--frames ] [--capture ] +PLAYER_USAGE_TEXT :: `Usage: [--project ] [--scene ] [--store ] [--headless] [--no-gpu] [--dedicated] [--frames ] [--capture ] [--validation] [--input