diff --git a/.gitignore b/.gitignore index 1580dea..f62e0bd 100644 --- a/.gitignore +++ b/.gitignore @@ -18,6 +18,7 @@ release_dist_*/ references/ docs/ +/dotnet/ examples/ .claude/ fivefury-*/ diff --git a/CHANGELOG.md b/CHANGELOG.md index fb6f7fc..d6ddf5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,15 +7,26 @@ The changelog is release-oriented and uses a small fixed set of categories: ## [Unreleased] +## [0.5.1] - 2026-09-12 + ### Added +- Batch construction through `Vector2.from_rows`, `Vector3.from_rows`, `Vector4.from_rows`, and `Quaternion.from_rows`, preserving nominal types and immutable, independent values. +- Configurable zlib compression levels in `ResourceSections.to_bytes()` and `build_rsc7()`. - Immutable YED evaluators snapshot expression selection and skeleton defaults for repeated typed evaluation with explicit per-actor variable state. - Compiled YCD clip samplers preserve the public phase/time evaluation interface while sharing immutable animation plans. - Immutable compiled YCD animation samplers retain owned channel data for repeated native evaluation without stale caches after source edits. - Per-track quaternion encoding overrides allow explicit facial rotations while keeping body and camera packing unchanged. +### Changed + +- Version 0.5.1 is the final planned feature release of the current Python implementation; new core development moves to .NET. +- The README presents the library for research and education, with concise installation instructions, runnable examples, and a format overview. +- YMAP, YTYP, and YDR writers use zlib level 6 to balance serialization speed and output size; compressed bytes and sizes may differ, while the general resource builder retains its level-9 default. + ### Fixed +- META root blocks stay separate from child blocks of the same structure type, preventing child data from being overwritten during graph construction. - YED program caches detect in-place operand and skeleton-default edits instead of relying on object identities and list lengths. - Explicit quaternion authoring stores all four static components instead of discarding W and reconstructing it from rounded XYZ values. - YCD precision diagnostics distinguish integer-frame and subframe component errors, report their frame locations and include quaternion angular error independently of component limits. @@ -24,8 +35,12 @@ The changelog is release-oriented and uses a small fixed set of categories: ### Performance -- Up to 2x faster repeated YED evaluation with compiled evaluators, avoiding repeated expression resolution and Python input/output conversions. -- Up to 14x faster repeated YCD sampling with compiled animation samplers, keeping typed outputs and cached-quaternion interpolation order. +- YMAP reads materialize nominal Python records through schema-checked native bindings, and writing uses the shared native META graph encoder. A measured 2,000-entity case improved by a further 5.6x for reading and 1.8x for writing over the preceding native field-codec implementation; results vary by workload. Unknown layouts and custom extension registrations retain generic conversion. +- Up to 2.9x faster YTYP reading and 9x faster writing with native hash-field assignment and fewer Python conversions. +- Faster YDR reading with lower memory usage by constructing typed vectors directly in C++ without intermediate tuple lists. +- YDR, YDD, and YFT writing reuses float64 position buffers and calculates bounds once per save; vectorized transforms make rigid YDR models write up to 3.4x faster. +- Up to 2x faster repeated YED evaluation with compiled evaluators that avoid repeated expression resolution. +- Up to 14x faster repeated YCD sampling by reusing immutable native animation plans. ## [0.5.0] - 2026-09-07 diff --git a/MANIFEST.in b/MANIFEST.in index 6184eac..a2a79f5 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -2,3 +2,4 @@ include LICENSE recursive-include native *.cpp *.h recursive-include fivefury/ydr *.xml recursive-include tests *.py *.md +prune dotnet diff --git a/README.md b/README.md index 8cf312b..a607a73 100644 --- a/README.md +++ b/README.md @@ -1,253 +1,68 @@ # FiveFury -FiveFury is a Python toolkit for authoring, inspecting, validating, converting, and packaging GTA V assets. It exposes typed models and declarative builders over the game's binary resource formats, metadata containers, and RPF archives without hiding the lower-level data needed for advanced workflows. +A Python library for **research and education** around GTA V asset formats. -FiveFury is designed for tools that need to do more than convert a single file: - -- Read, modify, and rebuild assets while preserving data that is not yet modeled. -- Create maps, collisions, navigation data, drawables, fragments, animations, and archives from Python objects. -- Target GTA V Legacy or Enhanced where their binary layouts differ. -- Validate resource pointers, packed limits, ownership, and format-specific invariants before writing. -- Index a game installation and resolve assets, hashes, textures, parent dictionaries, and map dependencies. -- Run expensive hashing, resource-layout, geometry, collision, and archive operations through the bundled native extension. +FiveFury provides typed models and binary readers and writers for studying resource structures, inspecting their relationships, and experimenting with data. It supports GTA V Legacy and Enhanced, with coverage depending on the format and edition. ## Installation ```bash -pip install fivefury +python -m pip install fivefury ``` -FiveFury requires Python 3.11 or newer. - -Static mesh import uses Trimesh and accepts file paths, bytes, `trimesh.Trimesh`, and `trimesh.Scene` objects. Supported source formats depend on Trimesh; call `supported_mesh_formats()` to inspect them at runtime. - -## Supported workflows - -| Area | Formats | Current coverage | -| --- | --- | --- | -| Drawables | `YDR`, `YDD`, `YTD` | Models and LODs, shader groups and parameters, samplers, embedded textures, lights, bounds, skeletons, skinning, drawable dictionaries, and texture dictionaries | -| Fragments and physics | `YFT` | Fragment drawables, damaged states, physics LODs/groups/children, contextual collision bounds, ownership validation, mass and inertia calculation, breakable glass, and environment cloth | -| World placement | `YMAP`, `YTYP` | Entities, physics dictionaries, MLO instances and definitions, archetypes, rooms, portals, entity sets, car generators, occluders, timecycle modifiers, and LOD lights | -| Streaming metadata | `YMF`, `GTXD`, `gta5_cache_y.dat` | Map/type dependencies, MLO registration, texture-dictionary parent chains, runtime PSO validation, and cache generation from in-memory or loose assets | -| Collision | `YBN` | Primitive, composite, geometry, and BVH bounds; materials, octants, MLO room IDs, and collision generation from triangle meshes | -| Navigation | `YND`, `YNV` | Road nodes and links, area partitioning, junction heightmaps, navmesh sectors/polygons/portals, typed network validation, in-memory cell builders, and Trimesh conversion | -| World data | `heightmap.dat`, `water.xml` | Quantized height grids, row RLE, water masks and queries, water surfaces, wave quads, and calming regions | -| Animation | `YCD`, `YED` | Skeletal, object, UV, camera, root-motion, and bone-scale tracks; clip dictionaries; expression dictionaries and spring data | -| Cutscenes | `.cut` | Binary cutscene read/write, declarative scene and audio authoring, validation, and YCD section generation | -| Audio and text | `AWC`, `REL`, `GXT2` | Audio containers and common codecs, typed audio metadata graphs, synth/curve/category records, and hashed text tables | -| Packaging | `RPF7`, DLC metadata | PC archive creation, nested archives, folder/ZIP conversion, standalone resource extraction, encrypted archive reading, generated DLC metadata, and typed CUT audio registration | -| Generic metadata | `YMT`, META, PSO, RBF | Typed known roots, generic binary containers, PSCH enums, and preservation of unknown schemas or payloads during supported rewrites | -| Console assets | `CDR`, PS3 `RPF7` | Read-only PS3 drawable decoding and automatic PS3 archive detection/extraction | +Requires **Python 3.11+**. Prebuilt wheels are available for Windows x64. Building the native extension from source requires a C++20 compiler. -Additional discovery support is available for embedded `YPT` texture dictionaries. `GameFileCache` can index `YWR` and `YVR`, but FiveFury does not yet expose dedicated parsers for them. `YFD`, `YPDB`, and `MRF` are not implemented. +## Examples -### Legacy and Enhanced +These examples create their own sample data. -Target-aware APIs use `GameTarget` instead of loose version labels: +### Create a small map ```python -from fivefury import GameTarget, YdrGen9Shader, trimesh_to_ydr - -trimesh_to_ydr( - "source/prop.glb", - "stream/prop.ydr", - game=GameTarget.GTA5_ENHANCED, - shader=YdrGen9Shader.DEFAULT, -) -``` - -The target changes runtime headers, resource versions, bounds, shader metadata, and vertex layouts only where the format requires it. Readers infer the edition from the asset when possible. Legacy remains the default. +from fivefury import Vector3, Ymap -Current target-aware authoring covers the main `YDR`, `YDD`, `YFT`, `YBN`, `YCD`, `YED`, `YND`, and `YNV` paths. Support is format-specific rather than a blanket claim that every GTA V file differs between editions. - -## Quick start - -### Build and package a map - -```python -from fivefury import Vector3, Ymap, create_rpf - -ymap = Ymap(name="example_map") +ymap = Ymap(name="example") ymap.entity( "prop_tree_pine_01", - position=Vector3(100.0, 200.0, 0.0), - lod_dist=150.0, -) -ymap.physics_dictionary("example_map") -ymap.car_gen("sultan", Vector3(110.0, 205.0, 0.0), heading=90.0) -ymap.save("example_map.ymap", auto_extents=True) - -archive = create_rpf("example_pack.rpf") -archive.file("stream/example_map.ymap", ymap) -archive.save("example_pack.rpf") -``` - -Factories such as `entity(...)` and `car_gen(...)` append the new object to the owning `Ymap`. Prebuilt objects are inserted directly through the corresponding typed collection, for example `ymap.entities.append(entity)`. - -### Assemble an Enhanced vehicle DLC - -```python -from fivefury import VehiclePackBuilder - -pack = VehiclePackBuilder( - "example_cars", - vehicles_meta, - handling_meta, - variations_meta, - carcols_meta, -) -pack.vehicle("example_car", vehicle_yft, textures=vehicle_ytd) -result = pack.save("build") -``` - -The builder cross-validates the four metadata documents, registers their runtime file types, packages streamed YFT/YTD assets in a nested RPF, rereads the complete DLC, and only then atomically writes `build/example_cars/dlc.rpf`. - -### Build a drawable from memory - -```python -from fivefury import YdrMeshInput, YdrShader, create_ydr - -ydr = create_ydr( - name="example_drawable", - shader=YdrShader.DEFAULT, - meshes=[ - YdrMeshInput( - positions=[ - (0.0, 0.0, 0.0), - (1.0, 0.0, 0.0), - (0.0, 1.0, 0.0), - ], - indices=[0, 1, 2], - texcoords=[[(0.0, 0.0), (1.0, 0.0), (0.0, 1.0)]], - ) - ], - material_textures={"DiffuseSampler": "example_diffuse"}, -) -ydr.save("example_drawable.ydr") -``` - -`read_ydr(...)` returns an editable asset with material, texture, bound, light, skeleton, and skinning helpers. Render geometry can also be converted into an embedded collision bound with `ydr.ensure_bound_from_render_geometry()`. - -`YdrSkeleton` owns its `bones` list: construction and assignment copy the input -collection, but share its bone objects. Edits through the original input list do -not change the skeleton. Use `skeleton.bones.append(...)`, slice assignment, -`reverse()`, `sort()`, or other ordinary list operations to edit membership and -order. Direct bone `name`, `tag`, and `parent_index` assignments invalidate every -owning collection's lookup indexes; duplicate names (case-insensitive) and tags -resolve to the first bone in current list order. Appends update valid indexes -incrementally. Other relevant mutations trigger one lazy rebuild, not a scan on -every lookup. These lookup updates do not renumber bones or normalize bindings; -`skeleton.build()` still finalizes hierarchy indices and sibling links. - -For low-level preparation, use `fivefury.ydr.prepare.select_layout` and -`fivefury.ydr.prepare.normalize_materials` instead of the private builder aliases -`_select_layout` and `_normalize_materials`. Replace `fivefury.ydr.prepare.resolve_shader` -with `fivefury.ydr.shaders.resolve_shader_reference`. - -### Skin vertices on the GPU - -```python -from fivefury import GpuShaderLanguage, GpuSkinning - -gpu = GpuSkinning( - positions, - blend_indices, - blend_weights, - normals=normals, + position=Vector3(100, 200, 30), ) - -# Upload these immutable streams when the mesh is created. -positions = gpu.streams.positions -influences = gpu.streams.influences -normals = gpu.streams.normals - -# Reuse the palette and upload only its contents each frame. -palette = gpu.palette(bone_count) -gpu.pack_palette(skinning_matrices, output=palette) - -shader = gpu.compute_shader(GpuShaderLanguage.GLSL) -groups = gpu.dispatch_groups() +ymap.save("example.ymap", auto_extents=True) ``` -`GpuSkinning` packs four bone indices and normalized weights into eight bytes per vertex and exposes specialized GLSL or HLSL vertex and compute sources. Compute output remains GPU-resident; renderers should bind the returned streams and palette using `GpuSkinningBindings` and avoid reading deformed vertices back to Python. - -### Generate collision +### Read, inspect and edit it ```python -from fivefury import ( - BoundMaterial, - BoundMaterialType, - build_bound_from_triangles, - save_ybn, -) +from fivefury import Vector3, Ymap -triangles = [ - ((0.0, 0.0, 0.0), (4.0, 0.0, 0.0), (0.0, 4.0, 0.0)), - ((4.0, 0.0, 0.0), (4.0, 4.0, 0.0), (0.0, 4.0, 0.0)), -] +ymap = Ymap.from_path("example.ymap") +entity = ymap.entities[0] +print(entity.archetype_name, entity.position) -bound = build_bound_from_triangles( - triangles, - material=BoundMaterial(type=BoundMaterialType.CONCRETE), -) -save_ybn(bound, "floor_collision.ybn") +entity.position = Vector3(110, 200, 30) +ymap.save("edited.ymap", auto_extents=True) ``` -The builder chunks oversized geometry, builds BVHs and octants, and validates packed limits before serialization. The same bounds model is shared by standalone `YBN`, embedded `YDR` collision, MLO collision, and `YFT` physics. - -### Build the runtime map cache from loose assets - -```python -from fivefury import build_gta5_cache_y_from_directory - -cache = build_gta5_cache_y_from_directory("build/stream") -cache.save("build/gta5_cache_y.dat") -``` - -The builder reads loose `YMAP`, `YTYP`, and `YBN` files, derives map and interior records, validates the result, and writes the binary cache atomically. - -### Index a game installation - -```python -from fivefury import GameFileCache - -cache = GameFileCache("path/to/Grand Theft Auto V") -cache.scan_game(use_index_cache=True) - -asset = cache.get_asset("prop_tree_pine_01", kind=".ydr") -cache.extract_asset(asset, "out/prop_tree_pine_01.ydr") -cache.extract_asset_textures(asset, "out/textures") -``` - -`GameFileCache` scans loose files and nested archives, performs lazy typed loading, resolves names and hashes, follows `YTD`/`GTXD` texture relationships, and supplies the dependency context used by map-manifest tooling. - -## API conventions - -FiveFury keeps the authoring layer close to the data model: - -- Typed collections use ordinary `append(...)` and `extend(...)` operations; there is no generic dispatcher that guesses the destination from a runtime type. -- Singular noun factories such as `entity(...)`, `bone(...)`, `light(...)`, and `car_gen(...)` construct, register, and return one domain object. -- One-to-one relationships use assignment, such as `ydr.bound = collision`; verbs are reserved for real operations such as `ensure_*`, `derive_*`, `normalize_*`, `bind_*`, and `resolve_*`. -- `AssetRef`, `AssetSet`, and `BuildContext` provide one typed path for resolving dependencies between assets and selecting Legacy or Enhanced behavior. -- `ValidationReport` carries stable diagnostic codes, severity, asset, and field paths instead of relying on unstructured output. -- `build()` derives normalized state, `validate()` inspects it, and `save()` validates and performs atomic binary serialization. -- Stable game-side values use enums for targets, shaders, LODs, flags, render masks, materials, and track formats. -- Core writers use atomic replacement and reject known invalid references, ownership, pointers, or packed ranges before replacing the destination. - -Local development documentation is kept in `docs/`, which is excluded from version control. +For experiments in memory, use `ymap.to_bytes()` and `Ymap.from_bytes(data)`. The `fivefury.meta` module exposes META schemas, data blocks and decoded fields for inspecting the underlying representation. -## Scope and guarantees +## Scope -FiveFury aims for runtime-compatible binary output, but GTA V formats contain edition-specific and asset-specific structures. Passing validation proves the modeled binary invariants; it is not a substitute for testing newly authored content in the target game. +| Area | Formats | +| --- | --- | +| Maps and world data | YMAP, YTYP, YMF, GTXD, `gta5_cache_y.dat`, `heightmap.dat`, `water.xml` | +| Geometry and textures | YDR, YDD, YFT, YTD | +| Collision and navigation | YBN, YND, YNV | +| Animation and scenes | YCD, YED, `.cut` | +| Audio and text | AWC, REL, GXT2 | +| Archives and metadata | RPF7, DLC metadata, YMT, META, PSO, RBF | -`YFT`, `REL`, `YED`, and `YMT` expose substantial read/write functionality, but not every runtime subtype is modeled semantically. Unknown metadata is preserved where the container supports lossless rewriting instead of being guessed. PS3 `CDR` and RPF support is currently focused on reading and extraction, while PC RPF7 supports authoring. +Support includes reading, writing and validation for the modeled structures. Some variants remain partially understood, particularly within YFT, REL, YED and YMT. Console CDR and PS3 RPF support is focused on reading and extraction. -## License +Use `GameTarget` where an API exposes edition-specific behavior. Validation checks the implemented binary contracts; newly authored assets still need verification in the target game. -FiveFury is released under [The Unlicense](LICENSE). +## Development -See [CHANGELOG.md](CHANGELOG.md) for release history and compatibility notes. +**0.5.1 is the last planned feature release of the current Python implementation.** Core development is moving to .NET. -## Credits +The [source code](https://github.com/Hancapo/fivefury/tree/main/fivefury) and [tests](https://github.com/Hancapo/fivefury/tree/main/tests) provide further examples and format-specific contracts. See the [changelog](https://github.com/Hancapo/fivefury/blob/main/CHANGELOG.md) for release history. -Thomas Vanini — GTA V Enhanced PT-BR dubbing project creator, workflow coordinator, and in-game runtime validation — https://linktr.ee/ThomasVanini +Released under [The Unlicense](https://github.com/Hancapo/fivefury/blob/main/LICENSE). diff --git a/fivefury/common.py b/fivefury/common.py index 8782b86..f530d6b 100644 --- a/fivefury/common.py +++ b/fivefury/common.py @@ -7,9 +7,10 @@ import tempfile from collections.abc import Callable, Mapping from contextlib import ExitStack -from enum import IntEnum +from enum import IntEnum, IntFlag +from functools import lru_cache from pathlib import Path -from typing import Any, TypeAlias +from typing import Any, TypeAlias, TypeVar from .hashing import jenk_hash from .metahash import MetaHash @@ -143,6 +144,19 @@ def _missing_(cls, value: object) -> FlexibleIntEnum: return member +_IntEnumT = TypeVar("_IntEnumT", bound=IntEnum | IntFlag) + + +@lru_cache(maxsize=1024) +def _int_enum_member(enum_type: type[_IntEnumT], value: int) -> _IntEnumT: + return enum_type(value) + + +def coerce_int_enum(enum_type: type[_IntEnumT], value: int | _IntEnumT) -> _IntEnumT: + """Reuse immutable enum values while bounding retention of unknown integers.""" + return value if isinstance(value, enum_type) else _int_enum_member(enum_type, int(value)) + + __all__ = [ "ByteSource", "FlexibleIntEnum", @@ -150,6 +164,7 @@ def _missing_(cls, value: object) -> FlexibleIntEnum: "atomic_write_bytes", "atomic_write_files", "clip_short_name", + "coerce_int_enum", "dataclass_init_values", "hash_value", "read_source_bytes", diff --git a/fivefury/map_extension_codec.py b/fivefury/map_extension_codec.py new file mode 100644 index 0000000..a77f30d --- /dev/null +++ b/fivefury/map_extension_codec.py @@ -0,0 +1,42 @@ +"""Bindings for known extension models, shared by their owning map formats.""" + +from functools import lru_cache, partial +from typing import get_origin + +from .colors import parse_css_rgb +from .map_extensions import KNOWN_EXTENSION_TYPES, LightAttrDef +from .meta.backed import _VALUE_TYPES, _model_annotations, _model_fields, _snake_to_camel +from .meta.defs import meta_name +from .meta.materialize import MetaModelCodec + +_decoded_rgb = lru_cache(maxsize=512)(parse_css_rgb) + + +def extension_model_codecs(schemas, model_types=KNOWN_EXTENSION_TYPES): + codecs = {} + + def bind(cls): + name_hash = meta_name(cls.META_NAME) + if name_hash in codecs: + return codecs[name_hash] + attributes, converters, arrays = {}, {}, {} + hints = _model_annotations(cls) + for field in _model_fields(cls): + name = cls.META_FIELD_MAP.get(field.name, _snake_to_camel(field.name)) + attributes[name] = field.name + annotation = hints[field.name] + if annotation in _VALUE_TYPES: + converters[name] = (annotation, tuple(item.name for item in _model_fields(annotation))) + elif get_origin(annotation) is list: + converters[name] = partial(cls._deserialize_field, field.name) + if (item_type := cls.META_LIST_TYPES.get(field.name)) is not None: + arrays[name] = {meta_name(item_type.META_NAME): bind(item_type)} + # Preserve the constructor's interpretation of component 1, including byte inputs. + if cls is LightAttrDef: + converters.update(colour=_decoded_rgb, volOuterColour=_decoded_rgb) + codec = MetaModelCodec(schemas[name_hash], cls, converters=converters, + attributes=attributes, array_codecs=arrays) + codecs[name_hash] = codec + return codec + + return {meta_name(cls.META_NAME): bind(cls) for cls in model_types} diff --git a/fivefury/meta/backed.py b/fivefury/meta/backed.py index 880e811..671c7a6 100644 --- a/fivefury/meta/backed.py +++ b/fivefury/meta/backed.py @@ -2,19 +2,38 @@ import dataclasses from collections.abc import Mapping +from functools import lru_cache from typing import Any, ClassVar, get_args, get_origin, get_type_hints +from .. import _native_abi3 from ..vector import Quaternion, Vector2, Vector3, Vector4 from .defs import meta_name _VALUE_TYPES = (Vector2, Vector3, Vector4, Quaternion) +@lru_cache(maxsize=512) def _snake_to_camel(value: str) -> str: head, *tail = value.split("_") return head + "".join(part[:1].upper() + part[1:] for part in tail) +@lru_cache(maxsize=256) +def _model_annotations(model_type: type) -> dict[str, Any]: + return get_type_hints(model_type) + + +@lru_cache(maxsize=256) +def _model_fields(model_type: type) -> tuple[dataclasses.Field, ...]: + return dataclasses.fields(model_type) + + +@lru_cache(maxsize=256) +def _field_bindings(model_type: type, field_map: tuple) -> tuple: + names = dict(field_map) + return tuple((field.name, names.get(field.name, _snake_to_camel(field.name))) for field in _model_fields(model_type)) + + @dataclasses.dataclass(slots=True) class MetaBackedStruct: META_NAME: ClassVar[str] = "" @@ -22,8 +41,8 @@ class MetaBackedStruct: META_LIST_TYPES: ClassVar[dict[str, type[MetaBackedStruct]]] = {} def __post_init__(self) -> None: - hints = get_type_hints(type(self)) - for field in dataclasses.fields(self): + hints = _model_annotations(type(self)) + for field in _model_fields(type(self)): annotation = hints.get(field.name) value = getattr(self, field.name) if annotation in _VALUE_TYPES and not isinstance(value, annotation): @@ -34,12 +53,11 @@ def __post_init__(self) -> None: raise TypeError(f"{type(self).__name__}.{field.name} must contain {element_type.__name__} values") def to_meta(self) -> dict[str, Any]: - data: dict[str, Any] = {"_meta_name_hash": meta_name(self.META_NAME)} - for field in dataclasses.fields(self): - attr = field.name - meta_field = self.META_FIELD_MAP.get(attr, _snake_to_camel(attr)) - data[meta_field] = self._serialize_field(attr, getattr(self, attr)) - return data + serialize = self._serialize_field + custom = getattr(serialize, "__func__", None) is not MetaBackedStruct._serialize_field + return _native_abi3.meta_struct_values(self, + _field_bindings(type(self), tuple(self.META_FIELD_MAP.items())), + {"_meta_name_hash": meta_name(self.META_NAME)}, serialize, custom) def _serialize_field(self, attr: str, value: Any) -> Any: if isinstance(value, list): @@ -53,9 +71,7 @@ def from_meta(cls, value: Any) -> MetaBackedStruct: if not isinstance(value, Mapping): return cls() kwargs: dict[str, Any] = {} - for field in dataclasses.fields(cls): - attr = field.name - meta_field = cls.META_FIELD_MAP.get(attr, _snake_to_camel(attr)) + for attr, meta_field in _field_bindings(cls, tuple(cls.META_FIELD_MAP.items())): if meta_field not in value: continue kwargs[attr] = cls._deserialize_field(attr, value.get(meta_field)) @@ -66,7 +82,7 @@ def _deserialize_field(cls, attr: str, value: Any) -> Any: item_type = cls.META_LIST_TYPES.get(attr) if item_type is not None: return [item_type.from_meta(item) if isinstance(item, Mapping) else item for item in (value or [])] - annotation = get_type_hints(cls).get(attr) + annotation = _model_annotations(cls).get(attr) if annotation in _VALUE_TYPES: return annotation.from_iterable(value) if get_origin(annotation) is list: diff --git a/fivefury/meta/builder.py b/fivefury/meta/builder.py index 349e2ab..9eaef53 100644 --- a/fivefury/meta/builder.py +++ b/fivefury/meta/builder.py @@ -2,13 +2,13 @@ import dataclasses import struct +from functools import lru_cache from collections.abc import Iterable, Mapping, Sequence from typing import Any from .. import _native_abi3 -from ..binary import align, pad_bytes +from ..binary import align from ..hashing import jenk_hash -from ..metahash import MetaHash from ..resource import get_resource_flags_from_blocks, get_resource_total_page_count from ..vector import Quaternion, Vector2, Vector3, Vector4 from . import ( @@ -16,30 +16,18 @@ META_FILE_VFT, META_ROOT_SIZE, RESOURCE_FILE_BASE_SIZE, - MetaArrayRef, - MetaDataRef, MetaEnumInfo, - MetaFieldInfo, - MetaPointer, MetaStructInfo, RawStruct, ) -from .codec import camel_to_snake, scalar_plan +from .codec import camel_to_snake, scalar_plan, schema_enum_bytes, schema_field_bytes from .defs import ( META_TYPE_NAME_ARRAYINFO, - META_TYPE_NAME_BYTE, - META_TYPE_NAME_FLOAT, - META_TYPE_NAME_HASH, - META_TYPE_NAME_POINTER, - META_TYPE_NAME_STRING, - META_TYPE_NAME_UINT, - META_TYPE_NAME_USHORT, - META_TYPE_NAME_VECTOR4, STRUCTS_BY_HASH, SYSTEM_BASE, MetaDataType, ) -from .utils import array_info_for_field +from .utils import _array_info_for_entries @dataclasses.dataclass(slots=True) @@ -71,11 +59,9 @@ def __init__( self.used_struct_hashes: set[int] = set() self.used_enum_hashes: set[int] = set() self.blocks: list[_WritableBlock] = [] - self._block_group: dict[int, list[int]] = {} self.page_size: int = 0x2000 self.page_count: int = 1 self.page_flags: int = 0 - self._scalar_plans: dict = {} def register_struct(self, info: MetaStructInfo) -> None: if info.name_hash not in self.struct_infos: @@ -88,15 +74,19 @@ def register_enum(self, info: MetaEnumInfo) -> None: self.enum_infos[info.name_hash] = info def build(self, root_name_hash: int = 0, root_value: Mapping[str, Any] | RawStruct | None = None) -> bytes: - self._scalar_plans.clear() self.blocks.clear() - self._block_group.clear() self.used_struct_hashes.clear() self.used_enum_hashes.clear() if root_name_hash and root_value is not None: - root_block_id = self._reserve_block(root_name_hash) - root_payload = self._encode_struct_payload(root_name_hash, root_value) - self._set_block(root_block_id, root_payload) + blocks, used_hashes = _native_abi3.meta_graph_write( + root_name_hash, root_value, self._graph_schema, RawStruct, Mapping, + _coerce_float_xyz, self._inline_size, jenk_hash, + self.MAX_BLOCK_LENGTH, FLOAT_XYZ_NAME_HASH, + ) + self.blocks.extend(_WritableBlock(name_hash, data) for name_hash, data in blocks) + for name_hash in used_hashes: + self._mark_struct_used(name_hash) + root_block_id = 1 else: root_block_id = 0 return self._compose_system_stream(root_block_id) @@ -125,35 +115,18 @@ def _layout_data_blocks(self, start_offset: int) -> tuple[int, list[int], int]: total_size = align(offset, page_size) return page_size, offsets, total_size - def _reserve_block(self, name_hash: int) -> int: - self.blocks.append(_WritableBlock(name_hash=name_hash)) - block_id = len(self.blocks) - self._block_group.setdefault(name_hash, []).append(block_id - 1) - return block_id - - def _set_block(self, block_id: int, data: bytes) -> None: - self.blocks[block_id - 1].data = bytearray(pad_bytes(data, 16)) + def _graph_schema(self, name_hash: int): + info = self._lookup_struct_info(name_hash) + return _graph_plan(info.structure_size, tuple(info.entries)) - def _add_block( - self, - name_hash: int, - data: bytes, - *, - align_item: int = 16, - group: bool = True, - ) -> MetaPointer: - item = pad_bytes(data, align_item) if align_item > 1 else bytes(data) - if group: - for index in self._block_group.get(name_hash, ()): - block = self.blocks[index] - if len(block.data) >= self.MAX_BLOCK_LENGTH: - continue - pointer = MetaPointer(block_id=index + 1, offset=len(block.data)) - block.data.extend(item) - return pointer - block_id = self._reserve_block(name_hash) - self.blocks[block_id - 1].data.extend(item) - return MetaPointer(block_id=block_id, offset=0) + def _inline_size(self, name_hash: int) -> int: + info = self.struct_infos.get(name_hash) + if info is not None: + return info.structure_size + definition = STRUCTS_BY_HASH.get(name_hash) + if definition is not None: + return definition.size + raise KeyError(f"No size information available for inline structure 0x{name_hash:08X}") def _lookup_struct_info(self, name_hash: int) -> MetaStructInfo: info = self.struct_infos.get(name_hash) @@ -191,131 +164,6 @@ def _mark_struct_used(self, name_hash: int) -> None: elif entry.data_type is MetaDataType.STRUCTURE and entry.reference_key: self._mark_struct_used(entry.reference_key) - def _encode_struct_payload(self, name_hash: int, value: Mapping[str, Any] | RawStruct) -> bytes: - self._mark_struct_used(name_hash) - if name_hash == FLOAT_XYZ_NAME_HASH: - x, y, z = _coerce_float_xyz(value) - return struct.pack(" Any: - if field_name in value: - return value[field_name] - snake_name = camel_to_snake(field_name) - return value.get(snake_name) - - def _encode_field(self, struct_info: MetaStructInfo, field_index: int, field: MetaFieldInfo, value: Any) -> bytes: - data_type = field.data_type - match data_type: - case MetaDataType.ARRAY_OF_CHARS: - length = field.reference_key & 0xFFFF - text = (value or "").encode("ascii", errors="ignore")[:length] - return text + (b"\x00" * (length - len(text))) - case MetaDataType.ARRAY_OF_BYTES: - count = field.reference_key & 0xFFFF - array_info = array_info_for_field(struct_info, field_index) - if array_info is None: - raw = bytes(value or b"")[:count] - return raw + (b"\x00" * (count - len(raw))) - return _pack_inline_array(array_info.data_type, value, count) - case MetaDataType.CHAR_POINTER: - if not value: - return MetaArrayRef(MetaPointer(0, 0), 0, 0, 0).to_bytes() - raw = str(value).encode("ascii", errors="ignore") + b"\x00" - pointer = self._add_block( - META_TYPE_NAME_STRING, raw, align_item=1, group=True - ) - return MetaArrayRef(pointer, len(raw) - 1, len(raw) - 1, 0).to_bytes() - case MetaDataType.DATA_BLOCK_POINTER: - raw = bytes(value or b"") - if not raw: - return MetaDataRef(MetaPointer(0, 0)).to_bytes() - target_hash = ( - field.reference_key - if field.reference_key not in (0, 2) - else META_TYPE_NAME_BYTE - ) - pointer = self._add_block(target_hash, raw, align_item=1, group=False) - return MetaDataRef(pointer).to_bytes() - case MetaDataType.STRUCTURE_POINTER: - if value is None: - return MetaDataRef(MetaPointer(0, 0)).to_bytes() - target_hash = _value_struct_hash(value, fallback=field.reference_key) - self._mark_struct_used(target_hash) - pointer = self._add_block( - target_hash, self._encode_struct_payload(target_hash, value) - ) - return MetaDataRef(pointer).to_bytes() - case MetaDataType.STRUCTURE: - target_hash = field.reference_key or _value_struct_hash( - value, fallback=0 - ) - self._mark_struct_used(target_hash) - if value is None: - nested_info = self.struct_infos.get(field.reference_key) - if nested_info is not None: - return bytes(nested_info.structure_size) - nested_def = STRUCTS_BY_HASH.get(field.reference_key) - if nested_def is not None: - return bytes(nested_def.size) - raise KeyError( - "No size information available for inline structure " - f"0x{field.reference_key:08X}" - ) - return self._encode_struct_payload(target_hash, value) - case MetaDataType.ARRAY: - return self._encode_array(struct_info, field_index, value) - case _: - raise NotImplementedError(f"Unsupported META field type {data_type}") - - def _encode_array(self, struct_info: MetaStructInfo, field_index: int, value: Any) -> bytes: - if not value: - return bytes(16) - items = list(value or []) - if field_index <= 0: - return MetaArrayRef(MetaPointer(0, 0), 0, 0, 0).to_bytes() - array_info = array_info_for_field(struct_info, field_index) - if array_info is None: - return MetaArrayRef(MetaPointer(0, 0), 0, 0, 0).to_bytes() - element_type = array_info.data_type - if not items: - return MetaArrayRef(MetaPointer(0, 0), 0, 0, 0).to_bytes() - if element_type is MetaDataType.STRUCTURE_POINTER: - raw = bytearray() - for item in items: - target_hash = _value_struct_hash(item, fallback=0) - self._mark_struct_used(target_hash) - pointer = self._add_block(target_hash, self._encode_struct_payload(target_hash, item)) - raw.extend(struct.pack(" bytes: struct_infos = [self.struct_infos[name_hash] for name_hash in self.struct_info_order if name_hash in self.used_struct_hashes] enum_infos = [self.enum_infos[name_hash] for name_hash in self.enum_info_order if name_hash in self.used_enum_hashes] @@ -328,18 +176,7 @@ def _compose_system_stream(self, root_block_id: int) -> bytes: struct_entry_offsets: dict[int, int] = {} struct_entry_payloads: list[tuple[MetaStructInfo, bytes, int]] = [] for info in struct_infos: - payload = b"".join( - struct.pack( - " bytes: enum_entry_offsets: dict[int, int] = {} enum_entry_payloads: list[tuple[MetaEnumInfo, bytes, int]] = [] for info in enum_infos: - payload = b"".join(struct.pack(" bytes: - if data_type is MetaDataType.FLOAT: - vals = [float(v) for v in items] - return struct.pack(f"<{len(vals)}f", *vals) - if data_type in (MetaDataType.UNSIGNED_INT, MetaDataType.HASH): - vals = [_coerce_hash(v) if data_type is MetaDataType.HASH else int(v) for v in items] - return struct.pack(f"<{len(vals)}I", *vals) - if data_type is MetaDataType.UNSIGNED_SHORT: - vals = [int(v) for v in items] - return struct.pack(f"<{len(vals)}H", *vals) - if data_type is MetaDataType.UNSIGNED_BYTE: - return bytes(int(item) & 0xFF for item in items) - if data_type is MetaDataType.FLOAT_XYZ: - flat: list[float] = [] - for item in items: - flat.extend(_coerce_vector(item, 3)) - flat.append(0.0) - return struct.pack(f"<{len(flat)}f", *flat) - raise NotImplementedError(f"Unsupported array element type {data_type}") - - -def _primitive_block_hash(data_type: MetaDataType) -> int: - if data_type is MetaDataType.FLOAT: - return META_TYPE_NAME_FLOAT - if data_type is MetaDataType.HASH: - return META_TYPE_NAME_HASH - if data_type is MetaDataType.UNSIGNED_INT: - return META_TYPE_NAME_UINT - if data_type is MetaDataType.UNSIGNED_SHORT: - return META_TYPE_NAME_USHORT - if data_type is MetaDataType.UNSIGNED_BYTE: - return META_TYPE_NAME_BYTE - if data_type is MetaDataType.FLOAT_XYZ: - return META_TYPE_NAME_VECTOR4 - raise NotImplementedError(f"No primitive block hash for {data_type}") - - -def _coerce_hash(value: Any) -> int: - if value is None: - return 0 - if isinstance(value, MetaHash): - return int(value) - if isinstance(value, str): - return jenk_hash(value) - return int(value) - - -def _coerce_vector(value: Any, size: int) -> tuple[float, ...]: - if value is None: - return tuple(0.0 for _ in range(size)) - if isinstance(value, (Vector2, Vector3, Vector4, Quaternion)): - components = value.components - elif isinstance(value, (str, bytes, bytearray)): - raise TypeError("Vector values must be numeric iterables") - else: - components = tuple(value) - if len(components) != size: - raise ValueError(f"Expected vector of length {size}") - return tuple(float(component) for component in components) - - -def _value_struct_hash(value: Any, *, fallback: int) -> int: - if isinstance(value, RawStruct): - return value.name_hash - if isinstance(value, Mapping): - meta_name_hash = value.get("_meta_name_hash") - if meta_name_hash: - return int(meta_name_hash) - if fallback: - return fallback - raise ValueError("Structure hash is required for pointer/inline structure encoding") - - def _coerce_float_xyz(value: Any) -> tuple[float, float, float]: if isinstance(value, Mapping): return ( @@ -564,36 +328,16 @@ def _coerce_float_xyz(value: Any) -> tuple[float, float, float]: return tuple(float(component) for component in padded[:3]) -def _pack_inline_array(data_type: MetaDataType, value: Any, count: int) -> bytes: - if count <= 0: - return b"" - items = list(value or []) - if len(items) < count: - items.extend(0 for _ in range(count - len(items))) - items = items[:count] - if data_type is MetaDataType.FLOAT: - return struct.pack(f"<{count}f", *(float(v) for v in items)) - if data_type is MetaDataType.UNSIGNED_INT: - return struct.pack(f"<{count}I", *(int(v) for v in items)) - if data_type is MetaDataType.HASH: - return struct.pack(f"<{count}I", *(_coerce_hash(v) for v in items)) - if data_type is MetaDataType.SIGNED_INT: - return struct.pack(f"<{count}i", *(int(v) for v in items)) - if data_type is MetaDataType.UNSIGNED_SHORT: - return struct.pack(f"<{count}H", *(int(v) for v in items)) - if data_type is MetaDataType.SIGNED_SHORT: - return struct.pack(f"<{count}h", *(int(v) for v in items)) - if data_type is MetaDataType.UNSIGNED_BYTE: - return bytes(int(item) & 0xFF for item in items) - if data_type is MetaDataType.SIGNED_BYTE: - return struct.pack(f"<{count}b", *(int(v) for v in items)) - raw = bytes(value or b"")[:count] - return raw + (b"\x00" * (count - len(raw))) - - - - - - - - +@lru_cache(maxsize=256) +def _graph_plan(size, entries): + scalar, complex_fields = scalar_plan(size, entries) + fields = [] + for index, entry, name in complex_fields: + array_info = _array_info_for_entries(entries, index) + # ARRAY's legacy contract ignores index zero even when its ref_index identifies ARRAYINFO. + if entry.data_type is MetaDataType.ARRAY and index == 0: + array_info = None + fields.append((name, camel_to_snake(name), entry.data_offset, int(entry.data_type), + entry.reference_key, int(array_info.data_type) if array_info else -1, + array_info.reference_key if array_info else 0)) + return _native_abi3.meta_graph_new(scalar, fields) diff --git a/fivefury/meta/codec.py b/fivefury/meta/codec.py index a45c059..d97a3b2 100644 --- a/fivefury/meta/codec.py +++ b/fivefury/meta/codec.py @@ -1,9 +1,15 @@ """Compile immutable META field descriptions for the shared native scalar codec.""" from functools import lru_cache +import struct from .. import _native_abi3 from .defs import META_TYPE_NAME_ARRAYINFO, MetaDataType +from .utils import _array_info_for_entries + +_INLINE_SCALARS = frozenset((MetaDataType.FLOAT, MetaDataType.HASH, + MetaDataType.SIGNED_BYTE, MetaDataType.UNSIGNED_BYTE, MetaDataType.SIGNED_SHORT, + MetaDataType.UNSIGNED_SHORT, MetaDataType.SIGNED_INT, MetaDataType.UNSIGNED_INT)) _SCALARS = frozenset( ( @@ -47,6 +53,24 @@ def scalar_plan(size, entries): scalars.append( (name, camel_to_snake(name), entry.data_offset, int(entry.data_type)) ) + elif (entry.data_type is MetaDataType.ARRAY_OF_BYTES + and (array_info := _array_info_for_entries(entries, index)) is not None + and array_info.data_type in _INLINE_SCALARS): + scalars.append((name, camel_to_snake(name), entry.data_offset, + int(array_info.data_type), entry.reference_key & 0xFFFF)) else: complex_fields.append((index, entry, name)) return _native_abi3.meta_scalars_new(size, scalars), tuple(complex_fields) + + +@lru_cache(maxsize=256) +def schema_field_bytes(entries): + """Encode an immutable field snapshot; owner headers and pointers remain per-file.""" + return b"".join(struct.pack(" tuple: + return (info.name_hash, info.key, info.unknown, info.unknown_ch, info.unknown_1ch, + info.structure_size, tuple(info.entries)) + + +class MetaModelCodec: + """Internal binding for a known model; converters establish its constructor invariants. + + Snapshots schema layout and bindings. Complex references share the META resolver. + Bind a root or an explicit child array, never all occurrences of a type in an inspection graph. + """ + + def __init__(self, info: MetaStructInfo, model_type: type, *, + converters: Mapping[str, Any], attributes: Mapping[str, str] | None = None, + array_items: Mapping[str, Any] | None = None, + array_codecs: Mapping[str, dict[int, 'MetaModelCodec']] | None = None): + attributes = attributes or {} + array_items = array_items or {} + array_codecs = array_codecs or {} + def attribute(name): + return attributes.get(name, camel_to_snake(name)) + scalars, complex_fields = scalar_plan(info.structure_size, tuple(info.entries)) + complex_indices = {index for index, _, _ in complex_fields} + bindings = [(attribute(entry.name), converters.get(entry.name)) for index, entry in enumerate(info.entries) + if entry.name_hash != META_TYPE_NAME_ARRAYINFO and index not in complex_indices] + self._layout = _layout(info) + complex_bindings = [] + references = [] + opaque_fields = False + for index, entry, name in complex_fields: + convert = converters.get(name) + # Packed inline float arrays also carry nominal vectors in some schemas. + if isinstance(convert, tuple): + convert = convert[0].from_iterable + complex_bindings.append((index, entry, attribute(name), convert, array_items.get(name), array_codecs.get(name))) + if entry.data_type is MetaDataType.ARRAY: + array_info = array_info_for_field(info, index) + element_type = int(array_info.data_type) if array_info is not None and array_info.data_type in _ARRAY_SCALARS and convert is None else 0 + references.append((attribute(name), entry.data_offset, int(entry.data_type), element_type, + array_items.get(name) if element_type else None)) + elif entry.data_type is MetaDataType.CHAR_POINTER and convert is None: + references.append((attribute(name), entry.data_offset, int(entry.data_type), 0, None)) + else: + opaque_fields = True + self._complex = tuple(complex_bindings) + self._plan = _native_abi3.meta_model_new(scalars, model_type, bindings, references, opaque_fields) + + def matches(self, info: MetaStructInfo) -> bool: + return _layout(info) == self._layout + + def read(self, reader, info: MetaStructInfo, raw: bytes): + model = _native_abi3.meta_model_read(self._plan, raw, reader._data_buffers) + return self.complete(reader, info, model, raw) + + def complete(self, reader, info: MetaStructInfo, model, raw: bytes): + for index, entry, attribute, convert, convert_item, codecs in self._complex: + if hasattr(model, attribute): + continue + value = reader._decode_field(info, index, entry, raw, codecs=codecs) + if convert_item is not None: + # Empty decoded arrays are already fresh owned lists. + if value: + value = [convert_item(item) for item in value] + elif convert is not None: + value = convert(value) + setattr(model, attribute, value) + return model diff --git a/fivefury/meta/read.py b/fivefury/meta/read.py index 65c6e99..aa65768 100644 --- a/fivefury/meta/read.py +++ b/fivefury/meta/read.py @@ -2,7 +2,11 @@ import dataclasses import struct -from typing import Any +from functools import lru_cache +from typing import TYPE_CHECKING, Any + +if TYPE_CHECKING: + from .materialize import MetaModelCodec from .. import _native_abi3 from ..binary import BinaryDocument, BinaryScalarType @@ -46,9 +50,12 @@ class ParsedMeta: system_flags: int | None = None graphics_flags: int | None = None _scalar_plans: dict = dataclasses.field(default_factory=dict, repr=False) + _codec_matches: dict[MetaModelCodec, bool] = dataclasses.field(default_factory=dict, repr=False) + _data_buffers: tuple[bytes, ...] = dataclasses.field(default=(), repr=False) + _array_bindings: dict = dataclasses.field(default_factory=dict, repr=False) @classmethod - def from_bytes(cls, data: bytes) -> ParsedMeta: + def from_bytes(cls, data: bytes, *, array_codecs=None, root_codec=None) -> ParsedMeta: resource_version = None system_flags = None graphics_flags = None @@ -99,8 +106,14 @@ def from_bytes(cls, data: bytes) -> ParsedMeta: resource_version=resource_version, system_flags=system_flags, graphics_flags=graphics_flags, + _data_buffers=tuple(block.data for block in data_blocks), ) - parsed.decoded_root = parsed.decode_block_by_id(root_block_index) + if (block := parsed.block_by_id(root_block_index)) is not None: + info = parsed.struct_infos.get(block.struct_name_hash) + if root_codec is not None and info is not None and root_codec.matches(info): + parsed.decoded_root = root_codec.read(parsed, info, block.data) + else: + parsed.decoded_root = parsed.decode_struct(block.struct_name_hash, block.data, array_codecs=array_codecs) return parsed def struct_info_for_hash(self, name_hash: int) -> MetaStructInfo | None: @@ -117,7 +130,7 @@ def decode_block_by_id(self, block_id: int) -> Any: return None return self.decode_struct(block.struct_name_hash, block.data) - def decode_struct(self, name_hash: int, raw: bytes) -> Any: + def decode_struct(self, name_hash: int, raw: bytes, *, array_codecs=None) -> Any: if name_hash == FLOAT_XYZ_NAME_HASH: return struct.unpack_from(" Any: capsule, complex_fields = plan values = _native_abi3.meta_scalars_read(capsule, raw) for index, entry, name in complex_fields: - values[name] = self._decode_field(struct_info, index, entry, raw) + codecs = array_codecs.get((name_hash, entry.name_hash)) if array_codecs else None + values[name] = self._decode_field(struct_info, index, entry, raw, codecs=codecs) values["_meta_name_hash"] = name_hash values["_meta_name"] = struct_info.name return values - def _decode_field(self, struct_info: MetaStructInfo, field_index: int, field: MetaFieldInfo, raw: bytes) -> Any: + def _decode_field(self, struct_info: MetaStructInfo, field_index: int, field: MetaFieldInfo, raw: bytes, *, codecs=None) -> Any: offset = field.data_offset data_type = field.data_type if data_type is MetaDataType.ARRAY: @@ -149,7 +163,7 @@ def _decode_field(self, struct_info: MetaStructInfo, field_index: int, field: Me if count == 0 or pointer & 0xFFF == 0: return [] return self._resolve_array( - struct_info, field_index, MetaArrayRef.from_bytes(raw, offset) + struct_info, field_index, MetaArrayRef.from_bytes(raw, offset), codecs=codecs ) match data_type: case MetaDataType.ARRAY_OF_CHARS: @@ -196,7 +210,7 @@ def _resolve_data_pointer(self, data_ref: MetaDataRef) -> bytes: return b"" return bytes(block.data[data_ref.pointer.offset :]) - def _resolve_struct_pointer(self, pointer: MetaPointer) -> Any: + def _resolve_struct_pointer(self, pointer: MetaPointer, codecs=None) -> Any: if pointer.is_null: return None block = self.block_by_id(pointer.block_id) @@ -210,20 +224,54 @@ def _resolve_struct_pointer(self, pointer: MetaPointer) -> Any: size = struct_def.size return RawStruct(block.struct_name_hash, block.data[pointer.offset : pointer.offset + size], None) size = struct_info.structure_size - return self.decode_struct(block.struct_name_hash, block.data[pointer.offset : pointer.offset + size]) - - def _decode_inline_structure(self, name_hash: int, raw: bytes) -> Any: + raw = block.data[pointer.offset : pointer.offset + size] + return self._decode_model_or_struct(struct_info, raw, codecs) + + def _decode_model_or_struct(self, struct_info, raw, codecs): + if codecs and (codec := codecs.get(struct_info.name_hash)) is not None: + matches = self._codec_matches.get(codec) + if matches is None: + matches = codec.matches(struct_info) + self._codec_matches[codec] = matches + if matches: + return codec.read(self, struct_info, raw) + return self.decode_struct(struct_info.name_hash, raw) + + def _resolve_pointer_value(self, pointer: int): + return self._resolve_struct_pointer(MetaPointer.from_uint64(pointer)) + + def _bindings_for_array(self, codecs): + key = tuple(codecs.items()) + if (bindings := self._array_bindings.get(key)) is not None: + return bindings + bindings = [] + for block in self.data_blocks: + codec = codecs.get(block.struct_name_hash) + info = self.struct_infos.get(block.struct_name_hash) + if codec is not None and info is not None: + matches = self._codec_matches.get(codec) + if matches is None: + matches = codec.matches(info) + self._codec_matches[codec] = matches + if matches: + bindings.append((codec._plan, codec.complete, info)) + continue + bindings.append(None) + self._array_bindings[key] = result = tuple(bindings) + return result + + def _decode_inline_structure(self, name_hash: int, raw: bytes, codecs=None) -> Any: struct_info = self.struct_infos.get(name_hash) struct_def = STRUCTS_BY_HASH.get(name_hash) if struct_info is not None: - return self.decode_struct(name_hash, raw[: struct_info.structure_size]) + return self._decode_model_or_struct(struct_info, raw[: struct_info.structure_size], codecs) if struct_def is not None: if struct_def.opaque: return RawStruct(name_hash, raw[: struct_def.size], None) return self.decode_struct(name_hash, raw[: struct_def.size]) return raw - def _resolve_array(self, struct_info: MetaStructInfo, field_index: int, array_ref: MetaArrayRef) -> Any: + def _resolve_array(self, struct_info: MetaStructInfo, field_index: int, array_ref: MetaArrayRef, *, codecs=None) -> Any: if array_ref.pointer.is_null or array_ref.count == 0: return [] if field_index >= len(struct_info.entries): @@ -235,19 +283,18 @@ def _resolve_array(self, struct_info: MetaStructInfo, field_index: int, array_re if block is None: return [] start = array_ref.pointer.offset - end = len(block.data) - data = block.data[start:end] element_type = array_info.data_type - document = BinaryDocument(block.data) + document = BinaryDocument(block.data) if element_type is not MetaDataType.STRUCTURE else None match element_type: case MetaDataType.STRUCTURE_POINTER: + pointers = document.read_array(start, array_ref.count, BinaryScalarType.UNSIGNED_LONG) + if codecs: + return _native_abi3.meta_models_read(self._bindings_for_array(codecs), self._data_buffers, pointers, self) return [ self._resolve_struct_pointer( - MetaPointer.from_uint64(int(pointer_value)) - ) - for pointer_value in document.read_array( - start, array_ref.count, BinaryScalarType.UNSIGNED_LONG + MetaPointer.from_uint64(int(pointer_value)), codecs ) + for pointer_value in pointers ] case MetaDataType.STRUCTURE: nested_hash = array_info.reference_key @@ -261,10 +308,18 @@ def _resolve_array(self, struct_info: MetaStructInfo, field_index: int, array_re else 0 ) if size <= 0: - return [RawStruct(nested_hash, data, nested_info)] + return [RawStruct(nested_hash, block.data[start:], nested_info)] + if codecs and (codec := codecs.get(nested_hash)) is not None and nested_info is not None: + matches = self._codec_matches.get(codec) + if matches is None: + matches = codec.matches(nested_info) + self._codec_matches[codec] = matches + if matches: + return _native_abi3.meta_model_array_read((codec._plan, codec.complete, nested_info), + self._data_buffers, array_ref.pointer.value, size, array_ref.count, self) return [ self._decode_inline_structure( - nested_hash, data[index * size : (index + 1) * size] + nested_hash, block.data[start + index * size : start + (index + 1) * size], codecs ) for index in range(array_ref.count) ] @@ -293,7 +348,7 @@ def _resolve_array(self, struct_info: MetaStructInfo, field_index: int, array_re components=3, ) case _: - return bytes(data) + return bytes(block.data[start:]) def _absolute_to_offset(pointer: int) -> int: @@ -317,10 +372,11 @@ def _read_struct_infos(data: bytes, pointer: int, count: int) -> dict[int, MetaS " dict[int, MetaS return infos +@lru_cache(maxsize=256) +def _read_schema_fields(data: bytes) -> tuple[MetaFieldInfo, ...]: + return tuple(MetaFieldInfo.from_bytes(data, offset) for offset in range(0, len(data), 16)) + + def _read_enum_infos(data: bytes, pointer: int, count: int) -> dict[int, MetaEnumInfo]: if pointer == 0 or count <= 0: return {} @@ -374,8 +435,8 @@ def _read_string_absolute(data: bytes, pointer: int) -> str: return data[offset:end].decode("ascii", errors="ignore") -def read_meta(data: bytes) -> ParsedMeta: - return ParsedMeta.from_bytes(data) +def read_meta(data: bytes, *, array_codecs=None, root_codec=None) -> ParsedMeta: + return ParsedMeta.from_bytes(data, array_codecs=array_codecs, root_codec=root_codec) def _inline_array_format(data_type: MetaDataType) -> tuple[str, int] | None: if data_type is MetaDataType.FLOAT: diff --git a/fivefury/meta/utils.py b/fivefury/meta/utils.py index 15a766d..13ed389 100644 --- a/fivefury/meta/utils.py +++ b/fivefury/meta/utils.py @@ -5,13 +5,17 @@ def array_info_for_field(struct_info: MetaStructInfo, field_index: int) -> MetaFieldInfo | None: - if not (0 <= field_index < len(struct_info.entries)): + return _array_info_for_entries(struct_info.entries, field_index) + + +def _array_info_for_entries(entries, field_index: int) -> MetaFieldInfo | None: + if not (0 <= field_index < len(entries)): return None - if field_index > 0 and struct_info.entries[field_index - 1].name_hash == META_TYPE_NAME_ARRAYINFO: - return struct_info.entries[field_index - 1] - ref_index = struct_info.entries[field_index].reference_type_index - if 0 <= ref_index < len(struct_info.entries): - candidate = struct_info.entries[ref_index] + if field_index > 0 and entries[field_index - 1].name_hash == META_TYPE_NAME_ARRAYINFO: + return entries[field_index - 1] + ref_index = entries[field_index].reference_type_index + if 0 <= ref_index < len(entries): + candidate = entries[ref_index] if candidate.name_hash == META_TYPE_NAME_ARRAYINFO: return candidate return None diff --git a/fivefury/metahash.py b/fivefury/metahash.py index 9b615dd..49eb28f 100644 --- a/fivefury/metahash.py +++ b/fivefury/metahash.py @@ -118,6 +118,9 @@ def __hash__(self) -> int: HashString = MetaHash HashLike = MetaHash | int | str +# A decoded uint establishes both constructor slots without resolving names. +_NATIVE_HASH_BINDING = (MetaHash, ("_value", "_cached_uint")) + def coerce_meta_hash(value: HashLike | None) -> MetaHash: return value if isinstance(value, MetaHash) else MetaHash(value) diff --git a/fivefury/ymap/codec.py b/fivefury/ymap/codec.py new file mode 100644 index 0000000..d77c002 --- /dev/null +++ b/fivefury/ymap/codec.py @@ -0,0 +1,84 @@ +"""Known entity layouts for direct domain reads; unknown layouts keep generic decoding.""" + +from functools import lru_cache, partial + +from ..common import _int_enum_member +from ..map_extensions import EXTENSION_TYPES_BY_META_NAME, KNOWN_EXTENSION_TYPES, extension_from_meta +from ..map_extension_codec import extension_model_codecs +from ..meta.defs import meta_name +from ..meta.materialize import MetaModelCodec +from ..metahash import _NATIVE_HASH_BINDING +from ..vector import Quaternion, Vector3 +from .defs import YMAP_STRUCT_INFOS +from .entities import EntityDef, MloInstanceDef, _entity_from_meta +from .enums import YmapContentFlags, YmapEntityFlags, YmapFlags, YmapLodLevel, YmapMloInstanceFlags, YmapPriorityLevel + + +def entity_array_codecs(): + # Custom registrations use the ordinary extension factory, including changes after warmup. + enabled = tuple(cls for cls in KNOWN_EXTENSION_TYPES if EXTENSION_TYPES_BY_META_NAME.get(cls.META_NAME) is cls) + return _entity_array_codecs(enabled) + + +@lru_cache(maxsize=8) +def _entity_array_codecs(enabled_extensions): + schemas = {info.name_hash: info for info in YMAP_STRUCT_INFOS} + extensions = extension_model_codecs(schemas, enabled_extensions) + converters = { + # Native scalar decoding has already produced exact Python integers. + "archetypeName": _NATIVE_HASH_BINDING, + "flags": partial(_int_enum_member, YmapEntityFlags), + "lodLevel": partial(_int_enum_member, YmapLodLevel), + "priorityLevel": partial(_int_enum_member, YmapPriorityLevel), + "position": (Vector3, ("x", "y", "z")), + "rotation": (Quaternion, ("x", "y", "z", "w")), + "MLOInstflags": partial(_int_enum_member, YmapMloInstanceFlags), + } + codecs = {meta_name(name): MetaModelCodec(schemas[meta_name(name)], cls, + converters=converters, attributes={"MLOInstflags": "mlo_inst_flags"}, + array_items={"extensions": extension_from_meta, "defaultEntitySets": _NATIVE_HASH_BINDING}, + array_codecs={"extensions": extensions}) + for name, cls in (("CEntityDef", EntityDef), ("CMloInstanceDef", MloInstanceDef))} + return {(meta_name("CMapData"), meta_name("entities")): codecs} + + +def map_model_codec(array_codecs): + entities = array_codecs[(meta_name("CMapData"), meta_name("entities"))] + return _map_model_codec(tuple(entities.items())) + + +def _from_meta_dict(model_type, value): + return model_type.from_meta(value) if isinstance(value, dict) else value + + +@lru_cache(maxsize=8) +def _map_model_codec(entities): + from .base import BlockDesc, PhysicsDictionary + from .cargens import CarGen + from .grass import InstancedMapData + from .lights import DistantLodLightsSoa, LodLightsSoa + from .model import Ymap + from .occluders import BoxOccluder, OccludeModel + from .timecycle import TimeCycleModifier + from .utils import coerce_container_lod + + info = next(info for info in YMAP_STRUCT_INFOS if info.name_hash == meta_name("CMapData")) + converters = { + "name": _NATIVE_HASH_BINDING, "parent": _NATIVE_HASH_BINDING, + "flags": partial(_int_enum_member, YmapFlags), "contentFlags": partial(_int_enum_member, YmapContentFlags), + "block": BlockDesc.from_meta, + **{name: (Vector3, ("x", "y", "z")) for name in + ("streamingExtentsMin", "streamingExtentsMax", "entitiesExtentsMin", "entitiesExtentsMax")}, + **{name: partial(_from_meta_dict, cls) for name, cls in + (("instancedData", InstancedMapData), ("LODLightsSOA", LodLightsSoa), ("DistantLODLightsSOA", DistantLodLightsSoa))}, + } + items = { + "entities": _entity_from_meta, "containerLods": coerce_container_lod, + "physicsDictionaries": PhysicsDictionary.from_meta, + **{name: partial(_from_meta_dict, cls) for name, cls in + (("boxOccluders", BoxOccluder), ("occludeModels", OccludeModel), + ("timeCycleModifiers", TimeCycleModifier), ("carGenerators", CarGen))}, + } + return MetaModelCodec(info, Ymap, converters=converters, array_items=items, + array_codecs={"entities": dict(entities)}, + attributes={"LODLightsSOA": "lod_lights", "DistantLODLightsSOA": "distant_lod_lights"}) diff --git a/fivefury/ymap/entities.py b/fivefury/ymap/entities.py index db0ecb5..c60efa6 100644 --- a/fivefury/ymap/entities.py +++ b/fivefury/ymap/entities.py @@ -221,3 +221,12 @@ def from_meta(cls, value: Any) -> MloInstanceDef: int(value.get("MLOInstflags", 0)) ), ) + + +def _entity_from_meta(value: Any): + if isinstance(value, dict): + if value.get("_meta_name") == "CMloInstanceDef": + return MloInstanceDef.from_meta(value) + if value.get("_meta_name") == "CEntityDef": + return EntityDef.from_meta(value) + return value diff --git a/fivefury/ymap/enums.py b/fivefury/ymap/enums.py index bbefc07..56d8747 100644 --- a/fivefury/ymap/enums.py +++ b/fivefury/ymap/enums.py @@ -2,7 +2,7 @@ import enum -from ..common import FlexibleIntEnum +from ..common import FlexibleIntEnum, coerce_int_enum class YmapFlags(enum.IntFlag): @@ -121,39 +121,39 @@ class YmapLodLightCategory(FlexibleIntEnum): def coerce_ymap_lod_level(value: int | YmapLodLevel) -> YmapLodLevel: - return value if isinstance(value, YmapLodLevel) else YmapLodLevel(int(value)) + return coerce_int_enum(YmapLodLevel, value) def coerce_ymap_priority_level(value: int | YmapPriorityLevel) -> YmapPriorityLevel: - return value if isinstance(value, YmapPriorityLevel) else YmapPriorityLevel(int(value)) + return coerce_int_enum(YmapPriorityLevel, value) def coerce_ymap_flags(value: int | YmapFlags) -> YmapFlags: - return value if isinstance(value, YmapFlags) else YmapFlags(int(value)) + return coerce_int_enum(YmapFlags, value) def coerce_ymap_content_flags(value: int | YmapContentFlags) -> YmapContentFlags: - return value if isinstance(value, YmapContentFlags) else YmapContentFlags(int(value)) + return coerce_int_enum(YmapContentFlags, value) def coerce_ymap_entity_flags(value: int | YmapEntityFlags) -> YmapEntityFlags: - return value if isinstance(value, YmapEntityFlags) else YmapEntityFlags(int(value)) + return coerce_int_enum(YmapEntityFlags, value) def coerce_ymap_cargen_flags(value: int | YmapCarGenFlags) -> YmapCarGenFlags: - return value if isinstance(value, YmapCarGenFlags) else YmapCarGenFlags(int(value)) + return coerce_int_enum(YmapCarGenFlags, value) def coerce_ymap_mlo_instance_flags(value: int | YmapMloInstanceFlags) -> YmapMloInstanceFlags: - return value if isinstance(value, YmapMloInstanceFlags) else YmapMloInstanceFlags(int(value)) + return coerce_int_enum(YmapMloInstanceFlags, value) def coerce_ymap_lod_light_type(value: int | YmapLodLightType) -> YmapLodLightType: - return value if isinstance(value, YmapLodLightType) else YmapLodLightType(int(value)) + return coerce_int_enum(YmapLodLightType, value) def coerce_ymap_lod_light_category(value: int | YmapLodLightCategory) -> YmapLodLightCategory: - return value if isinstance(value, YmapLodLightCategory) else YmapLodLightCategory(int(value)) + return coerce_int_enum(YmapLodLightCategory, value) __all__ = [ diff --git a/fivefury/ymap/mlo_validation.py b/fivefury/ymap/mlo_validation.py index e553f13..3585ab1 100644 --- a/fivefury/ymap/mlo_validation.py +++ b/fivefury/ymap/mlo_validation.py @@ -65,16 +65,16 @@ def build_mlo_instance(instance: MloInstanceDef, archetype: Any | None = None) - def validate_mlo_instance(instance: MloInstanceDef, archetype: Any | None = None) -> ValidationReport: - label = f"MLO instance {instance.archetype_name}" issues = ValidationReport() if not 0 <= int(instance.group_id) < 255: - issues.issue("ymap.mlo.group_id.range", f"{label} group_id must be between 0 and 254", path="group_id") + issues.issue("ymap.mlo.group_id.range", f"MLO instance {instance.archetype_name} group_id must be between 0 and 254", path="group_id") check_unsigned(issues, instance.floor_id, 32, code="ymap.mlo.floor_id.range", path="floor_id") check_unsigned(issues, instance.num_exit_portals, 32, code="ymap.mlo.exit_portals.range", path="num_exit_portals") if archetype is None: return issues + label = f"MLO instance {instance.archetype_name}" expected_exit_portals = exit_portal_count(archetype) if int(instance.num_exit_portals) != expected_exit_portals: issues.issue( diff --git a/fivefury/ymap/model.py b/fivefury/ymap/model.py index 16f7051..da82276 100644 --- a/fivefury/ymap/model.py +++ b/fivefury/ymap/model.py @@ -18,7 +18,8 @@ YMAP_STRUCT_INFOS, _ensure_base_name, ) -from .entities import EntityDef, MloInstanceDef +from .entities import EntityDef, MloInstanceDef, _entity_from_meta +from .codec import entity_array_codecs, map_model_codec from .enums import ( YmapContentFlags, YmapFlags, @@ -60,6 +61,18 @@ from ..ytyp import MloArchetypeDef, Ytyp +_CONTENT_FLAGS_BY_LOD = { + int(YmapLodLevel.DEPTH_HD): int(YmapContentFlags.ENTITIES_HD), + int(YmapLodLevel.DEPTH_ORPHANHD): int(YmapContentFlags.ENTITIES_HD), + int(YmapLodLevel.DEPTH_LOD): int(YmapContentFlags.ENTITIES_LOD), + int(YmapLodLevel.DEPTH_SLOD1): int(YmapContentFlags.ENTITIES_CRITICAL), + **dict.fromkeys( + (int(YmapLodLevel.DEPTH_SLOD2), int(YmapLodLevel.DEPTH_SLOD3), int(YmapLodLevel.DEPTH_SLOD4)), + int(YmapContentFlags.ENTITIES_CONTAINER_LOD | YmapContentFlags.ENTITIES_CRITICAL), + ), +} + + def _context_dependencies( context: BuildContext | None, ) -> tuple[tuple[Ytyp, ...] | None, dict[str, Ybn] | None]: @@ -378,29 +391,14 @@ def recalculate_extents( def recalculate_flags(self) -> Ymap: flags = self.flags & (YmapFlags.MANUAL_STREAM_ONLY | YmapFlags.IS_PARENT) - content_flags = YmapContentFlags.NONE + content_flags = 0 for entity in self.entities: - lod_level = YmapLodLevel(int(getattr(entity, "lod_level", YmapLodLevel.DEPTH_HD) or 0)) - match lod_level: - case YmapLodLevel.DEPTH_HD | YmapLodLevel.DEPTH_ORPHANHD: - content_flags |= YmapContentFlags.ENTITIES_HD - case YmapLodLevel.DEPTH_LOD: - content_flags |= YmapContentFlags.ENTITIES_LOD - case YmapLodLevel.DEPTH_SLOD1: - content_flags |= YmapContentFlags.ENTITIES_CRITICAL - case ( - YmapLodLevel.DEPTH_SLOD2 - | YmapLodLevel.DEPTH_SLOD3 - | YmapLodLevel.DEPTH_SLOD4 - ): - content_flags |= ( - YmapContentFlags.ENTITIES_CONTAINER_LOD - | YmapContentFlags.ENTITIES_CRITICAL - ) + lod_level = int(getattr(entity, "lod_level", YmapLodLevel.DEPTH_HD) or 0) + content_flags |= _CONTENT_FLAGS_BY_LOD.get(lod_level, 0) if isinstance(entity, MloInstanceDef) or getattr(entity, "_meta_name", "") == "CMloInstanceDef": - content_flags |= YmapContentFlags.MLO + content_flags |= int(YmapContentFlags.MLO) if self.block.version or self.block.flags or self.block.name or self.block.exported_by or self.block.owner or self.block.time: content_flags |= YmapContentFlags.BLOCKINFO @@ -417,7 +415,7 @@ def recalculate_flags(self) -> Ymap: content_flags |= YmapContentFlags.OCCLUDER self.flags = flags - self.content_flags = content_flags + self.content_flags = coerce_ymap_content_flags(content_flags) return self def build( @@ -554,18 +552,15 @@ def to_meta(self) -> Meta: @classmethod def from_bytes(cls, data: bytes) -> Ymap: - parsed = read_meta(data) + arrays = entity_array_codecs() + parsed = read_meta(data, array_codecs=arrays, root_codec=map_model_codec(arrays) if cls is Ymap else None) root = parsed.decoded_root + if isinstance(root, cls): + root.meta_name = parsed.name + return root if not isinstance(root, dict) or root.get("_meta_name") != "CMapData": raise ValueError("META payload is not a CMapData/YMAP") - entities: list[Any] = [] - for item in root.get("entities", []) or []: - if isinstance(item, dict) and item.get("_meta_name") == "CMloInstanceDef": - entities.append(MloInstanceDef.from_meta(item)) - elif isinstance(item, dict) and item.get("_meta_name") == "CEntityDef": - entities.append(EntityDef.from_meta(item)) - else: - entities.append(item) + entities = [_entity_from_meta(item) for item in root.get("entities", []) or []] container_lods = [coerce_container_lod(item) for item in root.get("containerLods", []) or []] return cls( name=root.get("name", 0), diff --git a/native/drawable/vector_bindings.cpp b/native/drawable/vector_bindings.cpp index 617141a..51831cc 100644 --- a/native/drawable/vector_bindings.cpp +++ b/native/drawable/vector_bindings.cpp @@ -1,7 +1,7 @@ #include "drawable/bindings.h" #include "math/vector.h" -#include "python/vector_factory.h" +#include "python/record_factory.h" #include #include @@ -48,26 +48,13 @@ PyObject* mod_vector_materialize(PyObject*, PyObject* args) { if (dimensions < 2 || dimensions > 4) { PyErr_SetString(PyExc_ValueError, "Vectors require 2 to 4 components"); return nullptr; } - VectorFactory factory(cls, fields.get()); + RecordFactory factory(cls, fields.get()); if (!factory) return nullptr; const auto count = PyTuple_Size(sequence.get()); PyHandle result(PyList_New(count)); if (!result) return nullptr; for (Py_ssize_t i = 0; i < count; ++i) { - PyHandle components(PySequence_Tuple(PyTuple_GetItem(sequence.get(), i))); - if (!components) return nullptr; - if (PyTuple_Size(components.get()) != dimensions) { - PyErr_SetString(PyExc_ValueError, "Incorrect vector component count"); return nullptr; - } - // The Python boundary supplies the nominal vector class. Match its - // float coercion while filling frozen slots before exposing the object. - PyHandle object(factory.create()); - if (!object) return nullptr; - for (Py_ssize_t j = 0; j < dimensions; ++j) { - PyHandle value(PyNumber_Float(PyTuple_GetItem(components.get(), j))); - if (!factory.assign(object.get(), j, value.get())) return nullptr; - } - if (!list_take(result.get(), i, object.release())) return nullptr; + if (!list_take(result.get(), i, factory.numeric(PyTuple_GetItem(sequence.get(), i)))) return nullptr; } return result.release(); } diff --git a/native/drawable/vertex_decode.cpp b/native/drawable/vertex_decode.cpp index 59c4c59..dde2496 100644 --- a/native/drawable/vertex_decode.cpp +++ b/native/drawable/vertex_decode.cpp @@ -1,5 +1,5 @@ #include "drawable/bindings.h" -#include "python/vector_factory.h" +#include "python/record_factory.h" #include #include @@ -378,9 +378,9 @@ PyObject* mod_ydr_decode_vertex_buffer(PyObject*, PyObject* args) { PyHandle names3(Py_BuildValue("(sss)", "x", "y", "z")); PyHandle names4(Py_BuildValue("(ssss)", "x", "y", "z", "w")); if (!names2 || !names3 || !names4) return nullptr; - VectorFactory vectors2(vector2_type, names2.get()); - VectorFactory vectors3(vector3_type, names3.get()); - VectorFactory vectors4(vector4_type, names4.get()); + RecordFactory vectors2(vector2_type, names2.get()); + RecordFactory vectors3(vector3_type, names3.get()); + RecordFactory vectors4(vector4_type, names4.get()); if (!vectors2 || !vectors3 || !vectors4) return nullptr; PyObject* result = PyDict_New(); diff --git a/native/python/module.cpp b/native/python/module.cpp index 6e81230..3e982c8 100644 --- a/native/python/module.cpp +++ b/native/python/module.cpp @@ -11,8 +11,15 @@ namespace fivefury_py { PyMethodDef module_methods[] = { + {"meta_model_new", guarded_call, METH_VARARGS, nullptr}, + {"meta_model_read", guarded_call, METH_VARARGS, nullptr}, + {"meta_models_read", guarded_call, METH_VARARGS, nullptr}, + {"meta_model_array_read", guarded_call, METH_VARARGS, nullptr}, + {"meta_struct_values", guarded_call, METH_VARARGS, nullptr}, {"vector_materialize", guarded_call, METH_VARARGS, nullptr}, {"vector_component_buffer", guarded_call, METH_VARARGS, nullptr}, + {"meta_graph_new", guarded_call, METH_VARARGS, nullptr}, + {"meta_graph_write", guarded_call, METH_VARARGS, nullptr}, {"meta_scalars_new", guarded_call, METH_VARARGS, nullptr}, {"meta_scalars_read", guarded_call, METH_VARARGS, nullptr}, {"meta_scalars_write", guarded_call, METH_VARARGS, nullptr}, diff --git a/native/python/ownership.h b/native/python/ownership.h index da6b39a..0159c00 100644 --- a/native/python/ownership.h +++ b/native/python/ownership.h @@ -13,6 +13,13 @@ class PyHandle { PyHandle(const PyHandle&) = delete; PyHandle& operator=(const PyHandle&) = delete; PyHandle(PyHandle&& other) noexcept : object_(other.release()) {} + PyHandle& operator=(PyHandle&& other) noexcept { + if (this != &other) { + auto* previous = std::exchange(object_, other.release()); + Py_XDECREF(previous); + } + return *this; + } PyObject* get() const noexcept { return object_; } PyObject* release() noexcept { return std::exchange(object_, nullptr); } explicit operator bool() const noexcept { return object_ != nullptr; } diff --git a/native/python/record_factory.h b/native/python/record_factory.h new file mode 100644 index 0000000..11694b9 --- /dev/null +++ b/native/python/record_factory.h @@ -0,0 +1,103 @@ +#pragma once + +#include "python/bridge.h" +#include +#include +#include + +namespace fivefury_py { + +// Fill prepared record/vector slots before exposing a new object to Python. +// Callers own field conversion and validation; descriptors still participate in assignment. +class RecordFactory { + PyObject* type_; + PyHandle names_; + PyHandle allocate_; + allocfunc allocate_native_ = nullptr; + struct Setter { PyHandle descriptor; descrsetfunc assign; }; + std::vector setters_; + bool valid_ = false; +public: + RecordFactory(PyObject* type, PyObject* names) + : type_(type), names_(PySequence_Tuple(names)), + allocate_(PyObject_GetAttrString(type, "__new__")) { + if (!names_ || !allocate_) return; + if (PyType_Check(type)) { + auto* cls = reinterpret_cast(type); + auto* create = PyType_GetSlot(cls, Py_tp_new); + if (create == reinterpret_cast(PyType_GenericNew) || create == PyType_GetSlot(&PyBaseObject_Type, Py_tp_new)) + allocate_native_ = reinterpret_cast(PyType_GetSlot(cls, Py_tp_alloc)); + if (PyErr_Occurred()) return; + } + for (Py_ssize_t i = 0; i < dimensions(); ++i) { + PyHandle descriptor(PyObject_GetAttr(type, PyTuple_GetItem(names_.get(), i))); + descrsetfunc setter = nullptr; + if (descriptor) { + setter = reinterpret_cast(PyType_GetSlot(Py_TYPE(descriptor.get()), Py_tp_descr_set)); + if (PyErr_Occurred()) return; + } else if (PyErr_ExceptionMatches(PyExc_AttributeError)) PyErr_Clear(); + else return; + setters_.push_back({std::move(descriptor), setter}); + } + valid_ = true; + } + explicit operator bool() const { return valid_; } + Py_ssize_t dimensions() const { return PyTuple_Size(names_.get()); } + PyObject* create() const { + if (allocate_native_) return allocate_native_(reinterpret_cast(type_), 0); + return PyObject_CallFunctionObjArgs(allocate_.get(), type_, nullptr); + } + bool assign(PyObject* object, Py_ssize_t index, PyObject* value) const { + const auto& setter = setters_[index]; + if (setter.assign) return value && setter.assign(setter.descriptor.get(), object, value) == 0; + return value && PyObject_GenericSetAttr(object, PyTuple_GetItem(names_.get(), index), value) == 0; + } + PyObject* numeric(PyObject* values) const { + PyHandle components(PySequence_Tuple(values)); + if (!components) return nullptr; + if (PyTuple_Size(components.get()) != dimensions()) { + PyErr_SetString(PyExc_ValueError, "Incorrect vector component count"); return nullptr; + } + PyHandle object(create()); + if (!object) return nullptr; + for (Py_ssize_t i = 0; i < dimensions(); ++i) { + PyHandle value(PyNumber_Float(PyTuple_GetItem(components.get(), i))); + if (!assign(object.get(), i, value.get())) return nullptr; + } + return object.release(); + } + PyObject* numeric(std::span values) const { + if (static_cast(values.size()) != dimensions()) { + PyErr_SetString(PyExc_ValueError, "Incorrect vector component count"); return nullptr; + } + PyHandle object(create()); + if (!object) return nullptr; + for (Py_ssize_t i = 0; i < dimensions(); ++i) { + PyHandle value(PyFloat_FromDouble(values[i])); + if (!assign(object.get(), i, value.get())) return nullptr; + } + return object.release(); + } + PyObject* scalar(PyObject* value) const { + PyHandle object(create()); + if (!object) return nullptr; + for (Py_ssize_t i = 0; i < dimensions(); ++i) + if (!assign(object.get(), i, value)) return nullptr; + return object.release(); + } + template + PyObject* materialize(const std::vector>& rows) const { + if (dimensions() != N) { + PyErr_SetString(PyExc_ValueError, "Incorrect vector component count"); return nullptr; + } + PyHandle result(PyList_New(static_cast(rows.size()))); + if (!result) return nullptr; + for (std::size_t i = 0; i < rows.size(); ++i) { + PyHandle object(numeric(std::span(rows[i]))); + if (!object) return nullptr; + if (!list_take(result.get(), i, object.release())) return nullptr; + } + return result.release(); + } +}; +} diff --git a/native/python/vector_factory.h b/native/python/vector_factory.h deleted file mode 100644 index ff64c4f..0000000 --- a/native/python/vector_factory.h +++ /dev/null @@ -1,46 +0,0 @@ -#pragma once - -#include "python/bridge.h" -#include -#include - -namespace fivefury_py { - -// Fill nominal frozen-vector slots before exposing a new object to Python. -// Both decoded native arrays and Python component rows use this boundary. -class VectorFactory { - PyObject* type_; - PyHandle names_; - PyHandle allocate_; -public: - VectorFactory(PyObject* type, PyObject* names) - : type_(type), names_(PySequence_Tuple(names)), - allocate_(PyObject_GetAttrString(reinterpret_cast(&PyBaseObject_Type), "__new__")) {} - explicit operator bool() const { return names_ && allocate_; } - Py_ssize_t dimensions() const { return PyTuple_Size(names_.get()); } - PyObject* create() const { - return PyObject_CallFunctionObjArgs(allocate_.get(), type_, nullptr); - } - bool assign(PyObject* object, Py_ssize_t index, PyObject* value) const { - return value && PyObject_GenericSetAttr(object, PyTuple_GetItem(names_.get(), index), value) == 0; - } - template - PyObject* materialize(const std::vector>& rows) const { - if (dimensions() != N) { - PyErr_SetString(PyExc_ValueError, "Incorrect vector component count"); return nullptr; - } - PyHandle result(PyList_New(static_cast(rows.size()))); - if (!result) return nullptr; - for (std::size_t i = 0; i < rows.size(); ++i) { - PyHandle object(create()); - if (!object) return nullptr; - for (std::size_t j = 0; j < N; ++j) { - PyHandle value(PyFloat_FromDouble(rows[i][j])); - if (!assign(object.get(), j, value.get())) return nullptr; - } - if (!list_take(result.get(), i, object.release())) return nullptr; - } - return result.release(); - } -}; -} diff --git a/native/resource/bindings.h b/native/resource/bindings.h index 046a3ff..b9665b1 100644 --- a/native/resource/bindings.h +++ b/native/resource/bindings.h @@ -3,9 +3,18 @@ #include "python/bridge.h" namespace fivefury_py { +bool meta_scalar_write(PyObject*, PyObject*, PyObject*, std::vector&); +bool meta_primitive_write(int, PyObject*, PyObject*, std::vector&); +PyObject* mod_meta_graph_new(PyObject*, PyObject*); +PyObject* mod_meta_graph_write(PyObject*, PyObject*); PyObject* mod_meta_scalars_new(PyObject*, PyObject*); PyObject* mod_meta_scalars_read(PyObject*, PyObject*); PyObject* mod_meta_scalars_write(PyObject*, PyObject*); +PyObject* mod_meta_model_new(PyObject*, PyObject*); +PyObject* mod_meta_model_read(PyObject*, PyObject*); +PyObject* mod_meta_models_read(PyObject*, PyObject*); +PyObject* mod_meta_model_array_read(PyObject*, PyObject*); +PyObject* mod_meta_struct_values(PyObject*, PyObject*); PyObject* mod_binary_document_new(PyObject*, PyObject* args); PyObject* mod_binary_document_size(PyObject*, PyObject* args); diff --git a/native/resource/meta_codec.cpp b/native/resource/meta_codec.cpp index 935d877..3c5ab04 100644 --- a/native/resource/meta_codec.cpp +++ b/native/resource/meta_codec.cpp @@ -1,4 +1,5 @@ #include "resource/bindings.h" +#include "python/record_factory.h" #include #include #include @@ -10,12 +11,38 @@ constexpr auto capsule_name = "fivefury.MetaScalars"; struct Field { PyHandle name, alias; int kind; - Py_ssize_t offset; + Py_ssize_t offset, count; }; struct Schema { Py_ssize_t size; std::vector fields; }; +constexpr auto model_capsule_name = "fivefury.MetaModel"; +struct ModelField { + PyHandle converter; + std::unique_ptr vector; + bool scalar_record = false; +}; +struct ReferenceField { + PyHandle name; + ModelField value; + Py_ssize_t offset; + int kind, element_kind; +}; +struct Model { + PyHandle schema_owner, type_owner; + Schema* schema; + RecordFactory factory; + std::vector fields; + std::vector references; + bool opaque_fields = false; + Model(PyHandle schema_ref, Schema* schema_value, PyHandle type_ref, PyObject* names) + : schema_owner(std::move(schema_ref)), type_owner(std::move(type_ref)), + schema(schema_value), factory(type_owner.get(), names) {} +}; +void destroy_model(PyObject* capsule) { + delete static_cast(PyCapsule_GetPointer(capsule, model_capsule_name)); +} std::size_t width(int kind) { switch (kind) { case 1: case 0x10: case 0x11: case 0x60: return 1; @@ -119,6 +146,56 @@ PyObject* decode(const char* in, int kind) { } } +namespace { +PyObject* decode_field(const char* data, const Field& field) { + const auto* in = data + field.offset; + if (field.count < 0) return decode(in, field.kind); + PyHandle result(PyTuple_New(field.count)); + if (!result) return nullptr; + const auto step = width(field.kind); + for (Py_ssize_t i = 0; i < field.count; ++i) + if (!tuple_take(result.get(), i, decode(in + i * step, field.kind))) return nullptr; + return result.release(); +} +bool encode_array_item(char* target, int kind, PyObject* item, PyObject* hash) { + switch (kind) { + case 0x21: if (!write_float(target, item)) return false; break; + case 0x11: { + PyHandle integer(PyNumber_Long(item)); + if (!integer) return false; + const auto number = PyLong_AsUnsignedLongLongMask(integer.get()); + if (PyErr_Occurred()) return false; + *target = static_cast(number & 0xff); break; + } + case 0x10: if (!write_integer(target, item)) return false; break; + case 0x12: if (!write_integer(target, item)) return false; break; + case 0x13: if (!write_integer(target, item)) return false; break; + case 0x14: if (!write_integer(target, item)) return false; break; + case 0x15: if (!write_integer(target, item)) return false; break; + default: if (!encode(target, kind, item, hash)) return false; break; + } + return true; +} +bool encode_field(char* data, const Field& field, PyObject* value, PyObject* hash) { + auto* out = data + field.offset; + if (field.count < 0) return encode(out, field.kind, value, hash); + if (field.count == 0) return true; + const int truth = PyObject_IsTrue(value); + if (truth < 0) return false; + if (!truth) return true; + PyHandle items(PySequence_Tuple(value)); + if (!items) return false; + const auto count = std::min(field.count, PyTuple_Size(items.get())); + const auto step = width(field.kind); + for (Py_ssize_t i = 0; i < count; ++i) { + auto* item = PyTuple_GetItem(items.get(), i); + auto* target = out + i * step; + if (!encode_array_item(target, field.kind, item, hash)) return false; + } + return true; +} +} + PyObject* mod_meta_scalars_new(PyObject*, PyObject* args) { Py_ssize_t size; PyObject* fields; if (!PyArg_ParseTuple(args, "nO", &size, &fields)) return nullptr; @@ -127,12 +204,15 @@ PyObject* mod_meta_scalars_new(PyObject*, PyObject* args) { PyHandle sequence(PySequence_Tuple(fields)); if (!sequence) return nullptr; for (Py_ssize_t i = 0; i < PyTuple_Size(sequence.get()); ++i) { - PyObject *name, *alias; int kind; Py_ssize_t offset; - if (!PyArg_ParseTuple(PyTuple_GetItem(sequence.get(), i), "UUni", &name, &alias, &offset, &kind)) return nullptr; - if (offset < 0 || !binary::contains(offset, width(kind), size)) + PyObject *name, *alias; int kind; Py_ssize_t offset, count = -1; + if (!PyArg_ParseTuple(PyTuple_GetItem(sequence.get(), i), "UUni|n", &name, &alias, &offset, &kind, &count)) return nullptr; + const auto step = width(kind); + if (count < -1 || (count >= 0 && (step > 4 || static_cast(count) > static_cast(size) / step))) + throw std::invalid_argument("Invalid META fixed array size"); + if (offset < 0 || !binary::contains(offset, step * (count < 0 ? 1 : count), size)) throw std::invalid_argument("META field exceeds structure size"); Py_INCREF(name); Py_INCREF(alias); - schema->fields.push_back({PyHandle(name), PyHandle(alias), kind, offset}); + schema->fields.push_back({PyHandle(name), PyHandle(alias), kind, offset, count}); } return owned_capsule(std::move(schema), capsule_name, destroy_schema); } @@ -145,11 +225,52 @@ PyObject* mod_meta_scalars_read(PyObject*, PyObject* args) { if (data.len < schema->size) throw std::invalid_argument("META structure is truncated"); PyHandle result(PyDict_New()); if (!result) return nullptr; for (const auto& field : schema->fields) { - PyHandle value(decode(static_cast(data.buf) + field.offset, field.kind)); + PyHandle value(decode_field(static_cast(data.buf), field)); if (!value || PyDict_SetItem(result.get(), field.name.get(), value.get()) < 0) return nullptr; } return result.release(); } +namespace { +bool encode_scalars(const Schema& schema, PyObject* values, PyObject* hash, char* out) { + for (const auto& field : schema.fields) { + PyHandle value(PyObject_GetItem(values, field.name.get())); + if (!value) { + if (!PyErr_ExceptionMatches(PyExc_KeyError)) return false; + PyErr_Clear(); + PyHandle alias(PyObject_GetItem(values, field.alias.get())); + if (!alias) { + if (!PyErr_ExceptionMatches(PyExc_KeyError)) return false; + PyErr_Clear(); continue; + } + if (!encode_field(out, field, alias.get(), hash)) return false; + } else if (!encode_field(out, field, value.get(), hash)) return false; + } + return true; +} +} + +bool meta_scalar_write(PyObject* capsule, PyObject* values, PyObject* hash, std::vector& data) { + auto* schema = static_cast(PyCapsule_GetPointer(capsule, capsule_name)); + if (!schema) return false; + data.assign(schema->size, 0); + return encode_scalars(*schema, values, hash, data.data()); +} + +bool meta_primitive_write(int kind, PyObject* items, PyObject* hash, std::vector& data) { + switch (kind) { + case 0x21: case 0x15: case 0x4a: case 0x13: case 0x11: case 0x33: break; + default: + PyErr_Format(PyExc_NotImplementedError, "Unsupported array element type %d", kind); + return false; + } + const auto count = PyTuple_Size(items); + const auto step = kind == 0x33 ? 16 : width(kind); + data.assign(checked_buffer_size(count, step), 0); + for (Py_ssize_t i = 0; i < count; ++i) + if (!encode_array_item(data.data() + i * step, kind, PyTuple_GetItem(items, i), hash)) return false; + return true; +} + PyObject* mod_meta_scalars_write(PyObject*, PyObject* args) { PyObject *capsule, *values, *hash; if (!PyArg_ParseTuple(args, "OOO", &capsule, &values, &hash)) return nullptr; @@ -159,18 +280,240 @@ PyObject* mod_meta_scalars_write(PyObject*, PyObject* args) { if (!result) return nullptr; auto* out = PyByteArray_AsString(result.get()); std::memset(out, 0, schema->size); - for (const auto& field : schema->fields) { - PyHandle value(PyObject_GetItem(values, field.name.get())); - if (!value) { - if (!PyErr_ExceptionMatches(PyExc_KeyError)) return nullptr; - PyErr_Clear(); - PyHandle alias(PyObject_GetItem(values, field.alias.get())); - if (!alias) { - if (!PyErr_ExceptionMatches(PyExc_KeyError)) return nullptr; - PyErr_Clear(); continue; + if (!encode_scalars(*schema, values, hash, out)) return nullptr; + return result.release(); +} + +namespace { +bool model_binding(PyObject* converter, int kind, Py_ssize_t count, ModelField& binding) { + if (PyTuple_Check(converter)) { + PyObject *type, *components; + if (!PyArg_ParseTuple(converter, "OO", &type, &components)) return false; + binding.vector = std::make_unique(type, components); + if (!*binding.vector) return false; + binding.scalar_record = kind == 0x4a && count < 0 && binding.vector->dimensions() == 2; + const auto dimensions = count < 0 ? (kind == 0x33 ? 3 : kind == 0x34 ? 4 : 0) : (kind == 0x21 ? count : 0); + if (!binding.scalar_record && ((dimensions != 3 && dimensions != 4) || binding.vector->dimensions() != dimensions)) + throw std::invalid_argument("META vector binding has incompatible dimensions"); + } else if (converter != Py_None && !PyCallable_Check(converter)) { + PyErr_SetString(PyExc_TypeError, "META model converter must be callable"); return false; + } + Py_INCREF(converter); + binding.converter = PyHandle(converter); + return true; +} +PyObject* convert_value(const ModelField& binding, PyObject* value) { + if (binding.vector) return binding.scalar_record ? binding.vector->scalar(value) : binding.vector->numeric(value); + if (binding.converter.get() != Py_None) return PyObject_CallFunctionObjArgs(binding.converter.get(), value, nullptr); + Py_INCREF(value); + return value; +} +} + +PyObject* mod_meta_model_new(PyObject*, PyObject* args) { + PyObject *capsule, *type, *bindings, *references = Py_None; int opaque = 0; + if (!PyArg_ParseTuple(args, "OOO|Op", &capsule, &type, &bindings, &references, &opaque)) return nullptr; + auto* schema = static_cast(PyCapsule_GetPointer(capsule, capsule_name)); + if (!schema) return nullptr; + PyHandle sequence(PySequence_Tuple(bindings)); + if (!sequence) return nullptr; + const auto count = PyTuple_Size(sequence.get()); + if (static_cast(count) != schema->fields.size()) + throw std::invalid_argument("Incorrect META model binding count"); + PyHandle names(PyList_New(count)); + if (!names) return nullptr; + std::vector fields; + fields.reserve(schema->fields.size()); + for (Py_ssize_t i = 0; i < count; ++i) { + PyObject *name, *converter; + if (!PyArg_ParseTuple(PyTuple_GetItem(sequence.get(), i), "UO", &name, &converter)) return nullptr; + Py_INCREF(name); + if (!list_take(names.get(), i, name)) return nullptr; + ModelField binding; + if (!model_binding(converter, schema->fields[i].kind, schema->fields[i].count, binding)) return nullptr; + fields.push_back(std::move(binding)); + } + Py_INCREF(capsule); Py_INCREF(type); + auto model = std::make_unique(PyHandle(capsule), schema, PyHandle(type), names.get()); + if (!model->factory) return nullptr; + model->fields = std::move(fields); + model->opaque_fields = opaque != 0; + if (references != Py_None) { + PyHandle reference_fields(PySequence_Tuple(references)); + if (!reference_fields) return nullptr; + for (Py_ssize_t i = 0; i < PyTuple_Size(reference_fields.get()); ++i) { + PyObject *name, *convert; Py_ssize_t offset; int kind, element_kind; + if (!PyArg_ParseTuple(PyTuple_GetItem(reference_fields.get(), i), "UniiO", &name, &offset, &kind, &element_kind, &convert)) return nullptr; + if (offset < 0 || !binary::contains(offset, 16, schema->size) || (kind != 0x52 && kind != 0x44)) + throw std::invalid_argument("Invalid META model reference field"); + if (element_kind != 0 && (width(element_kind) > 4 || kind != 0x52)) + throw std::invalid_argument("Invalid META model array element"); + ModelField binding; + if (!model_binding(convert, element_kind, -1, binding)) return nullptr; + Py_INCREF(name); + model->references.push_back({PyHandle(name), std::move(binding), offset, kind, element_kind}); + } + } + return owned_capsule(std::move(model), model_capsule_name, destroy_model); +} + +namespace { +struct BlockSlice { const char* data = nullptr; Py_ssize_t size = 0; }; +bool resolve_pointer(PyObject* blocks, std::uint64_t pointer, BlockSlice& result) { + const auto block_id = static_cast(pointer & 0xfff); + if (blocks == Py_None || block_id == 0 || block_id > PyTuple_Size(blocks)) return true; + char* data; Py_ssize_t size; + if (PyBytes_AsStringAndSize(PyTuple_GetItem(blocks, block_id - 1), &data, &size) < 0) return false; + const auto offset = static_cast((pointer >> 12) & 0xfffff); + result = {data + std::min(offset, size), std::max(Py_ssize_t(0), size - offset)}; + return true; +} +PyObject* read_model(Model* model, const char* data, Py_ssize_t size, PyObject* blocks, bool& pending) { + if (size < model->schema->size) throw std::invalid_argument("META structure is truncated"); + PyHandle result(model->factory.create()); + if (!result) return nullptr; + for (std::size_t i = 0; i < model->fields.size(); ++i) { + const auto& field = model->schema->fields[i]; + const auto& binding = model->fields[i]; + if (binding.vector && !binding.scalar_record) { + std::array components{}; + const auto count = binding.vector->dimensions(); + for (Py_ssize_t j = 0; j < count; ++j) components[j] = binary::load(data + field.offset + j * 4); + PyHandle converted(binding.vector->numeric(std::span(components.data(), count))); + if (!model->factory.assign(result.get(), i, converted.get())) return nullptr; + } else { + PyHandle value(decode_field(data, field)); + if (!value) return nullptr; + PyHandle converted(convert_value(binding, value.get())); + if (!model->factory.assign(result.get(), i, converted.get())) return nullptr; + } + } + pending = model->opaque_fields; + for (const auto& field : model->references) { + const auto pointer = binary::load(data + field.offset); + auto count = binary::load(data + field.offset + 8); + if (count && (pointer & 0xfff) && field.kind == 0x52 && field.element_kind == 0) { + pending = true; continue; + } + BlockSlice source; + if (!resolve_pointer(blocks, pointer, source)) return nullptr; + if (!source.data) count = 0; + PyHandle value; + if (field.kind == 0x44) { + value = PyHandle(PyUnicode_DecodeASCII(source.data ? source.data : "", std::min(count, source.size), "ignore")); + } else { + value = PyHandle(PyList_New(count)); + if (!value) return nullptr; + const auto step = count ? width(field.element_kind) : 0; + if (!binary::contains(0, count * step, source.size)) throw std::invalid_argument("META array is truncated"); + for (Py_ssize_t i = 0; i < count; ++i) { + PyHandle item(decode(source.data + i * step, field.element_kind)); + if (!item) return nullptr; + if (!list_take(value.get(), i, convert_value(field.value, item.get()))) return nullptr; } - if (!encode(out + field.offset, field.kind, alias.get(), hash)) return nullptr; - } else if (!encode(out + field.offset, field.kind, value.get(), hash)) return nullptr; + } + if (!value || PyObject_GenericSetAttr(result.get(), field.name.get(), value.get()) < 0) return nullptr; + } + return result.release(); +} +} + +PyObject* mod_meta_model_read(PyObject*, PyObject* args) { + PyObject *capsule, *raw, *blocks = Py_None; + if (!PyArg_ParseTuple(args, "OO|O", &capsule, &raw, &blocks)) return nullptr; + auto* model = static_cast(PyCapsule_GetPointer(capsule, model_capsule_name)); + if (!model) return nullptr; + PyHandle owned_blocks(blocks == Py_None ? PyTuple_New(0) : PySequence_Tuple(blocks)); + if (!owned_blocks) return nullptr; + Buffer data; + if (!data.acquire(raw)) return nullptr; + bool pending; + return read_model(model, static_cast(data.buf), data.len, owned_blocks.get(), pending); +} + +PyObject* mod_meta_struct_values(PyObject*, PyObject* args) { + PyObject *record, *fields, *prefix, *serialize; int custom; + if (!PyArg_ParseTuple(args, "OOOOp", &record, &fields, &prefix, &serialize, &custom)) return nullptr; + PyHandle names(PySequence_Tuple(fields)), result(PyDict_Copy(prefix)); + if (!names || !result) return nullptr; + for (Py_ssize_t i = 0; i < PyTuple_Size(names.get()); ++i) { + PyObject *attribute, *name; + if (!PyArg_ParseTuple(PyTuple_GetItem(names.get(), i), "UU", &attribute, &name)) return nullptr; + PyHandle value(PyObject_GetAttr(record, attribute)); + if (!value) return nullptr; + auto* object = value.get(); + const bool atomic = object == Py_None || PyBool_Check(object) || PyLong_CheckExact(object) || + PyFloat_CheckExact(object) || PyUnicode_CheckExact(object) || PyBytes_CheckExact(object) || PyTuple_CheckExact(object); + if (custom || !atomic) { + PyHandle converted(PyObject_CallFunctionObjArgs(serialize, attribute, object, nullptr)); + if (!converted) return nullptr; + value = std::move(converted); + } + if (PyDict_SetItem(result.get(), name, value.get()) < 0) return nullptr; + } + return result.release(); +} + +namespace { +PyObject* read_bound_model(PyObject* binding, const BlockSlice& source, PyObject* buffers, PyObject* context) { + PyObject *capsule, *complete, *info; + if (!PyArg_ParseTuple(binding, "OOO", &capsule, &complete, &info)) return nullptr; + auto* model = static_cast(PyCapsule_GetPointer(capsule, model_capsule_name)); + if (!model) return nullptr; + bool pending; + PyHandle value(read_model(model, source.data, source.size, buffers, pending)); + if (!value) return nullptr; + if (pending) { + PyHandle raw(PyBytes_FromStringAndSize(source.data, model->schema->size)); + if (!raw) return nullptr; + PyHandle completed(PyObject_CallFunctionObjArgs(complete, context, info, value.get(), raw.get(), nullptr)); + if (!completed) return nullptr; + } + return value.release(); +} +} + +PyObject* mod_meta_model_array_read(PyObject*, PyObject* args) { + PyObject *binding, *blocks, *context; unsigned long long pointer; Py_ssize_t stride, count; + if (!PyArg_ParseTuple(args, "OOKnnO", &binding, &blocks, &pointer, &stride, &count, &context)) return nullptr; + PyHandle buffers(PySequence_Tuple(blocks)); + if (!buffers) return nullptr; + BlockSlice source; + if (!resolve_pointer(buffers.get(), pointer, source)) return nullptr; + if (stride <= 0 || count < 0 || count > source.size / stride) + throw std::invalid_argument("META structure array is truncated"); + PyHandle result(PyList_New(count)); + if (!result) return nullptr; + for (Py_ssize_t i = 0; i < count; ++i) { + BlockSlice row{source.data + i * stride, stride}; + if (!list_take(result.get(), i, read_bound_model(binding, row, buffers.get(), context))) return nullptr; + } + return result.release(); +} + +PyObject* mod_meta_models_read(PyObject*, PyObject* args) { + PyObject *bindings, *blocks, *pointers, *context; + if (!PyArg_ParseTuple(args, "OOOO", &bindings, &blocks, &pointers, &context)) return nullptr; + PyHandle plans(PySequence_Tuple(bindings)), buffers(PySequence_Tuple(blocks)), refs(PySequence_Tuple(pointers)); + if (!plans || !buffers || !refs) return nullptr; + if (PyTuple_Size(plans.get()) != PyTuple_Size(buffers.get())) throw std::invalid_argument("META block binding count differs"); + PyHandle result(PyList_New(PyTuple_Size(refs.get()))); + if (!result) return nullptr; + for (Py_ssize_t i = 0; i < PyTuple_Size(refs.get()); ++i) { + auto* pointer_value = PyTuple_GetItem(refs.get(), i); + const auto pointer = PyLong_AsUnsignedLongLong(pointer_value); + if (PyErr_Occurred()) return nullptr; + const auto block_id = static_cast(pointer & 0xfff); + PyObject* binding = block_id && block_id <= PyTuple_Size(plans.get()) ? PyTuple_GetItem(plans.get(), block_id - 1) : Py_None; + PyHandle value; + if (binding == Py_None) { + value = PyHandle(PyObject_CallMethod(context, "_resolve_pointer_value", "O", pointer_value)); + } else { + BlockSlice source; + if (!resolve_pointer(buffers.get(), pointer, source)) return nullptr; + value = PyHandle(read_bound_model(binding, source, buffers.get(), context)); + } + if (!value || !list_take(result.get(), i, value.release())) return nullptr; } return result.release(); } diff --git a/native/resource/meta_graph_write.cpp b/native/resource/meta_graph_write.cpp new file mode 100644 index 0000000..af89eb9 --- /dev/null +++ b/native/resource/meta_graph_write.cpp @@ -0,0 +1,355 @@ +#include "resource/bindings.h" +#include +#include +#include + +namespace fivefury_py { +namespace { +namespace binary = fivefury_native::binary; +constexpr auto capsule_name = "fivefury.MetaGraph"; +struct Field { + PyHandle name, alias; + Py_ssize_t offset; + int kind, element_kind; + unsigned int reference, element_reference; +}; +struct Plan { + PyHandle scalar; + std::vector fields; +}; +void destroy_plan(PyObject* capsule) { + delete static_cast(PyCapsule_GetPointer(capsule, capsule_name)); +} +struct Recursion { + bool entered = Py_EnterRecursiveCall(" while encoding a META graph") == 0; + ~Recursion() { if (entered) Py_LeaveRecursiveCall(); } +}; +struct Block { + unsigned int name; + std::vector data; +}; +void pad(std::vector& data, std::size_t alignment) { + data.resize(data.size() + binary::alignment_padding(data.size(), alignment)); +} +std::uint64_t pointer_bits(std::size_t block, std::size_t offset) { + if (block > 0xfff || offset > 0xfffff) + throw std::overflow_error("META pointer exceeds packed block or offset limits"); + return (static_cast(offset) << 12) | block; +} +void reference(std::vector& out, std::uint64_t pointer, Py_ssize_t count = -1) { + if (count > 0xffff) throw std::overflow_error("META reference count exceeds uint16 range"); + out.assign(count < 0 ? 8 : 16, 0); + binary::store(out.data(), pointer); + if (count >= 0) { + binary::store(out.data() + 8, static_cast(count)); + binary::store(out.data() + 10, static_cast(count)); + } +} +bool unsigned_hash(PyObject* value, unsigned int& result) { + PyHandle number(PyNumber_Long(value)); + if (!number) return false; + const auto raw = PyLong_AsUnsignedLongLong(number.get()); + if (PyErr_Occurred()) return false; + if (raw > 0xffffffff) throw std::overflow_error("META structure hash exceeds uint32 range"); + result = static_cast(raw); + return true; +} +bool copy_bytes(PyObject* value, std::vector& out) { + PyHandle raw(PyObject_CallFunctionObjArgs(reinterpret_cast(&PyBytes_Type), value, nullptr)); + if (!raw) return false; + char* data; Py_ssize_t size; + if (PyBytes_AsStringAndSize(raw.get(), &data, &size) < 0) return false; + out.assign(data, data + size); + return true; +} +PyObject* field_value(PyObject* values, const Field& field) { + // Complex fields historically use membership, then get(alias), including custom Mapping overrides. + if (PyDict_CheckExact(values)) { + auto* result = PyDict_GetItemWithError(values, field.name.get()); + if (!result && !PyErr_Occurred()) result = PyDict_GetItemWithError(values, field.alias.get()); + if (PyErr_Occurred()) return nullptr; + if (!result) result = Py_None; + Py_INCREF(result); + return result; + } + const int present = PySequence_Contains(values, field.name.get()); + if (present < 0) return nullptr; + if (present) return PyObject_GetItem(values, field.name.get()); + return PyObject_CallMethod(values, "get", "O", field.alias.get()); +} +struct Graph { + PyObject *resolve, *raw_type, *mapping_type, *float_xyz, *inline_size, *hash; + unsigned int float_xyz_hash; + Py_ssize_t max_block_length; + std::vector blocks; + std::unordered_map> groups; + std::unordered_map plans; + std::unordered_set used; + + std::size_t reserve(unsigned int name, bool group = true) { + pointer_bits(blocks.size() + 1, 0); + const auto index = blocks.size(); + blocks.push_back({name, {}}); + if (group) groups[name].push_back(index); + return index; + } + std::uint64_t add(unsigned int name, std::vector data, std::size_t alignment = 16, bool group = true) { + pad(data, alignment); + if (group) { + auto& indices = groups[name]; + while (!indices.empty()) { + const auto index = indices.front(); + auto& block = blocks[index].data; + if (max_block_length <= 0 || block.size() >= static_cast(max_block_length)) { + indices.pop_front(); + continue; + } + const auto pointer = pointer_bits(index + 1, block.size()); + block.insert(block.end(), data.begin(), data.end()); + return pointer; + } + } + // Ungrouped data pointers still register their newly allocated block for subsequent grouped items. + const auto index = reserve(name); + blocks[index].data = std::move(data); + return pointer_bits(index + 1, 0); + } + Plan* plan(unsigned int name) { + auto found = plans.find(name); + if (found == plans.end()) { + PyHandle capsule(PyObject_CallFunction(resolve, "I", name)); + if (!capsule) return nullptr; + found = plans.emplace(name, std::move(capsule)).first; + } + return static_cast(PyCapsule_GetPointer(found->second.get(), capsule_name)); + } + bool target_hash(PyObject* value, unsigned int fallback, unsigned int& result) { + int instance = PyObject_IsInstance(value, raw_type); + if (instance < 0) return false; + if (instance) { + PyHandle name(PyObject_GetAttrString(value, "name_hash")); + return name && unsigned_hash(name.get(), result); + } + instance = PyObject_IsInstance(value, mapping_type); + if (instance < 0) return false; + if (instance) { + PyHandle name(PyObject_CallMethod(value, "get", "s", "_meta_name_hash")); + if (!name) return false; + const int truth = PyObject_IsTrue(name.get()); + if (truth < 0) return false; + if (truth) return unsigned_hash(name.get(), result); + } + if (fallback) { result = fallback; return true; } + PyErr_SetString(PyExc_ValueError, "Structure hash is required for pointer/inline structure encoding"); + return false; + } + bool structure(unsigned int name, PyObject* value, std::vector& out) { + Recursion recursion; + if (!recursion.entered) return false; + used.insert(name); + if (name == float_xyz_hash) { + PyHandle xyz(PyObject_CallFunctionObjArgs(float_xyz, value, nullptr)); + if (!xyz) return false; + PyHandle items(PyTuple_Pack(1, xyz.get())); + return items && meta_primitive_write(0x33, items.get(), hash, out); + } + const int raw = PyObject_IsInstance(value, raw_type); + if (raw < 0) return false; + if (raw) { + PyHandle data(PyObject_GetAttrString(value, "data")); + if (!data || !copy_bytes(data.get(), out)) return false; + pad(out, 16); + return true; + } + auto* schema = plan(name); + if (!schema || !meta_scalar_write(schema->scalar.get(), value, hash, out)) return false; + for (const auto& field : schema->fields) { + PyHandle item(field_value(value, field)); + if (!item) return false; + std::vector encoded; + if (!encode_field(field, item.get(), encoded)) return false; + if (field.offset < 0 || !binary::contains(field.offset, encoded.size(), out.size())) + throw std::invalid_argument("META complex field exceeds structure size"); + std::copy(encoded.begin(), encoded.end(), out.begin() + field.offset); + } + return true; + } + bool array(const Field& field, PyObject* value, std::vector& out) { + const int truth = PyObject_IsTrue(value); + if (truth < 0) return false; + if (!truth) { out.assign(16, 0); return true; } + PyHandle items(PySequence_Tuple(value)); + if (!items) return false; + const auto count = PyTuple_Size(items.get()); + if (!count || field.element_kind < 0) { out.assign(16, 0); return true; } + if (count > 0xffff) throw std::overflow_error("META reference count exceeds uint16 range"); + std::vector data; + unsigned int block_name; + if (field.element_kind == 0x07 || field.element_kind == 0x05) { + block_name = field.element_kind == 0x07 ? 0x07 : field.element_reference; + if (field.element_kind == 0x05) used.insert(block_name); + for (Py_ssize_t i = 0; i < count; ++i) { + auto* item = PyTuple_GetItem(items.get(), i); + unsigned int name = field.element_reference; + if (field.element_kind == 0x07 && !target_hash(item, 0, name)) return false; + std::vector payload; + if (!structure(name, item, payload)) return false; + if (field.element_kind == 0x07) { + const auto pointer = add(name, std::move(payload)); + const auto offset = data.size(); + data.resize(offset + 8); + binary::store(data.data() + offset, pointer); + } else { + // Inline array records concatenate without per-record alignment. + data.insert(data.end(), payload.begin(), payload.end()); + } + } + } else { + if (!meta_primitive_write(field.element_kind, items.get(), hash, data)) return false; + block_name = static_cast(field.element_kind); + } + reference(out, add(block_name, std::move(data)), count); + return true; + } + bool encode_field(const Field& field, PyObject* value, std::vector& out) { + switch (field.kind) { + case 0x40: { + const int truth = PyObject_IsTrue(value); + if (truth < 0) return false; + if (truth) { + PyHandle text; + if (PyUnicode_CheckExact(value)) { + text = PyHandle(PyUnicode_AsEncodedString(value, "ascii", "ignore")); + } else { + PyHandle method(PyObject_GetAttrString(value, "encode")); + PyHandle args(Py_BuildValue("(s)", "ascii")); + PyHandle kwargs(Py_BuildValue("{s:s}", "errors", "ignore")); + if (!method || !args || !kwargs) return false; + text = PyHandle(PyObject_Call(method.get(), args.get(), kwargs.get())); + } + if (!text || !copy_bytes(text.get(), out)) return false; + } + out.resize(field.reference & 0xffff); + return true; + } + case 0x50: { + if (field.element_kind >= 0 && (field.reference & 0xffff) == 0) return true; + const int truth = PyObject_IsTrue(value); + if (truth < 0) return false; + if (truth && field.element_kind >= 0) { + // The remaining inline ARRAYINFO kinds retain their raw-byte fallback contract. + PyHandle items(PySequence_List(value)); + if (!items) return false; + } + if (truth && !copy_bytes(value, out)) return false; + out.resize(field.reference & 0xffff); + return true; + } + case 0x44: { + const int truth = PyObject_IsTrue(value); + if (truth < 0) return false; + if (!truth) { out.assign(16, 0); return true; } + PyHandle text(PyObject_Str(value)); + if (!text) return false; + PyHandle raw(PyUnicode_AsEncodedString(text.get(), "ascii", "ignore")); + if (!raw || !copy_bytes(raw.get(), out)) return false; + const auto count = out.size(); + if (count > 0xffff) throw std::overflow_error("META string count exceeds uint16 range"); + out.push_back(0); + const auto pointer = add(0x10, std::move(out), 1); + reference(out, pointer, count); + return true; + } + case 0x59: { + const int truth = PyObject_IsTrue(value); + if (truth < 0) return false; + if (truth && !copy_bytes(value, out)) return false; + if (out.empty()) { out.assign(8, 0); return true; } + const auto name = field.reference != 0 && field.reference != 2 ? field.reference : 0x11; + const auto pointer = add(name, std::move(out), 1, false); + reference(out, pointer); + return true; + } + case 0x07: { + if (value == Py_None) { out.assign(8, 0); return true; } + unsigned int name; + if (!target_hash(value, field.reference, name) || !structure(name, value, out)) return false; + const auto pointer = add(name, std::move(out)); + reference(out, pointer); + return true; + } + case 0x05: { + unsigned int name = field.reference; + if (!name && !target_hash(value, 0, name)) return false; + used.insert(name); + if (value != Py_None) return structure(name, value, out); + PyHandle size(PyObject_CallFunction(inline_size, "I", name)); + if (!size) return false; + const auto length = PyLong_AsSsize_t(size.get()); + if (PyErr_Occurred()) return false; + if (length < 0) throw std::invalid_argument("Negative META structure size"); + out.assign(length, 0); + return true; + } + case 0x52: return array(field, value, out); + default: + PyErr_Format(PyExc_NotImplementedError, "Unsupported META field type %d", field.kind); + return false; + } + } +}; +} + +PyObject* mod_meta_graph_new(PyObject*, PyObject* args) { + PyObject *scalar, *fields; + if (!PyArg_ParseTuple(args, "OO", &scalar, &fields)) return nullptr; + if (!PyCapsule_IsValid(scalar, "fivefury.MetaScalars")) { + PyErr_SetString(PyExc_TypeError, "Expected a META scalar schema"); + return nullptr; + } + auto plan = std::make_unique(); + Py_INCREF(scalar); + plan->scalar = PyHandle(scalar); + PyHandle sequence(PySequence_Tuple(fields)); + if (!sequence) return nullptr; + for (Py_ssize_t i = 0; i < PyTuple_Size(sequence.get()); ++i) { + PyObject *name, *alias; + Py_ssize_t offset; + int kind, element_kind; + unsigned int target, element_target; + if (!PyArg_ParseTuple(PyTuple_GetItem(sequence.get(), i), "UUniIiI", + &name, &alias, &offset, &kind, &target, &element_kind, &element_target)) return nullptr; + Py_INCREF(name); Py_INCREF(alias); + plan->fields.push_back({PyHandle(name), PyHandle(alias), offset, kind, element_kind, target, element_target}); + } + return owned_capsule(std::move(plan), capsule_name, destroy_plan); +} + +PyObject* mod_meta_graph_write(PyObject*, PyObject* args) { + unsigned int root, xyz_hash; + PyObject *value, *resolve, *raw_type, *mapping_type, *xyz, *inline_size, *hash; + Py_ssize_t max_length; + if (!PyArg_ParseTuple(args, "IOOOOOOOnI", &root, &value, &resolve, &raw_type, &mapping_type, + &xyz, &inline_size, &hash, &max_length, &xyz_hash)) return nullptr; + Graph graph{resolve, raw_type, mapping_type, xyz, inline_size, hash, xyz_hash, max_length}; + graph.reserve(root, false); + std::vector payload; + if (!graph.structure(root, value, payload)) return nullptr; + pad(payload, 16); + graph.blocks[0].data = std::move(payload); + PyHandle blocks(PyList_New(graph.blocks.size())); + PyHandle used(PySet_New(nullptr)); + if (!blocks || !used) return nullptr; + for (std::size_t i = 0; i < graph.blocks.size(); ++i) { + const auto& block = graph.blocks[i]; + PyHandle data(PyByteArray_FromStringAndSize(block.data.data(), block.data.size())); + PyHandle name(PyLong_FromUnsignedLong(block.name)); + if (!data || !name || !list_take(blocks.get(), i, PyTuple_Pack(2, name.get(), data.get()))) return nullptr; + } + for (auto name : graph.used) { + PyHandle item(PyLong_FromUnsignedLong(name)); + if (!item || PySet_Add(used.get(), item.get()) < 0) return nullptr; + } + return PyTuple_Pack(2, blocks.get(), used.get()); +} +} // namespace fivefury_py diff --git a/pyproject.toml b/pyproject.toml index d593473..e5edb80 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta" [project] name = "fivefury" -version = "0.5.0" -description = "Python library for GTA V asset workflows including YMAP, YTYP, RPF, YTD and GameFileCache utilities." +version = "0.5.1" +description = "Python library for research and education on GTA V asset formats." readme = "README.md" requires-python = ">=3.11" authors = [ diff --git a/setup.py b/setup.py index bcc640e..0a28407 100644 --- a/setup.py +++ b/setup.py @@ -44,7 +44,7 @@ def finalize_options(self) -> None: "texture_bindings.cpp", "texture_index.cpp", ), - "resource": ("binary_document.cpp", "bindings.cpp", "layout.cpp", "meta_codec.cpp"), + "resource": ("binary_document.cpp", "bindings.cpp", "layout.cpp", "meta_codec.cpp", "meta_graph_write.cpp"), "rpf": ("archive.cpp", "bindings.cpp", "crypto.cpp", "read.cpp", "read_session.cpp", "scan.cpp"), "spatial": ("bindings.cpp",), } diff --git a/tests/archive/test_meta_rpf_contracts.py b/tests/archive/test_meta_rpf_contracts.py index f3cb601..878ef37 100644 --- a/tests/archive/test_meta_rpf_contracts.py +++ b/tests/archive/test_meta_rpf_contracts.py @@ -198,27 +198,68 @@ class MetaAndArchiveContractTests: def test_meta_builder_reuses_blocks_until_the_existing_group_reaches_the_limit( self, ) -> None: - from fivefury.meta.builder import MetaBuilder - - builder = MetaBuilder() - first = builder._add_block(305419896, bytes(16368), group=True) - second = builder._add_block(305419896, bytes(48), group=True) - assert first.block_id == second.block_id - assert len(builder.blocks) == 1 - assert len(builder.blocks[0].data) == 16416 + from fivefury.meta import MetaBuilder, MetaPointer, MetaStructInfo, RawStruct + from fivefury.meta.defs import MetaDataType + from fivefury.meta.utils import meta_field_entry + + info = MetaStructInfo(1, 0, 0, 16, [ + meta_field_entry("first", 0, MetaDataType.STRUCTURE_POINTER), + meta_field_entry("second", 8, MetaDataType.STRUCTURE_POINTER), + ]) + builder = MetaBuilder(struct_infos=[info]) + builder.build(1, { + "first": RawStruct(305419896, bytes(16368)), + "second": RawStruct(305419896, bytes(48)), + }) + first, second = (MetaPointer.from_uint64(value) + for value in struct.unpack_from(" None: - from fivefury.meta.builder import MetaBuilder + from fivefury.meta import MetaBuilder, MetaStructInfo + from fivefury.meta.defs import MetaDataType + from fivefury.meta.utils import meta_field_entry from fivefury.resource import get_resource_total_page_count - builder = MetaBuilder() - for _ in range(3): - builder._add_block(305419896, bytes(12288), group=False) - system = builder._compose_system_stream(0) + info = MetaStructInfo(1, 0, 0, 24, [ + meta_field_entry(name, index * 8, MetaDataType.DATA_BLOCK_POINTER) + for index, name in enumerate(("first", "second", "third")) + ]) + builder = MetaBuilder(struct_infos=[info]) + system = builder.build(1, { + name: bytes(12288) for name in ("first", "second", "third") + }) + assert [len(block.data) for block in builder.blocks[1:]] == [12288] * 3 pages_info = struct.unpack_from(" 1 assert pages_info[2] == get_resource_total_page_count(builder.page_flags) + @pytest.mark.parametrize("field_type, value, block_index, expected", [ + ("ARRAY_OF_BYTES", 3, 0, bytes(16)), + ("DATA_BLOCK_POINTER", 3, 1, bytes(3)), + ("ARRAY_OF_CHARS", "a\u00e9bc", 0, b"!ab" + bytes(13)), + ]) + def test_meta_builder_binary_fields_preserve_conversion_protocols( + self, field_type, value, block_index, expected, + ) -> None: + from fivefury.meta import MetaBuilder, MetaStructInfo + from fivefury.meta.defs import MetaDataType + from fivefury.meta.utils import meta_field_entry + + info = MetaStructInfo(1, 0, 0, 8, [ + meta_field_entry("data", 0, MetaDataType[field_type], ref_key=3), + ]) + class Text(str): + def encode(self, encoding="utf-8", *, errors="strict"): + return b"!" + super().encode(encoding, errors=errors) + + builder = MetaBuilder(struct_infos=[info]) + builder.build(1, {"data": Text(value) if isinstance(value, str) else value}) + assert builder.blocks[block_index].data == expected + @pytest.mark.integration def test_good_ymap_roundtrip_preserves_meta_layout_contract(self) -> None: from fivefury import read_ymap diff --git a/tests/performance/bench_perf.py b/tests/performance/bench_perf.py index c23b56a..cce6bef 100644 --- a/tests/performance/bench_perf.py +++ b/tests/performance/bench_perf.py @@ -25,7 +25,14 @@ from fivefury.cache import GameFileCache from fivefury.gamefile import guess_game_file_type from fivefury.hashing import jenk_hash -from fivefury.meta import MetaBuilder +from fivefury.meta import ( + META_TYPE_NAME_ARRAYINFO, + MetaBuilder, + MetaDataType, + MetaFieldInfo, + MetaStructInfo, + RawStruct, +) from fivefury.metahash import MetaHash from fivefury.resolver import clear_hash_resolver, register_name, resolve_hash from fivefury.rpf import create_rpf @@ -436,36 +443,31 @@ def run(): # --------------------------------------------------------------------------- -# 11. MetaBuilder._add_block — linear scan bottleneck +# 11. META graph construction and block grouping # --------------------------------------------------------------------------- class TestMetaBuilderBlockPerf: - def test_add_block_grouping_100(self, benchmark): - """Simulate adding 100 small data blocks with grouping enabled.""" + @pytest.mark.parametrize("count", [100, 500]) + def test_build_grouped_blocks(self, benchmark, count): + """Build grouped opaque records through the shared META graph writer.""" + name_hash = jenk_hash("PerfBlock") + info = MetaStructInfo(jenk_hash("PerfRoot"), 0, 0, 16, [ + MetaFieldInfo(META_TYPE_NAME_ARRAYINFO, 0, MetaDataType.STRUCTURE_POINTER, 0, 0, 0), + MetaFieldInfo(jenk_hash("entities"), 0, MetaDataType.ARRAY, 0, 0, 0), + ]) + payloads = [struct.pack("<4f", 1.0, 2.0, 3.0, float(i)) for i in range(count)] + root = {"entities": [RawStruct(name_hash, payload) for payload in payloads]} def run(): - builder = MetaBuilder(struct_infos={}, enum_infos={}) - name_hash = jenk_hash("CEntityDef") - for i in range(100): - builder._add_block( - name_hash, struct.pack("<4f", 1.0, 2.0, 3.0, float(i)) - ) - - benchmark(run) - - def test_add_block_grouping_500(self, benchmark): - """Simulate adding 500 small data blocks with grouping (O(n²) risk).""" - - def run(): - builder = MetaBuilder(struct_infos={}, enum_infos={}) - name_hash = jenk_hash("CEntityDef") - for i in range(500): - builder._add_block( - name_hash, struct.pack("<4f", 1.0, 2.0, 3.0, float(i)) - ) - - benchmark(run) + builder = MetaBuilder(struct_infos=[info]) + builder.build(info.name_hash, root) + return builder + + builder = benchmark(run) + blocks = [block for block in builder.blocks if block.name_hash == name_hash] + assert len(blocks) == 1 + assert blocks[0].data == b"".join(payloads) # --------------------------------------------------------------------------- diff --git a/tests/runtime/test_meta_model_codec.py b/tests/runtime/test_meta_model_codec.py new file mode 100644 index 0000000..99622ef --- /dev/null +++ b/tests/runtime/test_meta_model_codec.py @@ -0,0 +1,101 @@ +import dataclasses +import gc +import struct + +import pytest + +from fivefury import _native_abi3 as native +from fivefury.meta.backed import MetaBackedStruct +from fivefury.metahash import _NATIVE_HASH_BINDING, MetaHash, MetaHashFieldsMixin +from fivefury.vector import Vector3 + + +@dataclasses.dataclass(slots=True) +class DecodedRecord(MetaHashFieldsMixin): + _hash_fields = ("name",) + name: MetaHash + position: Vector3 + + +def model_plan(): + scalars = native.meta_scalars_new(16, [("hash", "hash", 0, 0x4A), ("pos", "pos", 4, 0x33)]) + return native.meta_model_new(scalars, DecodedRecord, + [("name", _NATIVE_HASH_BINDING), ("position", (Vector3, ("x", "y", "z")))]) + + +def test_model_reader_owns_schema_and_materializes_nominal_values(): + plan = model_plan() + gc.collect() + raw = struct.pack(" None: spec = importlib.util.spec_from_file_location('_native_abi3', sys.argv[1]) native = importlib.util.module_from_spec(spec) spec.loader.exec_module(native) +record = type('Record', (), {'__slots__': ('value',)}) +schema = native.meta_scalars_new(4, [('value', 'value', 0, 0x15)]) +plan = native.meta_model_new(schema, record, [('value', None)]) +assert native.meta_model_read(plan, bytes([42, 0, 0, 0])).value == 42 reader = native.rpf_reader_new(sys.argv[2], bytes(range(256)), None) assert native.rpf_reader_read(reader, 'payload.bin', 2) == (b'\\x00binary\\xff',) * 2 wav = native.awc_build_pcm_wav(bytes(64), 48000, 1, 16) diff --git a/tests/world/ymap/test_ymap_read_codec.py b/tests/world/ymap/test_ymap_read_codec.py new file mode 100644 index 0000000..bd5d053 --- /dev/null +++ b/tests/world/ymap/test_ymap_read_codec.py @@ -0,0 +1,90 @@ +import dataclasses + +import pytest + +from fivefury.map_extensions import ( + EXTENSION_TYPES_BY_META_NAME, + KNOWN_EXTENSION_TYPES, + CapsuleBoundDef, + LightAttrDef, + LightEffectExtension, + LightExtension, + ScriptChildExtension, + ScriptExtension, + VerletClothCustomBoundsExtension, + extensions_from_meta, +) +from fivefury.meta import MetaBuilder, ParsedMeta +from fivefury.meta.defs import meta_name +from fivefury.vector import Vector3 +from fivefury.ymap import EntityDef, Ymap +from fivefury.ymap.defs import YMAP_ENUM_INFOS, YMAP_STRUCT_INFOS + + +@pytest.mark.parametrize("nested_map", [False, True]) +def test_entity_codecs_preserve_generic_inspection_and_unknown_extension_payloads(nested_map): + entity = EntityDef(archetype_name="nested", guid=91) + unknown_extension = Ymap(entities=[entity]).to_meta_root() if nested_map else entity.to_meta() + source = Ymap(name="typed", entities=[EntityDef(archetype_name="outer", extensions=[unknown_extension])]) + data = source.to_bytes() + generic = ParsedMeta.from_bytes(data).decoded_root + assert isinstance(generic["entities"][0], dict) + restored = Ymap.from_bytes(data) + assert isinstance(restored.entities[0], EntityDef) + nested = restored.entities[0].extensions[0] + if nested_map: + assert isinstance(nested, dict) and nested["_meta_name"] == "CMapData" + nested = nested["entities"][0] + assert isinstance(nested, dict) and nested["_meta_name"] == "CEntityDef" + assert nested["guid"] == 91 + assert ParsedMeta.from_bytes(restored.to_bytes()).decoded_root == generic + + +@pytest.mark.parametrize("different_root", [False, True]) +def test_entity_read_uses_the_files_layout_when_known_schema_differs(different_root): + entity_hash = meta_name("CEntityDef") + original = next(info for info in YMAP_STRUCT_INFOS if info.name_hash == entity_hash) + offsets = {field.name: field.data_offset for field in original.entries} + swapped = {meta_name("guid"): offsets["tintValue"], meta_name("tintValue"): offsets["guid"]} + variant = dataclasses.replace(original, key=original.key + 1, + entries=[dataclasses.replace(field, data_offset=swapped.get(field.name_hash, field.data_offset)) + for field in original.entries]) + schemas = [variant if info.name_hash == entity_hash else info for info in YMAP_STRUCT_INFOS] + if different_root: + schemas = [dataclasses.replace(info, key=info.key + 1) if info.name_hash == meta_name("CMapData") else info for info in schemas] + source = Ymap(name="variant", entities=[EntityDef(archetype_name="test", guid=17, tint_value=29)]) + data = MetaBuilder(struct_infos=schemas, enum_infos=YMAP_ENUM_INFOS).build(meta_name("CMapData"), source.to_meta_root()) + restored = Ymap.from_bytes(data).entities[0] + assert restored.guid == 17 and restored.tint_value == 29 + first, second = ParsedMeta.from_bytes(data), ParsedMeta.from_bytes(data) + first.struct_infos[entity_hash].entries.clear() + assert len(second.struct_infos[entity_hash].entries) == len(original.entries) + + +def test_known_extension_codecs_match_generic_materialization_and_allow_edits(): + extensions = {cls: cls() for cls in KNOWN_EXTENSION_TYPES} + extensions[LightEffectExtension].instances = [LightAttrDef(posn=Vector3(1, 2, 3), colour=(10, 20, 30))] + extensions[LightEffectExtension].instances[0].colour = (1, 2, 3) + extensions[LightEffectExtension].instances[0].vol_outer_colour = (1, 0, 1) + extensions[ScriptExtension].children = [ScriptChildExtension(position=Vector3(4, 5, 6))] + extensions[VerletClothCustomBoundsExtension].collision_data = [CapsuleBoundDef(owner_name="capsule")] + data = Ymap(entities=[EntityDef(extensions=list(extensions.values()))]).to_bytes() + generic = ParsedMeta.from_bytes(data).decoded_root["entities"][0]["extensions"] + restored = Ymap.from_bytes(data) + assert restored.entities[0].extensions == extensions_from_meta(generic) + light = next(value for value in restored.entities[0].extensions if isinstance(value, LightEffectExtension)) + light.instances[0].posn = Vector3(7, 8, 9) + edited = Ymap.from_bytes(restored.to_bytes()) + assert edited.entities[0].extensions == restored.entities[0].extensions + + +def test_extension_registry_replacements_apply_after_codec_warmup(monkeypatch): + data = Ymap(entities=[EntityDef(extensions=[LightExtension()])]).to_bytes() + assert type(Ymap.from_bytes(data).entities[0].extensions[0]) is LightExtension + class CustomLight(LightExtension): + pass + monkeypatch.setitem(EXTENSION_TYPES_BY_META_NAME, LightExtension.META_NAME, CustomLight) + assert type(Ymap.from_bytes(data).entities[0].extensions[0]) is CustomLight + class CustomMap(Ymap): + pass + assert type(CustomMap.from_bytes(data)) is CustomMap