Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ release_dist_*/

references/
docs/
/dotnet/
examples/
.claude/
fivefury-*/
Expand Down
19 changes: 17 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,26 @@ The changelog is release-oriented and uses a small fixed set of categories:

## [Unreleased]

## [0.5.1] - 2026-09-12

### Added

- Batch construction through `Vector2.from_rows`, `Vector3.from_rows`, `Vector4.from_rows`, and `Quaternion.from_rows`, preserving nominal types and immutable, independent values.
- Configurable zlib compression levels in `ResourceSections.to_bytes()` and `build_rsc7()`.
- Immutable YED evaluators snapshot expression selection and skeleton defaults for repeated typed evaluation with explicit per-actor variable state.
- Compiled YCD clip samplers preserve the public phase/time evaluation interface while sharing immutable animation plans.
- Immutable compiled YCD animation samplers retain owned channel data for repeated native evaluation without stale caches after source edits.
- Per-track quaternion encoding overrides allow explicit facial rotations while keeping body and camera packing unchanged.

### Changed

- Version 0.5.1 is the final planned feature release of the current Python implementation; new core development moves to .NET.
- The README presents the library for research and education, with concise installation instructions, runnable examples, and a format overview.
- YMAP, YTYP, and YDR writers use zlib level 6 to balance serialization speed and output size; compressed bytes and sizes may differ, while the general resource builder retains its level-9 default.

### Fixed

- META root blocks stay separate from child blocks of the same structure type, preventing child data from being overwritten during graph construction.
- YED program caches detect in-place operand and skeleton-default edits instead of relying on object identities and list lengths.
- Explicit quaternion authoring stores all four static components instead of discarding W and reconstructing it from rounded XYZ values.
- YCD precision diagnostics distinguish integer-frame and subframe component errors, report their frame locations and include quaternion angular error independently of component limits.
Expand All @@ -24,8 +35,12 @@ The changelog is release-oriented and uses a small fixed set of categories:

### Performance

- Up to 2x faster repeated YED evaluation with compiled evaluators, avoiding repeated expression resolution and Python input/output conversions.
- Up to 14x faster repeated YCD sampling with compiled animation samplers, keeping typed outputs and cached-quaternion interpolation order.
- YMAP reads materialize nominal Python records through schema-checked native bindings, and writing uses the shared native META graph encoder. A measured 2,000-entity case improved by a further 5.6x for reading and 1.8x for writing over the preceding native field-codec implementation; results vary by workload. Unknown layouts and custom extension registrations retain generic conversion.
- Up to 2.9x faster YTYP reading and 9x faster writing with native hash-field assignment and fewer Python conversions.
- Faster YDR reading with lower memory usage by constructing typed vectors directly in C++ without intermediate tuple lists.
- YDR, YDD, and YFT writing reuses float64 position buffers and calculates bounds once per save; vectorized transforms make rigid YDR models write up to 3.4x faster.
- Up to 2x faster repeated YED evaluation with compiled evaluators that avoid repeated expression resolution.
- Up to 14x faster repeated YCD sampling by reusing immutable native animation plans.

## [0.5.0] - 2026-09-07

Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,4 @@ include LICENSE
recursive-include native *.cpp *.h
recursive-include fivefury/ydr *.xml
recursive-include tests *.py *.md
prune dotnet
253 changes: 34 additions & 219 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,253 +1,68 @@
# FiveFury

FiveFury is a Python toolkit for authoring, inspecting, validating, converting, and packaging GTA V assets. It exposes typed models and declarative builders over the game's binary resource formats, metadata containers, and RPF archives without hiding the lower-level data needed for advanced workflows.
A Python library for **research and education** around GTA V asset formats.

FiveFury is designed for tools that need to do more than convert a single file:

- Read, modify, and rebuild assets while preserving data that is not yet modeled.
- Create maps, collisions, navigation data, drawables, fragments, animations, and archives from Python objects.
- Target GTA V Legacy or Enhanced where their binary layouts differ.
- Validate resource pointers, packed limits, ownership, and format-specific invariants before writing.
- Index a game installation and resolve assets, hashes, textures, parent dictionaries, and map dependencies.
- Run expensive hashing, resource-layout, geometry, collision, and archive operations through the bundled native extension.
FiveFury provides typed models and binary readers and writers for studying resource structures, inspecting their relationships, and experimenting with data. It supports GTA V Legacy and Enhanced, with coverage depending on the format and edition.

## Installation

```bash
pip install fivefury
python -m pip install fivefury
```

FiveFury requires Python 3.11 or newer.

Static mesh import uses Trimesh and accepts file paths, bytes, `trimesh.Trimesh`, and `trimesh.Scene` objects. Supported source formats depend on Trimesh; call `supported_mesh_formats()` to inspect them at runtime.

## Supported workflows

| Area | Formats | Current coverage |
| --- | --- | --- |
| Drawables | `YDR`, `YDD`, `YTD` | Models and LODs, shader groups and parameters, samplers, embedded textures, lights, bounds, skeletons, skinning, drawable dictionaries, and texture dictionaries |
| Fragments and physics | `YFT` | Fragment drawables, damaged states, physics LODs/groups/children, contextual collision bounds, ownership validation, mass and inertia calculation, breakable glass, and environment cloth |
| World placement | `YMAP`, `YTYP` | Entities, physics dictionaries, MLO instances and definitions, archetypes, rooms, portals, entity sets, car generators, occluders, timecycle modifiers, and LOD lights |
| Streaming metadata | `YMF`, `GTXD`, `gta5_cache_y.dat` | Map/type dependencies, MLO registration, texture-dictionary parent chains, runtime PSO validation, and cache generation from in-memory or loose assets |
| Collision | `YBN` | Primitive, composite, geometry, and BVH bounds; materials, octants, MLO room IDs, and collision generation from triangle meshes |
| Navigation | `YND`, `YNV` | Road nodes and links, area partitioning, junction heightmaps, navmesh sectors/polygons/portals, typed network validation, in-memory cell builders, and Trimesh conversion |
| World data | `heightmap.dat`, `water.xml` | Quantized height grids, row RLE, water masks and queries, water surfaces, wave quads, and calming regions |
| Animation | `YCD`, `YED` | Skeletal, object, UV, camera, root-motion, and bone-scale tracks; clip dictionaries; expression dictionaries and spring data |
| Cutscenes | `.cut` | Binary cutscene read/write, declarative scene and audio authoring, validation, and YCD section generation |
| Audio and text | `AWC`, `REL`, `GXT2` | Audio containers and common codecs, typed audio metadata graphs, synth/curve/category records, and hashed text tables |
| Packaging | `RPF7`, DLC metadata | PC archive creation, nested archives, folder/ZIP conversion, standalone resource extraction, encrypted archive reading, generated DLC metadata, and typed CUT audio registration |
| Generic metadata | `YMT`, META, PSO, RBF | Typed known roots, generic binary containers, PSCH enums, and preservation of unknown schemas or payloads during supported rewrites |
| Console assets | `CDR`, PS3 `RPF7` | Read-only PS3 drawable decoding and automatic PS3 archive detection/extraction |
Requires **Python 3.11+**. Prebuilt wheels are available for Windows x64. Building the native extension from source requires a C++20 compiler.

Additional discovery support is available for embedded `YPT` texture dictionaries. `GameFileCache` can index `YWR` and `YVR`, but FiveFury does not yet expose dedicated parsers for them. `YFD`, `YPDB`, and `MRF` are not implemented.
## Examples

### Legacy and Enhanced
These examples create their own sample data.

Target-aware APIs use `GameTarget` instead of loose version labels:
### Create a small map

```python
from fivefury import GameTarget, YdrGen9Shader, trimesh_to_ydr

trimesh_to_ydr(
"source/prop.glb",
"stream/prop.ydr",
game=GameTarget.GTA5_ENHANCED,
shader=YdrGen9Shader.DEFAULT,
)
```

The target changes runtime headers, resource versions, bounds, shader metadata, and vertex layouts only where the format requires it. Readers infer the edition from the asset when possible. Legacy remains the default.
from fivefury import Vector3, Ymap

Current target-aware authoring covers the main `YDR`, `YDD`, `YFT`, `YBN`, `YCD`, `YED`, `YND`, and `YNV` paths. Support is format-specific rather than a blanket claim that every GTA V file differs between editions.

## Quick start

### Build and package a map

```python
from fivefury import Vector3, Ymap, create_rpf

ymap = Ymap(name="example_map")
ymap = Ymap(name="example")
ymap.entity(
"prop_tree_pine_01",
position=Vector3(100.0, 200.0, 0.0),
lod_dist=150.0,
)
ymap.physics_dictionary("example_map")
ymap.car_gen("sultan", Vector3(110.0, 205.0, 0.0), heading=90.0)
ymap.save("example_map.ymap", auto_extents=True)

archive = create_rpf("example_pack.rpf")
archive.file("stream/example_map.ymap", ymap)
archive.save("example_pack.rpf")
```

Factories such as `entity(...)` and `car_gen(...)` append the new object to the owning `Ymap`. Prebuilt objects are inserted directly through the corresponding typed collection, for example `ymap.entities.append(entity)`.

### Assemble an Enhanced vehicle DLC

```python
from fivefury import VehiclePackBuilder

pack = VehiclePackBuilder(
"example_cars",
vehicles_meta,
handling_meta,
variations_meta,
carcols_meta,
)
pack.vehicle("example_car", vehicle_yft, textures=vehicle_ytd)
result = pack.save("build")
```

The builder cross-validates the four metadata documents, registers their runtime file types, packages streamed YFT/YTD assets in a nested RPF, rereads the complete DLC, and only then atomically writes `build/example_cars/dlc.rpf`.

### Build a drawable from memory

```python
from fivefury import YdrMeshInput, YdrShader, create_ydr

ydr = create_ydr(
name="example_drawable",
shader=YdrShader.DEFAULT,
meshes=[
YdrMeshInput(
positions=[
(0.0, 0.0, 0.0),
(1.0, 0.0, 0.0),
(0.0, 1.0, 0.0),
],
indices=[0, 1, 2],
texcoords=[[(0.0, 0.0), (1.0, 0.0), (0.0, 1.0)]],
)
],
material_textures={"DiffuseSampler": "example_diffuse"},
)
ydr.save("example_drawable.ydr")
```

`read_ydr(...)` returns an editable asset with material, texture, bound, light, skeleton, and skinning helpers. Render geometry can also be converted into an embedded collision bound with `ydr.ensure_bound_from_render_geometry()`.

`YdrSkeleton` owns its `bones` list: construction and assignment copy the input
collection, but share its bone objects. Edits through the original input list do
not change the skeleton. Use `skeleton.bones.append(...)`, slice assignment,
`reverse()`, `sort()`, or other ordinary list operations to edit membership and
order. Direct bone `name`, `tag`, and `parent_index` assignments invalidate every
owning collection's lookup indexes; duplicate names (case-insensitive) and tags
resolve to the first bone in current list order. Appends update valid indexes
incrementally. Other relevant mutations trigger one lazy rebuild, not a scan on
every lookup. These lookup updates do not renumber bones or normalize bindings;
`skeleton.build()` still finalizes hierarchy indices and sibling links.

For low-level preparation, use `fivefury.ydr.prepare.select_layout` and
`fivefury.ydr.prepare.normalize_materials` instead of the private builder aliases
`_select_layout` and `_normalize_materials`. Replace `fivefury.ydr.prepare.resolve_shader`
with `fivefury.ydr.shaders.resolve_shader_reference`.

### Skin vertices on the GPU

```python
from fivefury import GpuShaderLanguage, GpuSkinning

gpu = GpuSkinning(
positions,
blend_indices,
blend_weights,
normals=normals,
position=Vector3(100, 200, 30),
)

# Upload these immutable streams when the mesh is created.
positions = gpu.streams.positions
influences = gpu.streams.influences
normals = gpu.streams.normals

# Reuse the palette and upload only its contents each frame.
palette = gpu.palette(bone_count)
gpu.pack_palette(skinning_matrices, output=palette)

shader = gpu.compute_shader(GpuShaderLanguage.GLSL)
groups = gpu.dispatch_groups()
ymap.save("example.ymap", auto_extents=True)
```

`GpuSkinning` packs four bone indices and normalized weights into eight bytes per vertex and exposes specialized GLSL or HLSL vertex and compute sources. Compute output remains GPU-resident; renderers should bind the returned streams and palette using `GpuSkinningBindings` and avoid reading deformed vertices back to Python.

### Generate collision
### Read, inspect and edit it

```python
from fivefury import (
BoundMaterial,
BoundMaterialType,
build_bound_from_triangles,
save_ybn,
)
from fivefury import Vector3, Ymap

triangles = [
((0.0, 0.0, 0.0), (4.0, 0.0, 0.0), (0.0, 4.0, 0.0)),
((4.0, 0.0, 0.0), (4.0, 4.0, 0.0), (0.0, 4.0, 0.0)),
]
ymap = Ymap.from_path("example.ymap")
entity = ymap.entities[0]
print(entity.archetype_name, entity.position)

bound = build_bound_from_triangles(
triangles,
material=BoundMaterial(type=BoundMaterialType.CONCRETE),
)
save_ybn(bound, "floor_collision.ybn")
entity.position = Vector3(110, 200, 30)
ymap.save("edited.ymap", auto_extents=True)
```

The builder chunks oversized geometry, builds BVHs and octants, and validates packed limits before serialization. The same bounds model is shared by standalone `YBN`, embedded `YDR` collision, MLO collision, and `YFT` physics.

### Build the runtime map cache from loose assets

```python
from fivefury import build_gta5_cache_y_from_directory

cache = build_gta5_cache_y_from_directory("build/stream")
cache.save("build/gta5_cache_y.dat")
```

The builder reads loose `YMAP`, `YTYP`, and `YBN` files, derives map and interior records, validates the result, and writes the binary cache atomically.

### Index a game installation

```python
from fivefury import GameFileCache

cache = GameFileCache("path/to/Grand Theft Auto V")
cache.scan_game(use_index_cache=True)

asset = cache.get_asset("prop_tree_pine_01", kind=".ydr")
cache.extract_asset(asset, "out/prop_tree_pine_01.ydr")
cache.extract_asset_textures(asset, "out/textures")
```

`GameFileCache` scans loose files and nested archives, performs lazy typed loading, resolves names and hashes, follows `YTD`/`GTXD` texture relationships, and supplies the dependency context used by map-manifest tooling.

## API conventions

FiveFury keeps the authoring layer close to the data model:

- Typed collections use ordinary `append(...)` and `extend(...)` operations; there is no generic dispatcher that guesses the destination from a runtime type.
- Singular noun factories such as `entity(...)`, `bone(...)`, `light(...)`, and `car_gen(...)` construct, register, and return one domain object.
- One-to-one relationships use assignment, such as `ydr.bound = collision`; verbs are reserved for real operations such as `ensure_*`, `derive_*`, `normalize_*`, `bind_*`, and `resolve_*`.
- `AssetRef`, `AssetSet`, and `BuildContext` provide one typed path for resolving dependencies between assets and selecting Legacy or Enhanced behavior.
- `ValidationReport` carries stable diagnostic codes, severity, asset, and field paths instead of relying on unstructured output.
- `build()` derives normalized state, `validate()` inspects it, and `save()` validates and performs atomic binary serialization.
- Stable game-side values use enums for targets, shaders, LODs, flags, render masks, materials, and track formats.
- Core writers use atomic replacement and reject known invalid references, ownership, pointers, or packed ranges before replacing the destination.

Local development documentation is kept in `docs/`, which is excluded from version control.
For experiments in memory, use `ymap.to_bytes()` and `Ymap.from_bytes(data)`. The `fivefury.meta` module exposes META schemas, data blocks and decoded fields for inspecting the underlying representation.

## Scope and guarantees
## Scope

FiveFury aims for runtime-compatible binary output, but GTA V formats contain edition-specific and asset-specific structures. Passing validation proves the modeled binary invariants; it is not a substitute for testing newly authored content in the target game.
| Area | Formats |
| --- | --- |
| Maps and world data | YMAP, YTYP, YMF, GTXD, `gta5_cache_y.dat`, `heightmap.dat`, `water.xml` |
| Geometry and textures | YDR, YDD, YFT, YTD |
| Collision and navigation | YBN, YND, YNV |
| Animation and scenes | YCD, YED, `.cut` |
| Audio and text | AWC, REL, GXT2 |
| Archives and metadata | RPF7, DLC metadata, YMT, META, PSO, RBF |

`YFT`, `REL`, `YED`, and `YMT` expose substantial read/write functionality, but not every runtime subtype is modeled semantically. Unknown metadata is preserved where the container supports lossless rewriting instead of being guessed. PS3 `CDR` and RPF support is currently focused on reading and extraction, while PC RPF7 supports authoring.
Support includes reading, writing and validation for the modeled structures. Some variants remain partially understood, particularly within YFT, REL, YED and YMT. Console CDR and PS3 RPF support is focused on reading and extraction.

## License
Use `GameTarget` where an API exposes edition-specific behavior. Validation checks the implemented binary contracts; newly authored assets still need verification in the target game.

FiveFury is released under [The Unlicense](LICENSE).
## Development

See [CHANGELOG.md](CHANGELOG.md) for release history and compatibility notes.
**0.5.1 is the last planned feature release of the current Python implementation.** Core development is moving to .NET.

## Credits
The [source code](https://github.com/Hancapo/fivefury/tree/main/fivefury) and [tests](https://github.com/Hancapo/fivefury/tree/main/tests) provide further examples and format-specific contracts. See the [changelog](https://github.com/Hancapo/fivefury/blob/main/CHANGELOG.md) for release history.

Thomas Vanini — GTA V Enhanced PT-BR dubbing project creator, workflow coordinator, and in-game runtime validation — https://linktr.ee/ThomasVanini
Released under [The Unlicense](https://github.com/Hancapo/fivefury/blob/main/LICENSE).
19 changes: 17 additions & 2 deletions fivefury/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,10 @@
import tempfile
from collections.abc import Callable, Mapping
from contextlib import ExitStack
from enum import IntEnum
from enum import IntEnum, IntFlag
from functools import lru_cache
from pathlib import Path
from typing import Any, TypeAlias
from typing import Any, TypeAlias, TypeVar

from .hashing import jenk_hash
from .metahash import MetaHash
Expand Down Expand Up @@ -143,13 +144,27 @@ def _missing_(cls, value: object) -> FlexibleIntEnum:
return member


_IntEnumT = TypeVar("_IntEnumT", bound=IntEnum | IntFlag)


@lru_cache(maxsize=1024)
def _int_enum_member(enum_type: type[_IntEnumT], value: int) -> _IntEnumT:
return enum_type(value)


def coerce_int_enum(enum_type: type[_IntEnumT], value: int | _IntEnumT) -> _IntEnumT:
"""Reuse immutable enum values while bounding retention of unknown integers."""
return value if isinstance(value, enum_type) else _int_enum_member(enum_type, int(value))


__all__ = [
"ByteSource",
"FlexibleIntEnum",
"JsonReport",
"atomic_write_bytes",
"atomic_write_files",
"clip_short_name",
"coerce_int_enum",
"dataclass_init_values",
"hash_value",
"read_source_bytes",
Expand Down
Loading