Skip to content

Independent recipe-data validation (89.3% exact match) + the dated column is dropped, so pre-1.19 legacy recipes are craftable #189

Description

@pretema

Context

I independently extracted a 1.0 crafting-recipe table from Wayback Machine
captures of the original 1.0 Lodestone (2,080 recipes, current-era only —
methodology in the attached RECONCILIATION.md), then reconciled it against
common/sql/seed/042_gamedata_recipes.sql (which is project-meteor-server
PR #73's gamedata_recipes.sql, already imported here — I checked, this
repo already has the fuller dataset, so this isn't a data-contribution ask,
it's a validation report + one finding worth a look).

Not proposing a bulk data PR — this repo's recipe table is already a
superset of what I extracted. Filing as an issue instead, per the two things
below.

1. Validation: 89.3% byte-exact agreement from an independent source

Matching by result item id against the 1,824 keys both datasets share: 1,628
(89.3%) match exactly — same class/bracket/materials/crystals — despite
being derived from two unrelated sources (yours is presumably capture-derived
from the live 1.0 servers; mine is a Lodestone scrape). That's a meaningful
correctness signal for the seed data already in this repo. Full breakdown,
including the ~46 genuine disagreements (disagreements.csv, attached), is
in RECONCILIATION.md §2.

One disagreement I chased down and fixed on my own data: グリフォンレザー
(Gryphon Leather) was misidentified as catalogID 10007124 (a blank-named
[en] row — see the companion issue on that) instead of the correct
10007208 ("Hippogryph Leather"). See RECONCILIATION.md §4 for the full
chain of evidence. Mentioning it here because your seed data already has the
correct 10007208 mapping — this is just confirmation your data is right
where mine was initially wrong, which is itself a useful data point.

2. Actionable finding: the dated column is discarded at load, so legacy recipes are live

gamedata_recipes.dated splits the 5,384 rows roughly evenly:
2,827 dated=1 (I believe these are pre-1.19 legacy recipes — see
below) vs. 2,557 dated=0 (current/1.23b-era). The 1.0 Lodestone's
recipe search hides dated=1 rows by default (there's a "show old
recipes" toggle in the original JP UI) — which is presumably why my
Lodestone-sourced extraction only ever surfaced the dated=0 side: measured
against just the current-era subset, my data and yours agree on 98% of
result items (1,790/1,825), not the ~38% the raw totals suggest — see
RECONCILIATION.md's addendum for the corrected math.

I checked Database::load_recipes in map-server/src/database.rs — the
SELECT explicitly lists columns and dated isn't one of them:

let mut stmt = c.prepare(
    r"SELECT id, craftedItem, craftedQuantity, job,
             crystal0ID, crystal0Quantity, crystal1ID, crystal1Quantity,
             material0, material1, material2, material3,
             material4, material5, material6, material7
      FROM gamedata_recipes
      ORDER BY craftedItem ASC",
)?;

(kind, facilities, and both subSkill* pairs are dropped too — that
part matches PR #73's own C# Recipe.cs, which never populated them
either, so it's an inherited gap, not something this port introduced.)

Net effect: every recipe in the table is loaded into the live
ingredient-driven resolver regardless of dated
, including the ~2,827
rows that (if my read of the Lodestone UI behavior is right) represent
recipes that predate whatever era this project is targeting. If 1.23b
authenticity matters here, this is worth a decision: filter dated=1 at
load, or confirm it's fine to leave them craftable (maybe they were still
valid in 1.23b and only hidden from the search UI, not actually removed
from the server — I don't have a way to confirm which from Lodestone data
alone, it only tells me what players could browse, not what the server
still accepted).

Supporting artifacts

The full reconciliation report (RECONCILIATION.md — methodology,
match-quality breakdown, the グリフォンレザー/10007124 correction chain,
and the dated-flag investigation) and disagreements.csv (all 196
non-exact-match rows, both sides' raw values) are ready to share — happy to
post them however works best for this repo (gist link, attached to a
follow-up comment, or a PR adding them under docs/).

Happy to help chase down any of the ~46 genuine disagreements or the
dated question further if useful — I have the full Lodestone capture set
and the reconciliation tooling.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Garlemald ServerFor work units related to Garlemald Server.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions