feat(format): read and write sparse structural pages

[Xuanwo]

Part of #7750

Depends on #7752.
Wire contract discussion: #7631. The validity-polarity text and vote remain a merge gate.

This PR freezes the Lance 2.3 SparseLayout wire contract and lands its safe, selective reader together with an explicit writer so the contract is exercised end to end. The reader rejects SparseLayout in pre-2.3 files, validates metadata, domains, semantic sets, descriptors, buffer ranges, compression headers, chunk sums, and complete chunk consumption, caches normalized plans, projects range/take selections through nested layers, reads only intersecting value chunks, and rebuilds no-value selections without value I/O. SparseLayout adds no format-specific size or descriptor-complexity quotas for otherwise representable buffers; callers retain responsibility for resource policy.

The writer can emit sparse pages only when field metadata explicitly sets lance-encoding:structural-encoding=sparse for a Lance 2.3 file. It does not add automatic selection or change the default encoding policy. Lance 2.3 is unstable, so tests generate pages at runtime and round-trip them; this PR contains no checked-in compatibility fixture or generated test artifact.

Coverage includes nullable primitive and struct, list/large-list/map/fixed-size-list, null versus empty lists, both validity polarities, all semantic position/count representations, scan, range, take, deeply nested structures, no-value selections, selective value I/O, malformed pages, large representable buffers/descriptors, and pre-2.3 rejection.

Validation:

• cargo fmt --all -- --check
• cargo check -p lance-encoding -p lance-file --features protoc
• protoc --descriptor_set_out=/dev/null --proto_path=protos protos/encodings_v2_1.proto protos/file2.proto
• cargo test -p lance-encoding -p lance-file (470 + 93 passed; 0 failed)
• focused sparse reader/writer and malformed-page tests
• uv run mkdocs build
• cargo clippy --all --tests --benches -- -D warnings

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: b6e931cc-7e4e-4692-9191-26dca09ce566

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch xuanwo/sparse-stack-3-sparse-reader

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[westonpace]

Some design questions before diving into implementation. Marking "changes requested" so we don't lose track.

[westonpace]

A few minor suggestions. I think the only significant suggestion is the cascading encoding suggestion but that would be a more substantial rework. Still, it is something to consider, either now or as a potential follow-up. Using cascading encoding might speed up a few niche cases but, more importantly, it simplifies the encoding (we don't have so many little micro-choices) and also, by forcing ourselves to use cascading encoding we force ourselves to make sure the block compression path can handle things like delta and range encoding which could give us speedups on non-sparse paths too.