HashMark digest index, glyph discovery indexes (containers, creators, media, wildcard search), REST rate-limit exemptions - #45
Open
cdonnachie wants to merge 5 commits into
Open
Conversation
Digest lookup is the one HashMark operation no existing Radiant
infrastructure can serve: blockchain.ref.get indexes Glyph refs rather
than data-output payloads, and every other scanner in this tree is gated
on its own magic, so a HashMark output leaves no trace in the DB. That
gap is what forced HashMark to consider running a second process to
follow blocks, handle reorgs and track confirmations — all of which
RXinDexer already does.
Detection is a 10-byte prefix compare against every output, cheap enough
to run unconditionally; the overwhelming majority of OP_RETURNs cost one
memcmp and are skipped. Rows live in a single 'HMd' keyspace, keyed
(algorithm, digest, height, txid, vout) so a prefix scan emerges in
height-ascending order and the API returns oldest first for free. Each
row stores the block hash it was mined in, which is what makes reorg
unwind exact rather than height guesswork.
Both record versions are indexed. v1 is 3-4 pushes ending in an optional
label; v2 adds a 20-byte committed signer and a 65-byte signature after
the digest, which moves the label from push 3 to push 5. That move is
exactly why an unknown version must never be read under a known
version's rules — a v1 parser let loose on a v2 record would index a
signature as a label — so anything outside the supported set is reported
as UNKNOWN_VERSION and skipped. The stored value is version-led, so a
row written before v2 existed reads back the old way and no rescan is
needed.
The signature is deliberately NOT verified here. Doing so needs
secp256k1 and the chain's genesis hash, and would change nothing:
HashMark re-fetches every hit and checks the signature against the
committed signer itself. A hit means only "this script is on chain at
this height", never that the digest describes what anyone claims — so a
bug here can cause a MISSED result but never a false one.
Exposed as hashmark.lookup / hashmark.stats over ElectrumX and as
GET /hashmark/{digest} and /hashmark/stats over REST. Digests are
matched whole: no prefix or partial matching, which would let a caller
enumerate the index one nibble at a time.
39 tests cover the parser against the spec vectors, both versions'
shapes and caps, the storage round trip including a pre-v2 row, lookup
ordering, and reorg unwind.
Note: the security middleware allowlist gains /containers and /users
alongside /hashmark in this commit. The three paths are one contiguous
block and splitting them would leave this commit's own routes blocked;
the container routes themselves arrive in the next commit.
Two derived indexes, and the API surface that makes them useful.
GCM (container_ref + member_ref) answers "what is in this container",
and GMT (type_hash + ref) answers "which tokens declare this metadata
type" — the latter is what makes user profiles listable, since a user is
a token whose metadata type is 'user' rather than a distinct kind of
thing.
Both are pure functions of the CBOR metadata already stored in GM rows,
so the v4->v5 and v5->v6 migrations backfill in place: they walk existing
GT rows, re-read each token's stored metadata, and write keys derived
from it. No radiantd rescan and no block reprocessing, which is what
keeps a schema bump from meaning a full reindex. Both share one driver
(_migrate_metadata_derived) and are idempotent, so a run interrupted
before the version stamp is safe to repeat.
The walk is paged through _read_token_page, which fully drains its
iterator before returning so a caller's write_batch never overlaps an
open RocksDB iterator — the same constraint _migrate_3_to_4 observes.
v5/v6 were developed as v4/v5 against a v3 base, before the recency
indexes took v4 upstream; they are renumbered here so the version chain
stays linear.
Also fixes token classification: CONTAINER is now checked before
AUTHORITY, so a container+authority token ([2,7,10]) classifies the same
way here as in get_token_type rather than differently depending on which
path reached it.
Exposed as glyph_list_containers, glyph_get_container_members and
glyph_list_users over ElectrumX, and as GET /containers,
/containers/{ref}, /containers/{ref}/members and /users over REST.
Covered by tests/test_glyph_schema_migrations.py: both migrations derive
the same keys the live path writes, CBOR tags are unwrapped, malformed
and non-string fields are skipped rather than indexed, tokens without
metadata are ignored, and re-running changes nothing.
REST_RATE_LIMIT_EXEMPT lists networks that bypass the per-IP limiter, with `private` as shorthand for the usual RFC1918 / loopback / link-local / ULA set. An internal caller — a dashboard, a paired application on the same network — is not the traffic the limiter exists to shed, and giving it a bucket only means the limiter runs out of room for callers that matter. The match is against the *resolved* client address, not the immediate peer, so a public client arriving through a private-IP reverse proxy is still limited. Unparseable tokens are skipped rather than failing startup: a typo in an environment variable should cost one exemption, not the whole service. Also fixes db_engine reporting in /health/db and /status, which read the attribute off the db object where it lives on db.env (see db.py, which uses self.env.db_engine) — so both had been reporting 'unknown' rather than the configured engine. The mock_db fixture had the same mistake, setting db_engine directly on the mock, which is why the wrong reporting went unnoticed. Corrected, or the fix would surface as three RecursionErrors when FastAPI tried to serialise an auto-created Mock attribute. Covered by tests in tests/server/test_rest_api.py: the allowlist defaults to empty, an exempt internal caller creates no bucket at all while a public one still does, a public client behind a private proxy is still limited, and unparseable tokens are ignored.
test_parser_halt_guard builds its BlockProcessor with __new__, bypassing __init__, and sets every overlay index to None by hand so the test covers the pure UTXO/ref core. hashmark_index was added to the real __init__ but not to that list, so advance_txs reached `if self.hashmark_index:` and raised AttributeError. The production path was never affected — __init__ sets the attribute unconditionally, to None when the index is disabled. Only a stub that skips __init__ could miss it.
… indexes
Three discovery gaps, and the two schema steps that close them.
v7 adds GA (creator_ref + work_ref) from the metadata `by` field. Nothing
in the tree read `by` before: the `author` field was serialized ('au'),
returned by the API and never once assigned, so it always read back null.
Populated now on both paths. Attribution is SELF-ASSERTED — `by` is
written by whoever minted the token, and nothing binds it to the
referenced token's owner — so get_creator_works answers "which tokens
claim this creator", never "which tokens this creator made", and says so
in the response.
v8 adds GMH (media sha256 + ref) and GDH (metadata_hash + ref). GMH keys
on a SINGLE sha256 deliberately: that is what a remote `h` already is, so
an embedded copy and a remote copy of one artwork land on the same key
and collide in one index. Embedded media had no content hash on the token
row at all — embedded_data_hash was only ever set from a remote `h` — so
it is computed and stored now, matching the index key exactly. Remote and
embed are extracted independently here, unlike the mutually-exclusive
token-field path, or a token carrying both would be findable by only one
hash.
Both steps backfill in place from stored CBOR through the existing
_migrate_metadata_derived driver, which gains the metadata_hash argument
GDH needs (a derived key may want the payload's identity rather than its
contents). No radiantd rescan.
Prefixes avoid the aliasing trap twice. 'GBY' for creators would sit
under BALANCE's 'GB', whose scans seek on GB + hashX(11); 'GPH' for
payloads would sit under BY_PROTO's 'GP', which is prefix-scanned as
GP + proto(1). 'GA' and 'GDH' collide with nothing, and a test asserts
it. GMH does sit under METADATA's 'GM', but safely: every GM access is an
exact get and nothing prefix-scans it.
Wildcard search needs no index and gets none. BY_NAME stores
sha256(name), so the stored name is one-way and admits exact equality
only — no ordered index over hashed keys can answer a prefix, let alone
an infix. So `*`/`?`/`[abc]` route to a bounded full scan over GT rows
covering BOTH name and ticker, with bare text treated as a substring.
The scan runs to completion rather than stopping at offset+limit: rows
arrive in ref order while results are alphabetical, so an early break
would sort whichever matches came first and page through a different set
each call. MAX_WILDCARD_SCAN bounds it, and responses carry
scanned/truncated so a capped answer is never mistaken for a complete
one.
media_sha256 and media_duplicates ride on detail responses only, behind
include_media_dupes. The count is a GMH prefix scan, which on a 500-row
list page would be 500 scans; a client uses media_duplicates > 1 as the
trigger to fetch the full list rather than probing per page view. The
count stops at 51 and flags media_duplicates_capped so "50+" can be
rendered without claiming a wrong exact figure. The hash is emitted with
.hex(), not hash_to_hex_str: that reverses byte order for txid display
and would produce a string no client-computed sha256(file) could match.
Exposed as glyph.get_creator_works over ElectrumX and as
GET /creators/{ref}/works, /media/{sha256}/glyphs and
/media/payload/{sha256}/glyphs over REST, all shaped like the existing
members endpoint so one row parser consumes them all.
The migration-chain test now derives from CURRENT_SCHEMA_VERSION instead
of hard-coding it, so bumping the schema without adding a migrator fails
the test rather than shipping another hard-fail to production.
47 tests: both backfills derive the keys the live path writes, embed and
remote forms of one file collide, all three CBORTag spellings agree,
double-hashing would break the collision, wildcard semantics and stable
paging, and the capped count.
cdonnachie
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three independent changes, one commit each, in dependency order.
1. HashMark digest index (
ba89072)Digest lookup is the one HashMark operation no existing Radiant infrastructure
can serve:
blockchain.ref.getindexes Glyph refs rather than data-outputpayloads, and every other scanner in this tree is gated on its own magic, so a
HashMark output leaves no trace in the DB. That gap is what would otherwise
force HashMark to run a second process duplicating block-following, reorg
handling and confirmation tracking.
Detection is a 10-byte prefix compare against every output. Rows live in one
HMdkeyspace keyed(algorithm, digest, height, txid, vout), so a prefix scanemerges height-ascending and the API returns oldest first for free. Each row
stores the block hash it was mined in, which makes reorg unwind exact.
Both record versions are indexed. v1 is 3–4 pushes ending in an optional
label; v2 adds a 20-byte committed signer and a 65-byte signature after the
digest, moving the label from push 3 to push 5. That move is exactly why an
unknown version must never be read under a known version's rules — a v1 parser
let loose on a v2 record would index a signature as a label. The stored value is
version-led, so rows written before v2 existed read back unchanged and no rescan
is needed.
The signature is deliberately not verified here: that needs secp256k1 and the
chain's genesis hash, and would change nothing, because HashMark re-fetches every
hit and checks the signature against the committed signer itself. A hit means
"this script is on chain at this height" and nothing more — so a bug here can
cause a missed result, never a false one.
Exposed as
hashmark.lookup/hashmark.statsover ElectrumX andGET /hashmark/{digest}//hashmark/statsover REST. Digests match whole: noprefix matching, which would let a caller enumerate the index a nibble at a time.
39 tests: parser vectors, both versions' shapes and caps, storage round trip
including a pre-v2 row, lookup ordering, reorg unwind.
One wrinkle worth flagging in review: the security middleware allowlist gains
/containersand/usersalongside/hashmarkin this commit. The three pathsare one contiguous block, and splitting them would leave this commit's own routes
blocked; the container routes arrive in commit 2.
2. Glyph container and metadata-type indexes (
7ef4700)GCManswers "what is in this container",GMTanswers "which tokens declarethis metadata type" — the latter is what makes user profiles listable, since a
user is a token whose metadata type is
userrather than a distinct kind.Both are pure functions of CBOR metadata already stored in
GMrows, so thev4→v5 and v5→v6 migrations backfill in place: no radiantd rescan, no block
reprocessing. They share one driver and are idempotent, so a run interrupted
before the version stamp is safe to repeat. The walk pages through
_read_token_page, which fully drains its iterator before returning so acaller's
write_batchnever overlaps an open RocksDB iterator.v5/v6 were developed as v4/v5 against a v3 base, before the recency indexes took
v4 upstream; they are renumbered so the chain stays linear.
Also fixes token classification:
CONTAINERis checked beforeAUTHORITY, so acontainer+authority token (
[2,7,10]) classifies the same way here as inget_token_type.3. REST rate-limit exemptions, and a db_engine fix (
55cd96c)REST_RATE_LIMIT_EXEMPTlists networks that bypass the per-IP limiter, withprivateas shorthand for the usual RFC1918 / loopback / link-local / ULA set.The match is against the resolved client address, not the immediate peer, so
a public client arriving through a private-IP reverse proxy is still limited.
Unparseable tokens are skipped rather than failing startup.
/health/dband/statusreaddb_engineoff the db object, where it actuallylives on
db.env(db.pyusesself.env.db_engine), so both had been reportingunknown. Themock_dbfixture had the same mistake, which is why it wentunnoticed — corrected here, since otherwise the fix surfaces as three
RecursionErrors when FastAPI tries to serialise an auto-created Mock attribute.
Testing
tests/test_hashmark_index.py39 passed ·tests/test_glyph_schema_migrations.pypassed ·
tests/server/test_rest_api.py102 passed.One failure remains,
TestTokenAnalyticsEndpoints::test_get_metadata, and it ispre-existing — it fails identically at
ca8a6a4, before any of this work.Not addressed here.
The HashMark index is also verified against mainnet: transaction
345565eb…88892(v1) anda1a86ab4…045916(v2, signed) are both indexed andreturned by digest lookup, and both re-verify against the chain.