Repository navigation
Report what is left in the JDBC metastore before a metadata scan truncates - #487
Merged
Merged
Conversation
`Get` decodes the body and, when that fails, replaces the real status with -1. An unauthorized response has no body at all, so "this server does not serve the endpoint" and "you may not read it here" arrive identically. A caller that has to tell those apart - the next commit does - cannot. `GetStream` keeps the status and decodes off the connection rather than reading the whole body into memory first, which is what a paged dump of a large table wants anyway. `DumpMetastore(page, size)` reads `/graph/v2/metastore/global`, ordered by `id` rather than `k`: `id` is the primary key, so the server's OFFSET pagination walks an index it already has, and sorting by `k` would order by a base64 string for no benefit. `MetastoreDumpRow` declares only the id, the key and the server's decode of the row. The encoded value, the property map, the timestamp and the direction are left out so the decoder walks past them instead of allocating a string per row for data that would be discarded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
em3s
force-pushed
the
feat/cli-metastore-headroom
branch
from
August 11, 2026 01:07
89729ac to
4eb766d
Compare
A metadata `DELETE` only tombstones. The row is written back with `Active.INACTIVE` and stays, and `JdbcHashLabel.scanStorage` applies the SQL `LIMIT` before anything filters on active, so tombstones spend the budget a live scan needs. Past that limit the page is truncated and nothing says so: `AbstractLabel.scan` computes `hasNext` and `DdlService.getAll` drops it building a `DdlPage`. There was no way to ask a server how close it is. `jdbc-metastore remaining` walks `/graph/v2/metastore/global`, the only endpoint showing the rows a metadata scan actually reads - tombstones included, since it does not filter on active - and counts every scan window. A window is `(src, labelId)`, which is what `getAll` builds its scan key from and what the limit applies to; both come off the server's own decode of each row, so nothing here re-implements the codec. `--capacity` is required, not defaulted. `DEFAULT_METADATA_LIMIT` is 1000, but a deployment overrides it with `graph.metadata-fetch-limit` and the server masks the value in actuator. Guessing the OSS default reported windows OVER that had half their room left, with a non-zero exit, which is the worst possible answer to hand a monitor. It is `capacity` rather than `limit` because unlike `--max-rows` and `--max-pages` beside it, it is not a bound this command imposes but the size of the thing being measured. Every window prints, grouped by namespace. Sources are `namespace:service`, and a metastore table can be shared - several deployments pointing at one table, each seeing the others' rows - so without grouping the report leads with somebody else's window. And no threshold decides which windows are worth showing: filtering to the fullest of each kind hid a window holding half the capacity behind one holding slightly more, which was the one still holding live rows. Bounded, because the table being counted is too large to hold and the server pays per request - the controller zips every page with `getTotalCount`, so each request runs a full `COUNT(*)` on top of its own `LIMIT/OFFSET` query. Rows fold into fixed-size counters per page and are dropped. Pages are 500 by default, since server work scales with request count. Five stop conditions: `last`, an empty page, `--max-rows`, `--max-pages` - which covers what the row cap cannot, a server answering with short pages that never say they are last - and reading as many rows as the table claims to hold. Whichever cap fires is named and exits non-zero, so an incomplete census never reads as a complete one. Ctrl-C stops at the next page and reports what it counted; nothing here writes, so there is nothing to unwind. Not using the server's own `prefix=` count: `buildQuery` turns it into `k LIKE '<prefix>%'` with no escaping, and the keys are base64url, whose alphabet includes `_` - a single-character LIKE wildcard. Roughly a third of prefixes carry one, and each makes the count match rows from other windows. The endpoint was removed after 0.4.x, so the command probes first and says so rather than failing obscurely. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
em3s
force-pushed
the
feat/cli-metastore-headroom
branch
from
August 11, 2026 01:41
4eb766d to
3df02f6
Compare
Contributor
Author
|
Optimistic |
1 task done
em3s
added a commit
that referenced
this pull request
Aug 11, 2026
#418 removed `MetastoreInspector` and its dump endpoints as dead code, which they were - nothing called them. Something does now. `jdbc-metastore remaining` (#487) measures how much room each metadata scan window has left, and this is the only view of the rows such a scan actually reads. It has to be the rows on disk, not the entities. A metadata `DELETE` rewrites its row with `Active.INACTIVE` rather than removing it, and `JdbcHashLabel.scanStorage` applies its `LIMIT` before anything filters on active, so a scan key crowded with tombstones truncates while `DdlService.getAll` drops the `hasNext` that would have said so. Every read path hides exactly what needs measuring. This is scaffolding for the move to EdgeIndex and goes when the JDBC metastore does. Named for that store rather than for the idea of a metastore: `JdbcMetastoreInspector`, `JdbcMetastoreController`, `JdbcMetastoreRow`. Only the JDBC one has scan windows that fill with tombstones, and a name that claimed the general concept would have to be repurposed or quietly keep meaning the old thing. Narrower than what was removed. No `prefix` filter: it built `k LIKE '<prefix>%'` with no escaping, and the keys are base64url, an alphabet containing `_`, a single-character LIKE wildcard, so roughly a third of prefixes matched rows from other windows. No `sort`, because `id ASC` is the only order worth serving - it is the primary key, so paging walks an index that already exists. No `/local`, which has no backing store on `main` any more. The response carries `id`, `k` and four decoded fields; the encoded value, property map, timestamp, direction and type are dropped, and `v` is the largest column of the lot. The request shape stays compatible with 0.4.x, which still serves the old endpoint, so one client reads both. Blocking JDBC work goes to `boundedElastic`, which the removed version did not do. Tested against a running server as well as in unit tests: create a service, delete it, and the row is still there with `active: false`, which is the whole premise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
em3s
added a commit
that referenced
this pull request
Aug 11, 2026
#418 removed `MetastoreInspector` and its dump endpoints as dead code, which they were. Something calls them now: `jdbc-metastore remaining` (#487) measures how much room each metadata scan window has left, and this is the only view of the rows such a scan actually reads. A `DELETE` rewrites its row with `Active.INACTIVE` instead of removing it, and `scanStorage` applies its `LIMIT` before anything filters on active, so every path above this one hides exactly what needs measuring. Scaffolding for the move to EdgeIndex, so it is named for the store it reads - `JdbcMetastoreInspector`, not a name claiming the general concept - and goes when that store does. Narrower than what was removed. No `prefix`: it built `k LIKE '<prefix>%'` unescaped over base64url keys, whose `_` is a LIKE wildcard. No `sort`: `id ASC` is the primary key, so paging walks an index that already exists. No `/local`: no backing store for it on `main`. The row is `id`, `k` and four decoded fields. The request shape still matches 0.4.x, so one client reads both. Blocking JDBC work goes to `boundedElastic`, which the removed version did not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
em3s
added a commit
that referenced
this pull request
Aug 11, 2026
#418 removed `MetastoreInspector` and its dump endpoints as dead code, which they were. Something calls them now: `jdbc-metastore remaining` (#487) measures how much room each metadata scan window has left, and this is the only view of the rows such a scan actually reads. A `DELETE` rewrites its row with `Active.INACTIVE` instead of removing it, and `scanStorage` applies its `LIMIT` before anything filters on active, so every path above this one hides exactly what needs measuring. Scaffolding for the move to EdgeIndex, so it is named for the store it reads - `JdbcMetastoreInspector`, not a name claiming the general concept - and goes when that store does. Narrower than what was removed. No `prefix`: it built `k LIKE '<prefix>%'` unescaped over base64url keys, whose `_` is a LIKE wildcard. No `sort`: `id ASC` is the primary key, so paging walks an index that already exists. No `/local`: no backing store for it on `main`. The row is `id`, `k` and four decoded fields. The request shape still matches 0.4.x, so one client reads both. Blocking JDBC work goes to `boundedElastic`, which the removed version did not do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JdbcHashLabelfetchesgraph.metadata-fetch-limitrows per scan key and filters inactive ones out afterwards. A metadataDELETEonly tombstones, so tombstones spend that budget, and past the limit the page is truncated silently —AbstractLabel.scancomputeshasNextandDdlService.getAlldrops it.EdgeIndex on v3 is the fix. Until then I needed to see how close deployed servers are, so this adds a command that measures it.
jdbc-metastore remainingwalks/graph/v2/metastore/global— the only endpoint that shows the rows a scan reads, tombstones included — and counts per(src, labelId), the scan keygetAllbuilds. Both values come off the server's decode, so nothing re-implements the codec. The endpoint is 0.4.x only, removed by #418, so the command probes first and says so.No server change. Same tombstone problem as #485.
Notes
--capacityis required: it is the server'sgraph.metadata-fetch-limit, deployments override the OSS default of 1000, and actuator masks the value.last, empty page,--max-rows,--max-pages, total reached). Whichever cap fires is named and exits non-zero. Ctrl-C reports what it counted.prefix=count:buildQuerybuildsk LIKE '<prefix>%'unescaped, and base64url keys contain_.Changes
jdbc-metastore remaining, with--capacity,--page-size,--max-pages,--max-rowsGetStreamto the HTTP client, which keeps the real status code and decodes off the connectionDumpMetastore(page, size)and the dump page modelHow to Test
Against a 0.4.x server. Lower both to see the classification without a large table:
All of these exit non-zero, as does a window over capacity:
Sample output and the last two paths came from a stub server; the rest ran against real 0.4.x servers.
Not touched
The root cause, which EdgeIndex addresses. Also
MetastoreInspector.buildQuery's_escaping, which is on 0.4.x and below rather thanmain, andCheckConnectionexiting 0 on an unreachable host, which predates this and affects every command.AI Assistance