Skip to content

Report what is left in the JDBC metastore before a metadata scan truncates - #487

Merged
em3s merged 2 commits into
mainfrom
feat/cli-metastore-headroom
Aug 11, 2026
Merged

em3s merged 2 commits into
mainfrom
feat/cli-metastore-headroom

Conversation

@em3s

@em3s em3s commented Aug 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

JdbcHashLabel fetches graph.metadata-fetch-limit rows per scan key and filters inactive ones out afterwards. A metadata DELETE only tombstones, so tombstones spend that budget, and past the limit the page is truncated silently — AbstractLabel.scan computes hasNext and DdlService.getAll drops it.

EdgeIndex on v3 is the fix. Until then I needed to see how close deployed servers are, so this adds a command that measures it.

jdbc-metastore remaining walks /graph/v2/metastore/global — the only endpoint that shows the rows a scan reads, tombstones included — and counts per (src, labelId), the scan key getAll builds. Both values come off the server's decode, so nothing re-implements the codec. The endpoint is 0.4.x only, removed by #418, so the command probes first and says so.

No server change. Same tombstone problem as #485.

Notes

  • --capacity is required: it is the server's graph.metadata-fetch-limit, deployments override the OSS default of 1000, and actuator masks the value.
  • Every window prints, grouped by namespace, since a metastore table can be shared between deployments.
  • Bounded: counters per page, 500-row pages, five stop conditions (last, empty page, --max-rows, --max-pages, total reached). Whichever cap fires is named and exits non-zero. Ctrl-C reports what it counted.
  • Not using the server's prefix= count: buildQuery builds k LIKE '<prefix>%' unescaped, and base64url keys contain _.

Changes

  • Add jdbc-metastore remaining, with --capacity, --page-size, --max-pages, --max-rows
  • Add GetStream to the HTTP client, which keeps the real status code and decodes off the connection
  • Add DumpMetastore(page, size) and the dump page model
  • Register the command

How to Test

cd cli && go test ./internal/command/ && go build ./... && go vet ./...

Against a 0.4.x server. Lower both to see the classification without a large table:

actionbase --host http://<0.4.x-host>:8080 -e "jdbc-metastore remaining --capacity 3 --page-size 2"
20 rows, capacity 3/window, pages of 2 (max 2000). Ctrl-C reports partial.
alpha         14 rows  4 windows
  service      OVER by 2   alpha:origin               5/3     live 5      tomb 0
  storage      OVER by 2   alpha:origin               5/3     live 1      tomb 4
  label        1 left      alpha:talk                 2/3     live 0      tomb 2
               1 left      alpha:wish                 2/3     live 0      tomb 2
beta           5 rows  2 windows
  service      2 left      beta:origin                1/3     live 1      tomb 0
  label        OVER by 1   beta:gift                  4/3     live 2      tomb 2
1 undecodable (window unknown, e.g. k="not-an-encoded-key")
20/20 rows, 6 windows, 2 namespaces. 3 over, 0 near. An over-capacity window is already truncated and does not say so.

All of these exit non-zero, as does a window over capacity:

4/20 rows, 1 window, 1 namespace. Stopped at --max-pages 2 - incomplete, 1 over is a lower bound.
--capacity is required: the server's graph.metadata-fetch-limit, which it does not expose (actuator masks it). OSS default 1000.
No /graph/v2/metastore/global on this server (0.4.x only), so this cannot be measured here.
Unauthorized. Set --authKey or ACT_API_KEY.

Sample output and the last two paths came from a stub server; the rest ran against real 0.4.x servers.

Not touched

The root cause, which EdgeIndex addresses. Also MetastoreInspector.buildQuery's _ escaping, which is on 0.4.x and below rather than main, and CheckConnection exiting 0 on an unreachable host, which predates this and affects every command.

AI Assistance

  • This PR was written largely with AI assistance.
    • Tool / model: Claude Code (Opus 5)

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Aug 10, 2026
@em3s em3s changed the title Report how much room each metadata scan window has left Report what is left in the JDBC metastore before a metadata scan truncates Aug 10, 2026
`Get` decodes the body and, when that fails, replaces the real status with -1. An
unauthorized response has no body at all, so "this server does not serve the
endpoint" and "you may not read it here" arrive identically. A caller that has to
tell those apart - the next commit does - cannot.

`GetStream` keeps the status and decodes off the connection rather than reading
the whole body into memory first, which is what a paged dump of a large table
wants anyway.

`DumpMetastore(page, size)` reads `/graph/v2/metastore/global`, ordered by `id`
rather than `k`: `id` is the primary key, so the server's OFFSET pagination walks
an index it already has, and sorting by `k` would order by a base64 string for no
benefit.

`MetastoreDumpRow` declares only the id, the key and the server's decode of the
row. The encoded value, the property map, the timestamp and the direction are left
out so the decoder walks past them instead of allocating a string per row for data
that would be discarded.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@em3s
em3s force-pushed the feat/cli-metastore-headroom branch from 89729ac to 4eb766d Compare August 11, 2026 01:07
A metadata `DELETE` only tombstones. The row is written back with
`Active.INACTIVE` and stays, and `JdbcHashLabel.scanStorage` applies the SQL
`LIMIT` before anything filters on active, so tombstones spend the budget a live
scan needs. Past that limit the page is truncated and nothing says so:
`AbstractLabel.scan` computes `hasNext` and `DdlService.getAll` drops it building
a `DdlPage`. There was no way to ask a server how close it is.

`jdbc-metastore remaining` walks `/graph/v2/metastore/global`, the only endpoint
showing the rows a metadata scan actually reads - tombstones included, since it
does not filter on active - and counts every scan window. A window is
`(src, labelId)`, which is what `getAll` builds its scan key from and what the
limit applies to; both come off the server's own decode of each row, so nothing
here re-implements the codec.

`--capacity` is required, not defaulted. `DEFAULT_METADATA_LIMIT` is 1000, but a
deployment overrides it with `graph.metadata-fetch-limit` and the server masks the
value in actuator. Guessing the OSS default reported windows OVER that had half
their room left, with a non-zero exit, which is the worst possible answer to hand
a monitor. It is `capacity` rather than `limit` because unlike `--max-rows` and
`--max-pages` beside it, it is not a bound this command imposes but the size of
the thing being measured.

Every window prints, grouped by namespace. Sources are `namespace:service`, and a
metastore table can be shared - several deployments pointing at one table, each
seeing the others' rows - so without grouping the report leads with somebody
else's window. And no threshold decides which windows are worth showing: filtering
to the fullest of each kind hid a window holding half the capacity behind one
holding slightly more, which was the one still holding live rows.

Bounded, because the table being counted is too large to hold and the server pays
per request - the controller zips every page with `getTotalCount`, so each request
runs a full `COUNT(*)` on top of its own `LIMIT/OFFSET` query. Rows fold into
fixed-size counters per page and are dropped. Pages are 500 by default, since
server work scales with request count. Five stop conditions: `last`, an empty
page, `--max-rows`, `--max-pages` - which covers what the row cap cannot, a server
answering with short pages that never say they are last - and reading as many rows
as the table claims to hold. Whichever cap fires is named and exits non-zero, so
an incomplete census never reads as a complete one. Ctrl-C stops at the next page
and reports what it counted; nothing here writes, so there is nothing to unwind.

Not using the server's own `prefix=` count: `buildQuery` turns it into
`k LIKE '<prefix>%'` with no escaping, and the keys are base64url, whose alphabet
includes `_` - a single-character LIKE wildcard. Roughly a third of prefixes carry
one, and each makes the count match rows from other windows.

The endpoint was removed after 0.4.x, so the command probes first and says so
rather than failing obscurely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@em3s
em3s force-pushed the feat/cli-metastore-headroom branch from 4eb766d to 3df02f6 Compare August 11, 2026 01:41
@em3s

em3s commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Optimistic

@em3s
em3s merged commit fa8accc into main Aug 11, 2026
3 checks passed
@em3s
em3s deleted the feat/cli-metastore-headroom branch August 11, 2026 02:11
em3s added a commit that referenced this pull request Aug 11, 2026
#418 removed `MetastoreInspector` and its dump endpoints as dead code, which they
were - nothing called them. Something does now. `jdbc-metastore remaining` (#487)
measures how much room each metadata scan window has left, and this is the only
view of the rows such a scan actually reads.

It has to be the rows on disk, not the entities. A metadata `DELETE` rewrites its
row with `Active.INACTIVE` rather than removing it, and `JdbcHashLabel.scanStorage`
applies its `LIMIT` before anything filters on active, so a scan key crowded with
tombstones truncates while `DdlService.getAll` drops the `hasNext` that would have
said so. Every read path hides exactly what needs measuring. This is scaffolding
for the move to EdgeIndex and goes when the JDBC metastore does.

Named for that store rather than for the idea of a metastore:
`JdbcMetastoreInspector`, `JdbcMetastoreController`, `JdbcMetastoreRow`. Only the
JDBC one has scan windows that fill with tombstones, and a name that claimed the
general concept would have to be repurposed or quietly keep meaning the old thing.

Narrower than what was removed. No `prefix` filter: it built `k LIKE '<prefix>%'`
with no escaping, and the keys are base64url, an alphabet containing `_`, a
single-character LIKE wildcard, so roughly a third of prefixes matched rows from
other windows. No `sort`, because `id ASC` is the only order worth serving - it is
the primary key, so paging walks an index that already exists. No `/local`, which
has no backing store on `main` any more. The response carries `id`, `k` and four
decoded fields; the encoded value, property map, timestamp, direction and type are
dropped, and `v` is the largest column of the lot.

The request shape stays compatible with 0.4.x, which still serves the old
endpoint, so one client reads both. Blocking JDBC work goes to `boundedElastic`,
which the removed version did not do.

Tested against a running server as well as in unit tests: create a service, delete
it, and the row is still there with `active: false`, which is the whole premise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
em3s added a commit that referenced this pull request Aug 11, 2026
#418 removed `MetastoreInspector` and its dump endpoints as dead code, which they
were. Something calls them now: `jdbc-metastore remaining` (#487) measures how much
room each metadata scan window has left, and this is the only view of the rows such
a scan actually reads. A `DELETE` rewrites its row with `Active.INACTIVE` instead
of removing it, and `scanStorage` applies its `LIMIT` before anything filters on
active, so every path above this one hides exactly what needs measuring.

Scaffolding for the move to EdgeIndex, so it is named for the store it reads -
`JdbcMetastoreInspector`, not a name claiming the general concept - and goes when
that store does.

Narrower than what was removed. No `prefix`: it built `k LIKE '<prefix>%'`
unescaped over base64url keys, whose `_` is a LIKE wildcard. No `sort`: `id ASC` is
the primary key, so paging walks an index that already exists. No `/local`: no
backing store for it on `main`. The row is `id`, `k` and four decoded fields.

The request shape still matches 0.4.x, so one client reads both. Blocking JDBC work
goes to `boundedElastic`, which the removed version did not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
em3s added a commit that referenced this pull request Aug 11, 2026
#418 removed `MetastoreInspector` and its dump endpoints as dead code, which they
were. Something calls them now: `jdbc-metastore remaining` (#487) measures how much
room each metadata scan window has left, and this is the only view of the rows such
a scan actually reads. A `DELETE` rewrites its row with `Active.INACTIVE` instead
of removing it, and `scanStorage` applies its `LIMIT` before anything filters on
active, so every path above this one hides exactly what needs measuring.

Scaffolding for the move to EdgeIndex, so it is named for the store it reads -
`JdbcMetastoreInspector`, not a name claiming the general concept - and goes when
that store does.

Narrower than what was removed. No `prefix`: it built `k LIKE '<prefix>%'`
unescaped over base64url keys, whose `_` is a LIKE wildcard. No `sort`: `id ASC` is
the primary key, so paging walks an index that already exists. No `/local`: no
backing store for it on `main`. The row is `id`, `k` and four decoded fields.

The request shape still matches 0.4.x, so one client reads both. Blocking JDBC work
goes to `boundedElastic`, which the removed version did not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant