Skip to content

Serve the JDBC metastore's rows again, for as long as it exists - #488

Merged
em3s merged 1 commit into
mainfrom
feat/restore-metastore-dump
Aug 11, 2026
Merged

em3s merged 1 commit into
mainfrom
feat/restore-metastore-dump

Conversation

@em3s

@em3s em3s commented Aug 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

#418 removed MetastoreInspector and its dump endpoints as dead code, which they were. Something calls them now: jdbc-metastore remaining (#487) measures how much room each metadata scan window has left, and this is the only view of the rows such a scan actually reads. A DELETE rewrites its row with Active.INACTIVE instead of removing it, and scanStorage applies its LIMIT before anything filters on active, so every path above this one hides exactly what needs measuring.

Scaffolding for the move to EdgeIndex. It exists to see where deployments stand until the JDBC metastore is gone, and goes with it — hence JdbcMetastore* rather than a name claiming the general concept.

Narrower than what was removed:

  • No prefix. It built k LIKE '<prefix>%' unescaped over base64url keys, whose _ is a single-character LIKE wildcard, so roughly a third of prefixes matched other windows' rows.
  • No sort. id ASC is the primary key, so paging walks an index that already exists. The parameter is accepted and ignored.
  • No /local. No backing store for it on main.
  • The row is id, k and four decoded fields. v, the property map, timestamp, direction and type are dropped.

The request shape still matches 0.4.x, so one client reads both. Blocking JDBC work goes to boundedElastic, which the removed version did not do.

Changes

  • Add JdbcMetastoreInspector with dump(limit, offset) and count()
  • Add JdbcMetastoreController serving GET /graph/v2/metastore/global
  • Register the bean and re-allow the path in the read-only filter

How to Test

./gradlew build
./gradlew :server:bootRun --args='--spring.profiles.active=local'
curl -XPOST localhost:8080/graph/v2/service/demo -H 'Content-Type: application/json' -d '{"desc":"demo"}'
curl 'localhost:8080/graph/v2/metastore/global?page=0&size=3&sort=id,asc'
{"id": 2, "k": "d2--KjRsb2NhbDpvcmlnaW4AK-m6YsMpfQ:3464656d6f00",
 "decoded": {"active": true, "src": "local:origin", "tgt": "demo", "labelId": 1773822659}}

Deactivate then delete that service: the row stays, with active: false, and the table is no shorter. That is the premise, and a test pins it.

The CLI reads it unchanged:

actionbase --host http://localhost:8080 -e "jdbc-metastore remaining --capacity 1000"
2 rows, capacity 1000/window, pages of 500 (max 2000). Ctrl-C reports partial.
local          2 rows  1 window
  service      998 left    local:origin               2/1000  live 2      tomb 0
2/2 rows, 1 window, 1 namespace. 0 over, 0 near.

Not touched

The COUNT(*) per page, which keeps the response a Spring Page and compatible with 0.4.x. The table is small enough that it does not matter.

The root cause — scanStorage still cannot filter on active in SQL, and getAll still discards hasNext. EdgeIndex is where that gets fixed.

AI Assistance

  • This PR was written largely with AI assistance.
    • Tool / model: Claude Code (Opus 5)

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Aug 11, 2026
@em3s
em3s force-pushed the feat/restore-metastore-dump branch 2 times, most recently from 2836f45 to d0db78c Compare August 11, 2026 02:59
#418 removed `MetastoreInspector` and its dump endpoints as dead code, which they
were. Something calls them now: `jdbc-metastore remaining` (#487) measures how much
room each metadata scan window has left, and this is the only view of the rows such
a scan actually reads. A `DELETE` rewrites its row with `Active.INACTIVE` instead
of removing it, and `scanStorage` applies its `LIMIT` before anything filters on
active, so every path above this one hides exactly what needs measuring.

Scaffolding for the move to EdgeIndex, so it is named for the store it reads -
`JdbcMetastoreInspector`, not a name claiming the general concept - and goes when
that store does.

Narrower than what was removed. No `prefix`: it built `k LIKE '<prefix>%'`
unescaped over base64url keys, whose `_` is a LIKE wildcard. No `sort`: `id ASC` is
the primary key, so paging walks an index that already exists. No `/local`: no
backing store for it on `main`. The row is `id`, `k` and four decoded fields.

The request shape still matches 0.4.x, so one client reads both. Blocking JDBC work
goes to `boundedElastic`, which the removed version did not do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@em3s
em3s force-pushed the feat/restore-metastore-dump branch from d0db78c to 17ff789 Compare August 11, 2026 04:22
@em3s

em3s commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Optimistic

@em3s
em3s merged commit 282d042 into main Aug 11, 2026
2 checks passed
@em3s
em3s deleted the feat/restore-metastore-dump branch August 11, 2026 04:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant