Skip to content

Export captured traffic as HAR and Markdown - #68

Merged
Twinsen81 merged 2 commits into
mainfrom
sarajevo-v2/twi-271
Oct 2, 2026
Merged

Twinsen81 merged 2 commits into
mainfrom
sarajevo-v2/twi-271

Conversation

@Twinsen81

Copy link
Copy Markdown
Owner

Summary

A transaction left Lustro only as JSON from the API, as a cURL command, or as plain text, one at a time. This adds the two formats that matter for sharing: HAR, which browser devtools and most HTTP tools import and agents parse without instructions, and Markdown, for a bug report, a pull request, or a chat.

HAR on the server. GET /api/v1/network/transactions/_/export?format=har&ids=a,b returns a HAR 1.2 document of the transactions in ids, or of every one when ids is left out, oldest first. It is built from the store, so it has the redacted values of the detail.

Transaction HAR
startedAt startedDateTime, ISO 8601 in UTC with milliseconds
durationMs time and timings.wait; send and receive are 0, the other phases -1
protocol httpVersion (h2), empty when unknown
headers, URL query headers, queryString (decoded)
request body postData; bytes in base64 with _encoding: base64
response body content.text; bytes in base64 with encoding: base64
requestBodyBytes, responseBodyBytes bodySize, -1 when null; content.size is the larger of the byte count and the size kept, since a gzip body is kept inflated
Location redirectURL
error response._error, where Chrome's own export puts it
isMocked, categories, truncation flags, responseComplete, error, id _lustro

_resourceType (fetch, or image) tells Chrome DevTools where to list the entry. Without it, DevTools listed every JSON response as a script.

CLI. lustro net export --har FILE [--ids ID ...] writes the document, or prints it for --har -. --ids takes short ids, and the command sends them in batches of 50 and joins the results, because the server rejects a request line and headers over 8 KB. It warns about ids that the app no longer has.

Console. Select adds a checkbox column, a header checkbox that selects every row the filters show (also those past the 200 rendered), and a bar with the count, Export HAR, and Copy Markdown. A filter change keeps only the selected rows that it still shows. Export HAR fetches the same batches and saves lustro-<date>-<time>.har. Copy Markdown fetches each detail, four at a time, and copies one document. The detail gets a Markdown button next to cURL and Copy.

A transaction in Markdown is the method and URL as a heading (one code span, so a URL can't become markup), a meta line, the headers in http blocks, and each body in a block tagged with a language for its content type. JSON is indented without changing a value. A fence is longer than any run of backticks in the body. A body that capture cut short says _Truncated: Lustro kept only the first part of this body._; a body kept as bytes, or in an encoding Lustro does not decode, says that instead of a block.

Contract. The OpenAPI document describes the route and the HAR shape (HarDocument, HarEntry). The golden fixture export-har.json has five entries: a POST with a JSON body, a mock, a GIF in base64, a truncated response, and a failure. A Kotlin test checks that the export writes exactly that fixture, the CLI tests validate it against the schema, and the CLI end-to-end run exports the sample's mocked request.

Decisions to check

  1. The route goes into protocol 1.2, which is not released, as the body route and captureFilter did. _meta still says 1.2. If 1.2 counts as released, this needs a bump to 1.3.
  2. An unknown id is left out, not a 404. A selection can outlive an eviction, and the export should still save what remains. The CLI warns, and the console's toast says how many were no longer captured.
  3. ids= with no value is an empty selection, not every transaction, so a client that sends an empty selection never gets the whole store.
  4. The export is built in memory. With the default 50 MB capture budget full, a single export of everything is a large string on the device. The CLI and the console send ids in batches of 50, which bounds each response; a plain curl without ids gets it all at once.
  5. postData._encoding is our own field: HAR 1.2 has encoding on content only. Tools that ignore it see base64 text for a binary request body.
  6. Markdown is oldest first and pretty-prints JSON, which makes a large body larger: the 250 KB echo is about 780 K characters. It copies as is, with no size cap.
  7. Copy Markdown of a selection writes a ClipboardItem with a promise when the page is a secure context, because Safari allows a copy only during the click and the details arrive later. It falls back to the text copy elsewhere.

Verification

On a physical device (Android 17), with the debug sample's traffic: GET, POST, a 500, JSON, XML, a JPEG, a PNG request body, the 250 KB JSON echo, a br response, and a 2 s timeout. The console ran in headless Chrome, driven over CDP with real mouse events.

  • Select all: 10 of 10 selected. Turning off the 5xx filter: 9 of 9 selected, and the header checkbox stays checked. Unchecking one row: 8 of 9, header indeterminate. Leaving Select mode clears it.
  • Export HAR saved lustro-20261002-121122.har. lustro net export --ids with the same 9 ids wrote the same bytes (cmp).
  • The DevTools frontend that Chrome serves imported the console's file, the CLI's export of all 10 transactions, and the golden fixture through the Network panel's own import path (onLoadFromFile), with no HAR load failure and no console message. It listed every request with its method, status, media type, protocol, and duration; JSON under Fetch, the JPEG under Img with its base64 bytes decoded, and the 250 KB echo with its 262,144 characters.
  • The selection's Markdown (8 transactions, without the 780 K-character echo) rendered through GitHub's GFM render API with 8 headings, 17 sections, 7 rules, and json, xml, and http highlighting; the JPEG, PNG, and br notes rendered as italics.
  • The detail's Markdown of the 250 KB echo starts with the request heading and marks the truncated response.
  • lustro net export --har on the device: 10 transactions, 735 KB, with _lustro.responseBodyTruncated on the echo and error: "Canceled" with status 0 on the timeout.

Tests:

  • ./gradlew :lustro:testDebugUnitTest :lustro:lintDebug detekt apiCheck checkFacadeParity passes, with NetworkHarExportTest (12 tests) and HarExportGoldenTest.
  • pytest in lustro-cli: 238 pass, 13 of them new in test_cli_export.py, plus schema checks of the golden HAR.
  • node --test lustro/src/test/js/*.test.js: 100 pass, 14 of them new in network-export.test.js for the Markdown and the batch merge.

Type of change

  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (public API or wire protocol)
  • Documentation / tooling only

Checklist

  • Tests added or updated for the change
  • ./gradlew apiCheck passes (no unintended public API changes; dump updated if intended)
  • ./gradlew detekt passes
  • CHANGELOG.md updated under [Unreleased]
  • Public API changes are documented with KDoc (none)
  • Wire-protocol or schema changes are reflected in wire-protocol/ and tab OpenAPI fragments
  • All commits are signed off (DCO): git commit -s

Compatibility notes

Additive: a new route, within the unreleased protocol 1.2. No Kotlin API change.

A transaction left Lustro only as JSON from the API, as a cURL command,
or as plain text, one at a time. HAR is the archive format that browser
devtools and most HTTP tools import, and Markdown is what people paste
into a bug report, a pull request, or a chat.

GET network/transactions/_/export?format=har&ids=... returns the
transactions in ids, or every one when ids is left out, as a HAR 1.2
document, oldest first. It is built from the store, so headers, the URL,
and text bodies are redacted as in the detail. An id that the app no
longer has is left out. Each entry spends its durationMs in
timings.wait, with send and receive at 0 and the other phases at -1. A
body kept as bytes is base64: content.encoding for a response, and
postData._encoding for a request, since HAR gives postData no encoding
field. _resourceType tells Chrome DevTools to list the entry under
Fetch/XHR or Img; without it, a JSON response is listed as a script.
_lustro has the transaction id, isMocked, categories, both truncation
flags, responseComplete, and error. The route is part of protocol 1.2,
which is not released yet.

lustro net export --har FILE [--ids ID ...] saves the document, or
writes it to stdout for -. The ids can be short ids, and the command
sends them in batches of 50, because the server takes a request line and
headers of up to 8 KB. It warns about ids that the app no longer has.

In the Network tab, Select adds a checkbox column, a header checkbox
that selects every row the filters show, and a bar with the count,
Export HAR, and Copy Markdown. A filter change keeps only the selected
rows that it still shows. Export HAR fetches the selection in the same
batches and saves one file, the same bytes that the CLI writes for the
same ids. Copy Markdown fetches each detail and copies one document; the
detail's new Markdown button copies one transaction. A transaction is
the method and URL as a heading, the headers in http blocks, and each
body in a block tagged with a language for its content type, with JSON
indented without changing a value. A fence is longer than any run of
backticks in the body. A body that capture cut short, kept as bytes, or
did not decode says so.

The OpenAPI document describes the route and the HAR shape, the golden
fixture export-har.json shows five kinds of entry, and a unit test
checks that the export writes exactly that fixture. The CLI end-to-end
run exports the sample's mocked request.

Verified on a physical device with the debug sample and the console in
headless Chrome: select all, a filter that drops the 500 (10 of 10 to
9 of 9 selected), and Export HAR, which saved the same bytes as
lustro net export --ids for those ids. The DevTools Network panel that
Chrome serves imported that file, the CLI's export of all 10, and the
golden fixture through its own import path, with no load error and no
console message. The selection's Markdown rendered through GitHub's GFM
API with its headings and json, xml, and http blocks, and the detail's
Markdown of the 250 KB echo marked the truncated response.

Signed-off-by: Evgenii Plokhov <plokhov@gmail.com>
Copy Markdown read the selection in reverse list order. The list is in
the order captures were stored, and with a full capture backlog a new
capture is stored before older queued ones, so the document could have
two requests out of order. The selection is now sorted by startedAt;
the HAR export already sorted its entries.

A body kept as bytes is not included in the Markdown, and its note now
also says when capture kept only the first part of it.

Signed-off-by: Evgenii Plokhov <plokhov@gmail.com>
@Twinsen81
Twinsen81 marked this pull request as ready for review October 2, 2026 10:35
@Twinsen81
Twinsen81 merged commit 7a65bd9 into main Oct 2, 2026
8 checks passed
@Twinsen81
Twinsen81 deleted the sarajevo-v2/twi-271 branch October 2, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant