Export captured traffic as HAR and Markdown - #68
Merged
Merged
Conversation
A transaction left Lustro only as JSON from the API, as a cURL command, or as plain text, one at a time. HAR is the archive format that browser devtools and most HTTP tools import, and Markdown is what people paste into a bug report, a pull request, or a chat. GET network/transactions/_/export?format=har&ids=... returns the transactions in ids, or every one when ids is left out, as a HAR 1.2 document, oldest first. It is built from the store, so headers, the URL, and text bodies are redacted as in the detail. An id that the app no longer has is left out. Each entry spends its durationMs in timings.wait, with send and receive at 0 and the other phases at -1. A body kept as bytes is base64: content.encoding for a response, and postData._encoding for a request, since HAR gives postData no encoding field. _resourceType tells Chrome DevTools to list the entry under Fetch/XHR or Img; without it, a JSON response is listed as a script. _lustro has the transaction id, isMocked, categories, both truncation flags, responseComplete, and error. The route is part of protocol 1.2, which is not released yet. lustro net export --har FILE [--ids ID ...] saves the document, or writes it to stdout for -. The ids can be short ids, and the command sends them in batches of 50, because the server takes a request line and headers of up to 8 KB. It warns about ids that the app no longer has. In the Network tab, Select adds a checkbox column, a header checkbox that selects every row the filters show, and a bar with the count, Export HAR, and Copy Markdown. A filter change keeps only the selected rows that it still shows. Export HAR fetches the selection in the same batches and saves one file, the same bytes that the CLI writes for the same ids. Copy Markdown fetches each detail and copies one document; the detail's new Markdown button copies one transaction. A transaction is the method and URL as a heading, the headers in http blocks, and each body in a block tagged with a language for its content type, with JSON indented without changing a value. A fence is longer than any run of backticks in the body. A body that capture cut short, kept as bytes, or did not decode says so. The OpenAPI document describes the route and the HAR shape, the golden fixture export-har.json shows five kinds of entry, and a unit test checks that the export writes exactly that fixture. The CLI end-to-end run exports the sample's mocked request. Verified on a physical device with the debug sample and the console in headless Chrome: select all, a filter that drops the 500 (10 of 10 to 9 of 9 selected), and Export HAR, which saved the same bytes as lustro net export --ids for those ids. The DevTools Network panel that Chrome serves imported that file, the CLI's export of all 10, and the golden fixture through its own import path, with no load error and no console message. The selection's Markdown rendered through GitHub's GFM API with its headings and json, xml, and http blocks, and the detail's Markdown of the 250 KB echo marked the truncated response. Signed-off-by: Evgenii Plokhov <plokhov@gmail.com>
Copy Markdown read the selection in reverse list order. The list is in the order captures were stored, and with a full capture backlog a new capture is stored before older queued ones, so the document could have two requests out of order. The selection is now sorted by startedAt; the HAR export already sorted its entries. A body kept as bytes is not included in the Markdown, and its note now also says when capture kept only the first part of it. Signed-off-by: Evgenii Plokhov <plokhov@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A transaction left Lustro only as JSON from the API, as a cURL command, or as plain text, one at a time. This adds the two formats that matter for sharing: HAR, which browser devtools and most HTTP tools import and agents parse without instructions, and Markdown, for a bug report, a pull request, or a chat.
HAR on the server.
GET /api/v1/network/transactions/_/export?format=har&ids=a,breturns a HAR 1.2 document of the transactions inids, or of every one whenidsis left out, oldest first. It is built from the store, so it has the redacted values of the detail.startedAtstartedDateTime, ISO 8601 in UTC with millisecondsdurationMstimeandtimings.wait;sendandreceiveare0, the other phases-1protocolhttpVersion(h2), empty when unknownheaders,queryString(decoded)postData; bytes in base64 with_encoding: base64content.text; bytes in base64 withencoding: base64requestBodyBytes,responseBodyBytesbodySize,-1when null;content.sizeis the larger of the byte count and the size kept, since a gzip body is kept inflatedLocationredirectURLerrorresponse._error, where Chrome's own export puts itisMocked,categories, truncation flags,responseComplete,error,id_lustro_resourceType(fetch, orimage) tells Chrome DevTools where to list the entry. Without it, DevTools listed every JSON response as a script.CLI.
lustro net export --har FILE [--ids ID ...]writes the document, or prints it for--har -.--idstakes short ids, and the command sends them in batches of 50 and joins the results, because the server rejects a request line and headers over 8 KB. It warns about ids that the app no longer has.Console. Select adds a checkbox column, a header checkbox that selects every row the filters show (also those past the 200 rendered), and a bar with the count, Export HAR, and Copy Markdown. A filter change keeps only the selected rows that it still shows. Export HAR fetches the same batches and saves
lustro-<date>-<time>.har. Copy Markdown fetches each detail, four at a time, and copies one document. The detail gets a Markdown button next to cURL and Copy.A transaction in Markdown is the method and URL as a heading (one code span, so a URL can't become markup), a meta line, the headers in
httpblocks, and each body in a block tagged with a language for its content type. JSON is indented without changing a value. A fence is longer than any run of backticks in the body. A body that capture cut short says_Truncated: Lustro kept only the first part of this body._; a body kept as bytes, or in an encoding Lustro does not decode, says that instead of a block.Contract. The OpenAPI document describes the route and the HAR shape (
HarDocument,HarEntry). The golden fixtureexport-har.jsonhas five entries: a POST with a JSON body, a mock, a GIF in base64, a truncated response, and a failure. A Kotlin test checks that the export writes exactly that fixture, the CLI tests validate it against the schema, and the CLI end-to-end run exports the sample's mocked request.Decisions to check
captureFilterdid._metastill says1.2. If 1.2 counts as released, this needs a bump to 1.3.ids=with no value is an empty selection, not every transaction, so a client that sends an empty selection never gets the whole store.curlwithoutidsgets it all at once.postData._encodingis our own field: HAR 1.2 hasencodingoncontentonly. Tools that ignore it see base64 text for a binary request body.ClipboardItemwith a promise when the page is a secure context, because Safari allows a copy only during the click and the details arrive later. It falls back to the text copy elsewhere.Verification
On a physical device (Android 17), with the debug sample's traffic: GET, POST, a 500, JSON, XML, a JPEG, a PNG request body, the 250 KB JSON echo, a
brresponse, and a 2 s timeout. The console ran in headless Chrome, driven over CDP with real mouse events.10 of 10 selected. Turning off the 5xx filter:9 of 9 selected, and the header checkbox stays checked. Unchecking one row:8 of 9, header indeterminate. Leaving Select mode clears it.lustro-20261002-121122.har.lustro net export --idswith the same 9 ids wrote the same bytes (cmp).onLoadFromFile), with no HAR load failure and no console message. It listed every request with its method, status, media type, protocol, and duration; JSON under Fetch, the JPEG under Img with its base64 bytes decoded, and the 250 KB echo with its 262,144 characters.json,xml, andhttphighlighting; the JPEG, PNG, andbrnotes rendered as italics.lustro net export --haron the device: 10 transactions, 735 KB, with_lustro.responseBodyTruncatedon the echo anderror: "Canceled"with status 0 on the timeout.Tests:
./gradlew :lustro:testDebugUnitTest :lustro:lintDebug detekt apiCheck checkFacadeParitypasses, withNetworkHarExportTest(12 tests) andHarExportGoldenTest.pytestinlustro-cli: 238 pass, 13 of them new intest_cli_export.py, plus schema checks of the golden HAR.node --test lustro/src/test/js/*.test.js: 100 pass, 14 of them new innetwork-export.test.jsfor the Markdown and the batch merge.Type of change
Checklist
./gradlew apiCheckpasses (no unintended public API changes; dump updated if intended)./gradlew detektpassesCHANGELOG.mdupdated under[Unreleased]wire-protocol/and tab OpenAPI fragmentsgit commit -sCompatibility notes
Additive: a new route, within the unreleased protocol 1.2. No Kotlin API change.