Skip to content

Step 3: Ingest and normalization #17

Description

@craig-dt

From PLAN.md step 3. Spec: docs/spec.md. Phase 1.

Parallel-safe — touches files no other concurrent step touches.

Files: src/flabel/ingest.py, tests/test_ingest.py, tests/fixtures/make_awkward.py

Changes: Format sniffing by magic bytes (never extension), gzip decompression, flabel's own record-header walk for packet count and truncation offset, editcap -F pcap conversion, multi-datalink dominant-type selection. Emits NormalizedCapture with everything provenance needs.

make_awkward.py extends the existing canary generator to emit the nasty inputs: truncated pcap, truncated pcapng, multi-datalink pcapng, bad header, gzipped variants.

Test that proves it: each awkward fixture produces its specified outcome — truncated pcap gives input_status: partial with a correct offset; truncated pcapng and bad header raise CaptureError and create no output; multi-datalink keeps the dominant type and records the discards; gzip is transparent. Round-trip: a plain pcap normalizes to a byte-identical file.

Depends on: 2. Parallel with: 4, 5, 6.

Metadata

Metadata

Assignees

No one assigned

    Labels

    plan-stepOne step from PLAN.md

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions