Skip to content

Use magic for binary detection - #22

Merged
andrew merged 1 commit into
mainfrom
magic-integration
Jul 31, 2026
Merged

Use magic for binary detection#22
andrew merged 1 commit into
mainfrom
magic-integration

Conversation

@andrew

@andrew andrew commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Use github.com/git-pkgs/magic to classify files before packing. Preserve the existing binary skip result and accept only UTF-8 text, while catching binary signatures, disallowed controls, and NUL bytes beyond the old 8 KiB probe.

Add coverage for UTF-16, malformed UTF-8, late NULs, binary formats without NUL, Unicode, and empty content.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates Outline’s packing pipeline to use github.com/git-pkgs/magic for content classification so that only UTF-8 text is packed while reliably skipping binary content (including files with binary signatures, disallowed control bytes, malformed UTF-8, and late NULs beyond the prior 8 KiB probe).

Changes:

  • Replace the previous NUL/UTF-8 prefix probe with magic.Detect-based text gating via isPackableText.
  • Expand packing tests to cover PDF detection, UTF-16, malformed UTF-8, disallowed controls, late NUL bytes, Unicode, and empty content.
  • Add the github.com/git-pkgs/magic v0.1.0 module dependency.

Reviewed changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated no comments.

File Description
pack.go Switches file packability checks from a fixed-size binary sniff to magic-based classification while preserving the "binary" skip reason.
pack_test.go Adds/updates tests to validate the new classification behavior across more encodings and binary/text edge cases.
go.mod Introduces the github.com/git-pkgs/magic dependency.
go.sum Adds checksums for the new magic dependency.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@andrew
andrew merged commit bd97249 into main Jul 31, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants