Skip to content

feat: configurable fragment separator mode - #14

Merged
jgordijn merged 3 commits into
mainfrom
configurable-fragment-separator
Mar 4, 2026
Merged

jgordijn merged 3 commits into
mainfrom
configurable-fragment-separator

Conversation

@jgordijn

@jgordijn jgordijn commented Mar 4, 2026

Copy link
Copy Markdown
Owner

What

Add per-resource fragment splitting configuration so fragment feeds can use either:

  • Auto (default): existing heuristic + AI paragraph-based splitting
  • Separated: split by an explicit text separator (e.g. ~~~ for aishepherd.nl/moments)

Changes

Backend

  • SplitFragmentsBySeparator() in internal/engine/fragment.go — walks DOM children, groups elements between separator-matching elements
  • Fetcher dispatch in internal/engine/fetcher.go — routes to separator or auto mode based on resource config
  • DB migration in cmd/knowledgehub/collections.go — adds fragment_mode (select: auto/separated) and fragment_separator (text) fields
  • Test collection schema in internal/testutil/testutil.go — includes new fields
  • 6 new tests covering: basic split, not found, leading/trailing, consecutive separators, different separators, mixed elements

Frontend

  • ResourceForm.svelte — split mode dropdown + separator text input, shown when Fragment feed is checked
  • resources/+page.svelte — passes new props in edit flow, badge shows Fragment (~~~) for separated mode

OpenSpec

  • Archived change with proposal, design, specs, and tasks
  • Synced delta specs to content-fetching and resource-management main specs

Testing

All existing tests pass. 6 new SplitFragmentsBySeparator tests pass. Frontend builds cleanly.

jgordijn added 3 commits March 4, 2026 13:00
Add per-resource fragment splitting configuration:
- fragment_mode field: 'auto' (heuristic + AI) or 'separated' (text separator)
- fragment_separator field: the separator string (e.g. '~~~')

Backend:
- SplitFragmentsBySeparator() splits HTML by matching DOM elements
- Fetcher dispatches to separator or auto mode based on resource config
- DB migration adds both fields via addFieldIfMissing

Frontend:
- ResourceForm shows split mode dropdown when Fragment feed is checked
- Separator text input shown when mode is 'separated'
- Resource list badge shows separator value (e.g. 'Fragment (~~~)')

Tests: 6 new test cases for separator splitting
Sync delta specs to main specs:
- content-fetching: separator-based splitting, auto mode, no AI for separated
- resource-management: fragment mode configuration, badge display

Archive change to openspec/changes/archive/2026-03-04-configurable-fragment-separator/
@jgordijn
jgordijn merged commit e07c947 into main Mar 4, 2026
1 check passed
@jgordijn
jgordijn deleted the configurable-fragment-separator branch March 4, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant