fix: treat literal tokenizer markers as ordinary content

[yyiilluu]

Summary

• Prevent profile/playbook prompt formatting from failing when ordinary interaction text contains literal tokenizer markers such as <|endoftext|>.
• Encode those marker strings as ordinary text while preserving the existing token budget and head/tail slicing behavior.

Changes

• Pass disallowed_special=() in the shared content slicer, matching the existing billing and embedding tokenizers.
• Extend regression coverage for all cl100k marker strings, truncation, prompt rendering, and visible evidence.
• Exercise the full SQLite publish/profile workflow with a literal marker. Disable the canned service shortcut in that test while keeping LiteLLM mocked, so the test reaches real prompt construction.

Test Plan

• Confirmed the added formatter regressions fail on the original implementation.
• Confirmed the literal-marker publish case returns an extraction warning before the fix, while the ordinary-text control succeeds.
• After the fix: 95 passed across service utilities, playbook prompt utilities, and the parameterized publish/profile workflow.
• Ruff lint/format checks and scoped Pyright passed.
• No paid model calls, provider configuration changes, or deployment were performed.

uv run pytest tests/server/services/test_service_utils.py \
  tests/server/services/playbook/test_playbook_service_utils.py \
  tests/e2e_tests/test_profile_workflows.py::test_publish_interaction_profile_only \
  -q -o 'addopts='

Summary by CodeRabbit

• Bug Fixes

  • Content containing literal tokenizer markers is now processed as ordinary text instead of being rejected.
  • Token-based content trimming now behaves consistently across formatted history and visible evidence, including when truncation is disabled.
  • Publishing workflows now support interactions containing literal end-of-text markers.

• Tests

  • Expanded coverage for token handling, content prefixes, truncation behavior, and publishing scenarios.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 88457917-353b-4cb5-98c1-36b90788cda9

📥 Commits

Reviewing files that changed from the base of the PR and between bdcc212 and 4c63ffb.

📒 Files selected for processing (3)

• reflexio/server/services/service_utils.py
• tests/e2e_tests/test_profile_workflows.py
• tests/server/services/test_service_utils.py

Included review availability: 3 reviews are currently available. Your included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.

---

📝 Walkthrough

Walkthrough

The token slicing utility now treats literal tokenizer markers as ordinary text. Unit tests cover truncation and prompt evidence. The profile workflow test covers publishing content with a literal end-of-text marker.

Changes

Literal Token Handling

Layer / File(s)	Summary
Ordinary token slicing behavior reflexio/server/services/service_utils.py, tests/server/services/test_service_utils.py, tests/e2e_tests/test_profile_workflows.py	slice_content_by_tokens uses permissive encoding and preserves literal tokenizer markers. Tests cover budgeted content, head-and-tail truncation, prompt history, visible evidence, and profile publishing.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to 4c63f

The change treats literal tokenizer markers as ordinary content while preserving token slicing behavior, with targeted regressions and workflow checks passing. No actionable merge-blocking risk remains beyond normal review and checks.

Suggested reviewers: yilu331

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: treating literal tokenizer markers as ordinary content.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 3 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/fix-literal-tokenizer-markers

Warning

Your free Security trial is over. An organization admin can activate Security or dismiss this notice.

---

_{Comment @coderabbitai help to get the list of available commands.}