Skip to content

release: v0.3.3 add SPECTER2 + TEI support#9

Merged
rkrug merged 1 commit into
mainfrom
claude/specter2-support
Jun 3, 2026
Merged

release: v0.3.3 add SPECTER2 + TEI support#9
rkrug merged 1 commit into
mainfrom
claude/specter2-support

Conversation

@rkrug

@rkrug rkrug commented Jun 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds backend_specter2_tei() — thin wrapper around backend_config() for a local TEI server serving the merged SPECTER2 proximity model.
  • Adds one-time setup material under inst/scripts/:
    • prepare_specter2_merged.py — Python merge of the proximity adapter into specter2_base into a HF-format model dir TEI can serve directly. Default output under R_user_dir() cache, overridable via OVC_SPECTER2_PATH.
    • start_tei_specter2.sh — launcher that follows the same path convention.
  • Adds vignettes/specter2-setup.qmd — end-to-end walkthrough (Python env, merge, TEI start, R smoke test).
  • Bumps version to 0.3.3.
  • NOTES.md (Rbuildignored) records the follow-up option to publish the merged model to HuggingFace Hub so users can skip the merge step.
  • CLAUDE.md is a Claude Code guide for the repo (init).

Rationale

For 4M+ academic-paper embedding, SPECTER2 is domain-tuned (citation-linked papers), produces 768-dim vectors, and runs locally on TEI at zero per-token cost. The proximity adapter is required for the topic-similarity objective, but TEI does not load adapter-transformers adapters — hence the one-time merge step.

Model preparation deliberately stays out of the R API surface to avoid a Python dependency on this R-first package (see DEVELOPMENT_CONTINUITY.md decision log entry).

Test plan

  • Run inst/scripts/prepare_specter2_merged.py in a fresh venv with transformers + adapters + torch — produces a model dir with config.json and weights.
  • Start TEI via inst/scripts/start_tei_specter2.sh and confirm /info reports the local model path and /embed returns 768-dim vectors.
  • From R: backend_specter2_tei() returns the expected backend_config and backend_embed_texts() produces a 1 x 768 matrix.
  • devtools::check(args = c("--no-manual","--no-multiarch","--no-examples","--ignore-vignettes")) passes locally.
  • Render vignettes/specter2-setup.qmd locally.

🤖 Generated with Claude Code

Add a documented path for using SPECTER2's proximity-adapter model with the
existing TEI backend, plus a thin R helper for the boilerplate config.

- backend_specter2_tei(): convenience wrapper around backend_config() for a
  local TEI server serving the merged SPECTER2 proximity model.
- inst/scripts/prepare_specter2_merged.py: one-time Python script that fuses
  the proximity adapter into specter2_base and saves a HF-format model dir
  TEI can serve directly. Default output under R_user_dir() cache,
  overridable via OVC_SPECTER2_PATH.
- inst/scripts/start_tei_specter2.sh: launcher that resolves the same path
  convention and runs text-embeddings-router.
- vignettes/specter2-setup.qmd: end-to-end setup + serve + smoke test.
- NOTES.md (Rbuildignored): captures the option to publish the merged model
  to HuggingFace Hub so users can skip the merge step entirely.
- CLAUDE.md: repo guide for future Claude Code sessions.

Model preparation deliberately stays out of the R API surface: TEI cannot
load adapter-transformers adapters, so a one-time merge is required, but
forcing a Python dependency on this R-first package conflicts with the
design principle that the package does not manage external services.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@rkrug rkrug merged commit e2eb96f into main Jun 3, 2026
6 checks passed
@rkrug rkrug deleted the claude/specter2-support branch June 3, 2026 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant