Skip to content

Add NGA-KR/law-embed-0.6b-r70 with task-routed legal adapters - #5797

Open
dk-nga wants to merge 2 commits into
embeddings-benchmark:mainfrom
dk-nga:add-nga-law-r70
Open

dk-nga wants to merge 2 commits into
embeddings-benchmark:mainfrom
dk-nga:add-nga-law-r70

Conversation

@dk-nga

@dk-nga dk-nga commented Oct 27, 2026 •

Copy link
Copy Markdown

Add NGA-KR/law-embed-0.6b-r70: 595,874,816 parameters, fine-tuned backbone and three output adapters.

Pinned release, model card and experiment history: https://huggingface.co/NGA-KR/law-embed-0.6b-r70/tree/e36d2b0b22d9502a55ef9059820975dbd1c9d65d

MTEB(Law,v1) Mean(Task): 73.50425 (MTEB 2.20.11, BF16, batch 4, max length 32768). The 4B teacher is training-only.

Disclosures

  • Task routing: GerDaLIRSmall query uses query then symmetric adapters; corpus uses symmetric. LeCaRDv2 uses its relational-distilled symmetric adapter on both sides. Other tasks use fixed prompts/backbone. Please assess eligibility as a routed model.
  • Not zero-shot: GerDaLIR and Chinese legal/CAIL training, including inherited GerDaLIRSmall/LeCaRDv2 exposure. Full sources are in the model card. Exact/near-overlap filtering cannot exclude semantic overlap.
  • Repeated evaluations on the public Law test suite informed later experiments; stopping target was 73.50. This is not an untouched holdout estimate. Aggregate history is included in the release.
  • Original evaluation used a patched wrapper and base-model metadata. This loader reproduces it with pinned public artifacts; scores are unchanged, metadata corrected to this model/revision.

Validation

  • 28/28 compatible text mock tasks passed on MTEB 2.20.11.
  • All 16 Law query/document paths exactly match the original on synthetic multilingual inputs (max embedding difference 0). This checks integration equivalence, not an independent full benchmark rerun.
  • Ruff check/format passed; reports included.

Checklist

  • ModelMeta, public weights/adapters/code, loader and mock tests
  • Training exposure and test reuse disclosed
  • Current-main registry integration verified in CI

@github-actions

Copy link
Copy Markdown
Contributor

This pull request description is marked as long, 2606 characters exceeding the 2000 limit.

This is usually a sign that the description is too verbose and e.g. due to being auto-generated. This poses undue strain on our contributors. Please reduce the length of the comment. Reviewers are free to close long PRs without providing a reason.

See our contributing guidelines for details on what a good PR description should contain.

Comment on lines +18 to +32
PROMPTS = {
"query": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:",
"document": " ",
"AILACasedocs-query": "Instruct: Given a legal case scenario, retrieve the most relevant case documents\nQuery:",
"AILAStatutes-query": "Instruct: Given a legal scenario, retrieve the most relevant statute documents\nQuery:",
"GerDaLIRSmall-query": "Instruct: Given a passage from a German court decision, retrieve the legal case documents cited by that passage.\nQuery: ",
"LeCaRDv2-query": "Instruct: Retrieve text based on user query.\nQuery:",
"LegalBenchConsumerContractsQA-query": "Instruct: Retrieve text based on user query.\nQuery:",
"LegalBenchCorporateLobbying-query": "Instruct: Retrieve text based on user query.\nQuery:",
"LegalQuAD-query": "Instruct: Given a legal question, retrieve relevant legal documents that answer the question\nQuery:",
"LegalSummarization-query": "Instruct: Given a query, retrieve relevant legal documents for summarization\nQuery:",
}


class NGALawEncoder(SentenceTransformerEncoderWrapper):

@Samoed Samoed Oct 27, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think better to use InstructSentenceTransformerModel

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

long PR desc PR Description is long & too verbose.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants