Skip to content

improve coverage on mteb - #794

Merged
KennethEnevoldsen merged 1 commit into
mainfrom
coverage-5589
Oct 21, 2026
Merged

improve coverage on mteb#794
KennethEnevoldsen merged 1 commit into
mainfrom
coverage-5589

Conversation

@KennethEnevoldsen

@KennethEnevoldsen KennethEnevoldsen commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

improving: embeddings-benchmark/mteb#5589

Checklist

  • My model has a model sheet, report, or similar
  • My model has a reference implementation in mteb/models/model_implementations/, this can be as an API. Instruction on how to add a model can be found here
    • No, but there is an existing PR ___
  • The results submitted are obtained using the reference implementation
  • My model is available, either as a publicly accessible API or publicly on e.g., Huggingface
  • I solemnly swear that for all results submitted I have not trained on the evaluation dataset including training splits. If I have, I have disclosed it clearly.

@github-actions

github-actions Bot commented Oct 19, 2026

Copy link
Copy Markdown

Model Results Comparison

Reference models: intfloat/multilingual-e5-large, google/gemini-embedding-001
New models evaluated: Qwen/Qwen3-Embedding-0.6B, Qwen/Qwen3-Embedding-4B, Qwen/Qwen3-Embedding-8B, codefuse-ai/F2LLM-v2-0.6B, codefuse-ai/F2LLM-v2-1.7B, codefuse-ai/F2LLM-v2-14B, codefuse-ai/F2LLM-v2-4B, codefuse-ai/F2LLM-v2-8B, intfloat/multilingual-e5-large-instruct, jinaai/jina-embeddings-v5-omni-nano, microsoft/harrier-oss-v1-0.6b, microsoft/harrier-oss-v1-270m

Results for Qwen/Qwen3-Embedding-0.6B

task_name Qwen/Qwen3-Embedding-0.6B google/gemini-embedding-001 intfloat/multilingual-e5-large Max result Model with max result In Training Data
AILACasedocs .345 .483 .264 .656 Octen/Octen-Embedding-8B-INT8 False
AllegroReviews .475 nan .410 .671 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloProfClusteringP2P .719 nan .630 .761 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloProfClusteringS2S .579 nan .323 .645 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloprofRetrieval .587 nan .393 .738 mongodb/voyage-3 False
AmazonReviewsClassification .549 nan .431 .641 KaLM-Embedding/KaLM-embedding-multilingual-mini-instruct-v2.5 False
ArguAna-NL .567 nan .489 .740 infly/inf-retriever-v1 False
BSARDRetrieval .577 nan .631 .829 mongodb/voyage-3 False
BengaliSentimentAnalysis .695 nan .831 .920 LingoIITGN/FlowVec-v1 False
BlurbsClusteringP2P .407 nan .415 .477 Salesforce/SFR-Embedding-Mistral False
BlurbsClusteringS2S .172 nan .168 .220 Linq-AI-Research/Linq-Embed-Mistral False
CBD .633 nan .673 .841 BAAI/bge-multilingual-gemma2 False
CDSC-E .692 nan .745 .799 sdadas/mmlw-roberta-large False
CDSC-R .883 nan .910 .976 Gameselo/STS-multilingual-mpnet-base-v2 False
ESCIReranking .818 nan .832 .846 VPLabs/SearchMap_Preview False
EightTagsClustering .366 nan .265 .503 BAAI/bge-multilingual-gemma2 False
FalseFriendsGermanEnglish .505 nan .534 .712 aari1995/German_Semantic_STS_V2 False
FiQA2018-NL .282 nan .328 .511 infly/inf-retriever-v1 False
GermanDPR .791 nan .829 .856 malteos/most-embed-de False
GermanQuAD-Retrieval .920 nan .932 .971 nvidia/Nemotron-3-Embed-1B-BF16 False
HALClusteringS2S .265 nan .224 .308 Alibaba-NLP/gte-Qwen2-7B-instruct False
HindiDiscourseClassification .364 nan .387 .507 Sailesh97/Hinvec False
IN22ConvBitextMining .452 nan .582 .669 LingoIITGN/FlowVec-v1 False
JQaRAReranking .385 nan .489 .642 sbintuitions/sarashina-embedding-v2-1b False
JSTS .811 nan .818 .861 NGA-KR/ja-embed-sts False
JaCWIRReranking .843 nan .871 .899 sionic-ai/comsat-embed-ja-8b-preview False
JaCWIRRetrieval .831 nan .863 .895 sionic-ai/comsat-embed-ja-8b-preview False
JaGovFaqsRetrieval .678 nan .724 .790 sionic-ai/comsat-embed-ja-0.3b-preview False
JapaneseSentimentClassification .792 nan .858 .947 sbintuitions/sarashina-embedding-v2-1b False
KLUE-NLI .712 nan .659 .777 NGA-KR/ko-embed-nli False
KLUE-STS .846 nan .834 .915 alphabrothers/alpha-sts-v0 False
KLUE-TC .606 nan .608 .844 NGA-KR/ko-embed-cls False
KlueMrcDomainClustering .826 nan nan .814 BAAI/bge-multilingual-gemma2 False
KlueYnatMrcCategoryClustering .586 nan nan .814 dragonkue/snowflake-arctic-embed-l-v2.0-ko False
KorNLI .800 nan nan .841 NGA-KR/ko-embed-nli False
KorSTS .842 nan .805 .977 Gameselo/STS-multilingual-mpnet-base-v2 False
LEMBNeedleRetrieval .505 nan .280 .932 mteb/baseline-bm25s False
LEMBQMSumRetrieval .477 nan .243 .832 mteb/baseline-bm25s False
LEMBSummScreenFDRetrieval .970 nan .711 .985 DataScience-UIBK/uibk-embedding-v1 False
LEMBWikimQARetrieval .897 nan .568 .999 lightonai/GTE-ModernColBERT-v1 False
LeCaRDv2 .745 nan .558 .778 Mira190/Euler-Legal-Embedding-V1 False
LegalBenchConsumerContractsQA .799 nan .733 .878 minetta/nemotron-3-embed-8b-legal False
LegalQuAD .525 .655 .432 .772 nvidia/Nemotron-3-Embed-8B-BF16 False
LegalSummarization .623 .712 .621 .797 mongodb/voyage-4-large (embed_dim=2048, evolved_prompts=True) False
LivedoorNewsClustering.v2 .476 nan .565 .606 cl-nagoya/ruri-v3-130m False
MTOPDomainClassification .912 .980 .902 1.000 mongodb/voyage-3-m-exp False
MalayalamNewsClassification .729 nan .760 .918 LingoIITGN/FlowVec-v1 False
MasakhaNEWSClusteringP2P .661 nan .519 .772 HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1 False
MedicalQARetrieval .718 nan .698 .801 codefuse-ai/F2LLM-v2-330M False
MewsC16JaClustering .560 nan .454 .617 intfloat/e5-mistral-7b-instruct False
MintakaRetrieval .316 .618 .304 .625 BAAI/bge-multilingual-gemma2 False
NFCorpus-NL .266 nan .298 .370 infly/inf-retriever-v1 False
NLPJournalAbsArticleRetrieval.V2 .995 nan .834 .997 sionic-ai/comsat-embed-ja-8b-preview False
NLPJournalAbsIntroRetrieval.V2 .983 nan .860 .995 sbintuitions/sarashina-embedding-v2-1b False
NLPJournalTitleAbsRetrieval.V2 .952 nan .953 .981 cl-nagoya/ruri-v3-130m False
NLPJournalTitleIntroRetrieval.V2 .928 nan .726 .978 sbintuitions/sarashina-embedding-v2-1b False
PSC .982 nan .994 .997 Salesforce/SFR-Embedding-2_R False
PlscClusteringP2P .506 nan nan .489 codefuse-ai/F2LLM-v2-330M False
PlscClusteringS2S .460 nan nan .446 hotchpotch/bekko-embedding-v1-a25m False
PolEmo2.0-IN .691 nan .771 .904 BAAI/bge-multilingual-gemma2 False
PublicHealthQA .826 nan .850 .914 Octen/Octen-Embedding-8B False
Quora-NL .796 nan .821 .857 infly/inf-retriever-v1 False
SCIDOCS-NL .167 nan .131 .248 infly/inf-retriever-v1 False
SIB200Classification .614 nan .540 .950 codefuse-ai/F2LLM-v2-330M False
SICK-E-PL .676 nan .760 .932 Gameselo/STS-multilingual-mpnet-base-v2 False
SICK-R-PL .692 nan .751 .942 Gameselo/STS-multilingual-mpnet-base-v2 False
SICKFr .800 nan .792 .862 Gameselo/STS-multilingual-mpnet-base-v2 False
STS22 .706 .718 .637 .831 OrdalieTech/Solon-embeddings-mini-beta-1.1 False
SanskritShlokasClassification .545 nan .739 .842 ai-forever/ru-en-RoSBERTa False
SciFact-NL .656 nan .684 .802 infly/inf-retriever-v1 False
SciFact-PL .607 nan .658 .776 infly/inf-retriever-v1 False
SpanishNewsClassification.v2 .901 .910 .886 .913 openai/text-embedding-3-large False
SpanishPassageRetrievalS2P .422 .489 .420 .510 jinaai/jina-embeddings-v5-text-nano False
SpanishPassageRetrievalS2S .705 .771 .723 .771 google/gemini-embedding-001 False
SpanishSentimentClassification.v2 .850 .966 .924 .966 google/gemini-embedding-001 False
SummEvalFr .291 nan .309 .330 intfloat/multilingual-e5-base False
SyntecReranking .887 nan .854 .940 Alibaba-NLP/gte-Qwen2-1.5B-instruct False
SyntecRetrieval .851 nan .824 .924 intfloat/e5-mistral-7b-instruct False
TRECCOVID-NL .747 nan .698 .735 jinaai/jina-embeddings-v3 False
TRECCOVID-PL .723 nan .699 .827 BAAI/bge-multilingual-gemma2 False
TenKGnadClusteringP2P .551 nan .448 .582 jinaai/jina-embeddings-v5-text-small False
TenKGnadClusteringS2S .365 nan .308 .437 jinaai/jina-embeddings-v5-text-small False
TweetSentimentClassification .482 nan .503 .598 google/embeddinggemma-300m False
UrduRomanSentimentClassification .421 nan .416 .620 LingoIITGN/FlowVec-v1 False
WRIMEClassification .286 nan .284 .351 google/embeddinggemma-300m False
XMarket .258 nan .175 .360 infly/inf-retriever-v1 False
XPQARetrieval .473 .669 .473 .774 BAAI/bge-multilingual-gemma2 False
XQuADRetrieval .927 nan .971 .971 telepix/PIXIE-Rune-v1.0 False
Average .638 .725 .612 .760 nan -

Model have high performance on these tasks: KlueMrcDomainClustering,TRECCOVID-NL,PlscClusteringP2P,PlscClusteringS2S

Results for Qwen/Qwen3-Embedding-4B

task_name Qwen/Qwen3-Embedding-4B google/gemini-embedding-001 intfloat/multilingual-e5-large Max result Model with max result In Training Data
AILACasedocs .398 .483 .264 .656 Octen/Octen-Embedding-8B-INT8 False
AllegroReviews .498 nan .410 .671 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloProfClusteringP2P .762 nan .630 .761 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloProfClusteringS2S .622 nan .323 .645 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloprofRetrieval .654 nan .393 .738 mongodb/voyage-3 False
ArguAna-Fa.v2 .539 nan .413 .585 google/embeddinggemma-300m False
ArguAna-NL .579 nan .489 .740 infly/inf-retriever-v1 False
Assin2STS .821 nan .783 .841 iara-project/e5-large-matryoshka-sts-pt False
BSARDRetrieval .689 nan .631 .829 mongodb/voyage-3 False
BengaliSentimentAnalysis .760 nan .831 .920 LingoIITGN/FlowVec-v1 False
BeytooteClustering .708 nan .615 .677 Alibaba-NLP/gte-Qwen2-7B-instruct False
BlurbsClusteringS2S .225 nan .168 .220 Linq-AI-Research/Linq-Embed-Mistral False
CBD .775 nan .673 .841 BAAI/bge-multilingual-gemma2 False
CDSC-E .734 nan .745 .799 sdadas/mmlw-roberta-large False
CDSC-R .914 nan .910 .976 Gameselo/STS-multilingual-mpnet-base-v2 False
ChemHotpotQARetrieval .849 nan .798 .953 infly/inf-retriever-v1 False
ChemNQRetrieval .631 nan .662 .705 intfloat/multilingual-e5-small False
DeepSentiPers.v2 .518 nan .577 .723 MCINext/Hakim False
DigikalamagClassification .892 nan .871 .910 MCINext/Hakim False
DigikalamagClustering .632 nan .399 .791 MCINext/Hakim False
EightTagsClustering .440 nan .265 .503 BAAI/bge-multilingual-gemma2 False
FalseFriendsGermanEnglish .559 nan .534 .712 aari1995/German_Semantic_STS_V2 False
FarsiParaphraseDetection .973 nan .976 .996 MCINext/Hakim False
Farsick .736 nan .707 .770 jinaai/jina-embeddings-v3 False
GermanDPR .829 nan .829 .856 malteos/most-embed-de False
GermanQuAD-Retrieval .953 nan .932 .971 nvidia/Nemotron-3-Embed-1B-BF16 False
HALClusteringS2S .291 nan .224 .308 Alibaba-NLP/gte-Qwen2-7B-instruct False
HamshahriClustring .755 nan .674 .756 myrkur/sentence-transformer-parsbert-fa False
HateSpeechPortugueseClassification .600 nan .573 .628 intfloat/e5-mistral-7b-instruct False
HindiDiscourseClassification .351 nan .387 .507 Sailesh97/Hinvec False
JSTS .858 nan .818 .861 NGA-KR/ja-embed-sts False
JaGovFaqsRetrieval .683 nan .724 .790 sionic-ai/comsat-embed-ja-0.3b-preview False
JapaneseSentimentClassification .777 nan .858 .947 sbintuitions/sarashina-embedding-v2-1b False
LEMBSummScreenFDRetrieval .980 nan .711 .985 DataScience-UIBK/uibk-embedding-v1 False
LEMBWikimQARetrieval .939 nan .568 .999 lightonai/GTE-ModernColBERT-v1 False
LegalBenchConsumerContractsQA .835 nan .733 .878 minetta/nemotron-3-embed-8b-legal False
LegalQuAD .666 .655 .432 .772 nvidia/Nemotron-3-Embed-8B-BF16 False
LegalSummarization .664 .712 .621 .797 mongodb/voyage-4-large (embed_dim=2048, evolved_prompts=True) False
LivedoorNewsClustering.v2 .499 nan .565 .606 cl-nagoya/ruri-v3-130m False
MSMARCO-FaHardNegatives .675 nan .687 .687 intfloat/multilingual-e5-large True
MTOPDomainClassification .955 .980 .902 1.000 mongodb/voyage-3-m-exp False
MTOPIntentClassification .868 nan .672 .944 jinaai/jina-embeddings-v5-text-small False
MalayalamNewsClassification .919 nan .760 .918 LingoIITGN/FlowVec-v1 False
MasakhaNEWSClusteringP2P .700 nan .519 .772 HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1 False
MedicalQARetrieval .726 nan .698 .801 codefuse-ai/F2LLM-v2-330M False
MewsC16JaClustering .573 nan .454 .617 intfloat/e5-mistral-7b-instruct False
MintakaRetrieval .595 .618 .304 .625 BAAI/bge-multilingual-gemma2 False
NFCorpus-NL .346 nan .298 .370 infly/inf-retriever-v1 False
NLPJournalAbsIntroRetrieval.V2 .990 nan .860 .995 sbintuitions/sarashina-embedding-v2-1b False
NLPJournalTitleAbsRetrieval.V2 .979 nan .953 .981 cl-nagoya/ruri-v3-130m False
NLPJournalTitleIntroRetrieval.V2 .966 nan .726 .978 sbintuitions/sarashina-embedding-v2-1b False
NLPTwitterAnalysisClassification.v2 .793 nan .766 .800 MCINext/Hakim False
NLPTwitterAnalysisClustering .847 nan .785 .860 intfloat/e5-mistral-7b-instruct False
PSC .990 nan .994 .997 Salesforce/SFR-Embedding-2_R False
ParsinluEntail .742 nan .655 .777 PartAI/Tooka-SBERT False
ParsinluQueryParaphPC .893 nan .878 .900 BAAI/bge-m3 False
PerShopDomainClassification .453 nan .552 .761 PartAI/Tooka-SBERT-V2-Large False
PerShopIntentClassification .831 nan .907 .928 sentence-transformers/LaBSE False
PersianFoodSentimentClassification .695 nan .821 .865 MCINext/Hakim False
PersianTextEmotion.v2 .400 nan .609 .865 MCINext/Hakim False
PlscClusteringP2P .526 nan nan .489 codefuse-ai/F2LLM-v2-330M False
PlscClusteringS2S .503 nan nan .446 hotchpotch/bekko-embedding-v1-a25m False
PolEmo2.0-IN .775 nan .771 .904 BAAI/bge-multilingual-gemma2 False
PubChemAISentenceParaphrasePC .963 nan .966 .975 sentence-transformers/multi-qa-mpnet-base-dot-v1 False
PubChemSMILESBitextMining .003 nan .002 .007 ICT-TIME-and-Querit/BOOM_4B_v1 False
PubChemSMILESPC .150 nan .108 .161 ICT-TIME-and-Querit/BOOM_4B_v1 False
PubChemSynonymPC .754 nan .640 .735 openai/text-embedding-3-large False
PubChemWikiPairClassification .947 nan .945 .965 hotchpotch/bekko-embedding-v1-a8m False
PubChemWikiParagraphsPC .429 nan .192 .518 codefuse-ai/F2LLM-v2-330M False
PublicHealthQA .866 nan .850 .914 Octen/Octen-Embedding-8B False
QuoraRetrieval-Fa.v2 .757 nan .779 .803 BAAI/bge-m3 False
SAMSumFa .916 nan .924 .992 MCINext/Hakim False
SCIDOCS-Fa.v2 .187 nan .122 .162 Alibaba-NLP/gte-Qwen2-7B-instruct False
SCIDOCS-NL .219 nan .131 .248 infly/inf-retriever-v1 False
SDSEyeProtectionClassification .649 nan .712 .830 minishlab/potion-multilingual-128M False
SDSGlovesClassification .670 nan .637 .753 sentence-transformers/static-similarity-mrl-multilingual-v1 False
SIB200Classification .695 nan .540 .950 codefuse-ai/F2LLM-v2-330M False
SICK-BR-STS .846 nan .782 .918 iara-project/e5-large-matryoshka-sts-pt False
SICK-E-PL .801 nan .760 .932 Gameselo/STS-multilingual-mpnet-base-v2 False
SICK-R-PL .779 nan .751 .942 Gameselo/STS-multilingual-mpnet-base-v2 False
SICKFr .842 nan .792 .862 Gameselo/STS-multilingual-mpnet-base-v2 False
SIDClassification.v2 .649 nan .614 .685 MCINext/Hakim False
SIDClustring .461 nan .387 .548 MCINext/Hakim False
SNLHierarchicalClusteringP2P .658 .614 .559 .651 Salesforce/SFR-Embedding-2_R False
STS22 .720 .718 .637 .831 OrdalieTech/Solon-embeddings-mini-beta-1.1 False
STSBenchmarkMultilingualSTS .899 .887 .832 .955 Gameselo/STS-multilingual-mpnet-base-v2 False
SanskritShlokasClassification .635 nan .739 .842 ai-forever/ru-en-RoSBERTa False
SciFact-Fa.v2 .700 nan .604 .652 google/embeddinggemma-300m False
SciFact-NL .756 nan .684 .802 infly/inf-retriever-v1 False
SciFact-PL .724 nan .658 .776 infly/inf-retriever-v1 False
SpanishNewsClassification.v2 .920 .910 .886 .913 openai/text-embedding-3-large False
SpanishPassageRetrievalS2P .492 .489 .420 .510 jinaai/jina-embeddings-v5-text-nano False
SpanishPassageRetrievalS2S .747 .771 .723 .771 google/gemini-embedding-001 False
SpanishSentimentClassification.v2 .823 .966 .924 .966 google/gemini-embedding-001 False
StyleClassification .572 nan .649 .977 sbunlp/fabert False
SummEvalFr .313 nan .309 .330 intfloat/multilingual-e5-base False
SynPerChatbotConvSAToneChatbotClassification .433 nan .581 .989 MCINext/Hakim False
SynPerChatbotConvSAToneUserClassification .464 nan .526 .973 MCINext/Hakim False
SynPerChatbotRAGFAQPC .780 nan .630 .934 MCINext/Hakim False
SynPerChatbotRAGFAQRetrieval .471 nan .235 .542 MCINext/Hakim False
SynPerChatbotRAGSumSRetrieval .573 nan .498 .796 MCINext/Hakim False
SynPerChatbotSatisfactionLevelClassification .335 nan .252 .598 MCINext/Hakim False
SynPerChatbotSumSRetrieval .350 nan .276 .699 MCINext/Hakim False
SynPerQAPC .917 nan .952 .978 MCINext/Hakim False
SynPerSTS .898 nan .880 .903 Alibaba-NLP/gte-Qwen2-7B-instruct False
SynPerTextKeywordsPC .949 nan .948 .987 MCINext/Hakim False
SynPerTextToneClassification.v3 .646 nan .841 .985 PartAI/TookaBERT-Base False
SyntecReranking .920 nan .854 .940 Alibaba-NLP/gte-Qwen2-1.5B-instruct False
SyntecRetrieval .899 nan .824 .924 intfloat/e5-mistral-7b-instruct False
TenKGnadClusteringS2S .440 nan .308 .437 jinaai/jina-embeddings-v5-text-small False
ToxicChatClassification .833 nan .754 .872 GritLM/GritLM-7B False
TweetSentimentClassification .481 nan .503 .598 google/embeddinggemma-300m False
UrduRomanSentimentClassification .471 nan .416 .620 LingoIITGN/FlowVec-v1 False
WRIMEClassification .270 nan .284 .351 google/embeddinggemma-300m False
WikipediaBioMetChemClassification .997 nan .988 .998 ICT-TIME-and-Querit/BOOM_4B_v1 False
WikipediaBiolumNeurochemClassification .960 nan .957 .985 openai/text-embedding-3-large False
WikipediaChemEngSpecialtiesClassification .739 nan .320 .798 bedrock/cohere-embed-english-v3 False
WikipediaChemistryTopicsClassification .785 nan .846 .937 openai/text-embedding-3-large False
WikipediaChemistryTopicsClustering .550 nan .652 .790 openai/text-embedding-3-large False
WikipediaCompChemSpectroscopyClassification .814 nan .747 .826 VPLabs/SearchMap_Preview False
WikipediaCryobiologySeparationClassification .868 nan .920 .963 bedrock/amazon-titan-embed-text-v1 False
WikipediaCrystallographyAnalyticalClassification .972 nan .930 .986 ICT-TIME-and-Querit/ICT-TIME-and-Querit-embedding-v1 False
WikipediaGreenhouseEnantiopureClassification .972 nan .974 .989 VPLabs/SearchMap_Preview False
WikipediaIsotopesFissionClassification .894 nan .907 .933 openai/text-embedding-3-large False
WikipediaLuminescenceClassification .896 nan .879 .934 bedrock/amazon-titan-embed-text-v1 False
WikipediaOrganicInorganicClassification .919 nan .886 .921 ICT-TIME-and-Querit/ICT-TIME-and-Querit-embedding-v1 False
WikipediaSaltsSemiconductorsClassification .854 nan .855 .924 VPLabs/SearchMap_Preview False
WikipediaSolidStateColloidalClassification .796 nan .787 .855 bedrock/amazon-titan-embed-text-v1 False
WikipediaSpecialtiesInChemistryClustering .274 nan .006 .469 VPLabs/SearchMap_Preview False
XGlueWPRReranking .710 nan .725 .742 google/embeddinggemma-300m False
XPQARetrieval .501 .669 .473 .774 BAAI/bge-multilingual-gemma2 False
XQuADRetrieval .964 nan .971 .971 telepix/PIXIE-Rune-v1.0 False
Average .693 .729 .646 .778 nan -

Model have high performance on these tasks: MalayalamNewsClassification,SpanishNewsClassification.v2,AlloProfClusteringP2P,PubChemSynonymPC,BeytooteClustering,SciFact-Fa.v2,SNLHierarchicalClusteringP2P,PlscClusteringP2P,PlscClusteringS2S,TenKGnadClusteringS2S,BlurbsClusteringS2S,SCIDOCS-Fa.v2

Results for Qwen/Qwen3-Embedding-8B

task_name Qwen/Qwen3-Embedding-8B google/gemini-embedding-001 intfloat/multilingual-e5-large Max result Model with max result In Training Data
AllegroReviews .498 nan .410 .671 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloProfClusteringP2P .748 nan .630 .761 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloProfClusteringS2S .643 nan .323 .645 Alibaba-NLP/gte-Qwen2-7B-instruct False
AlloprofRetrieval .667 nan .393 .738 mongodb/voyage-3 False
ArguAna-Fa.v2 .596 nan .413 .585 google/embeddinggemma-300m False
ArguAna-NL .633 nan .489 .740 infly/inf-retriever-v1 False
Assin2STS .827 nan .783 .841 iara-project/e5-large-matryoshka-sts-pt False
BSARDRetrieval .743 nan .631 .829 mongodb/voyage-3 False
BengaliSentimentAnalysis .862 nan .831 .920 LingoIITGN/FlowVec-v1 False
BeytooteClustering .703 nan .615 .677 Alibaba-NLP/gte-Qwen2-7B-instruct False
BlurbsClusteringS2S .233 nan .168 .220 Linq-AI-Research/Linq-Embed-Mistral False
CBD .783 nan .673 .841 BAAI/bge-multilingual-gemma2 False
CDSC-E .745 nan .745 .799 sdadas/mmlw-roberta-large False
CDSC-R .917 nan .910 .976 Gameselo/STS-multilingual-mpnet-base-v2 False
ChemHotpotQARetrieval .868 nan .798 .953 infly/inf-retriever-v1 False
ChemNQRetrieval .625 nan .662 .705 intfloat/multilingual-e5-small False
DeepSentiPers.v2 .619 nan .577 .723 MCINext/Hakim False
DigikalamagClassification .881 nan .871 .910 MCINext/Hakim False
DigikalamagClustering .659 nan .399 .791 MCINext/Hakim False
EightTagsClustering .480 nan .265 .503 BAAI/bge-multilingual-gemma2 False
FalseFriendsGermanEnglish .565 nan .534 .712 aari1995/German_Semantic_STS_V2 False
FarsiParaphraseDetection .972 nan .976 .996 MCINext/Hakim False
Farsick .762 nan .707 .770 jinaai/jina-embeddings-v3 False
GermanDPR .845 nan .829 .856 malteos/most-embed-de False
GermanQuAD-Retrieval .962 nan .932 .971 nvidia/Nemotron-3-Embed-1B-BF16 False
HALClusteringS2S .297 nan .224 .308 Alibaba-NLP/gte-Qwen2-7B-instruct False
HamshahriClustring .740 nan .674 .756 myrkur/sentence-transformer-parsbert-fa False
HateSpeechPortugueseClassification .640 nan .573 .628 intfloat/e5-mistral-7b-instruct False
HindiDiscourseClassification .412 nan .387 .507 Sailesh97/Hinvec False
JSTS .862 nan .818 .861 NGA-KR/ja-embed-sts False
JaGovFaqsRetrieval .731 nan .724 .790 sionic-ai/comsat-embed-ja-0.3b-preview False
JapaneseSentimentClassification .835 nan .858 .947 sbintuitions/sarashina-embedding-v2-1b False
LEMBSummScreenFDRetrieval .984 nan .711 .985 DataScience-UIBK/uibk-embedding-v1 False
LEMBWikimQARetrieval .929 nan .568 .999 lightonai/GTE-ModernColBERT-v1 False
LegalBenchConsumerContractsQA .858 nan .733 .878 minetta/nemotron-3-embed-8b-legal False
LivedoorNewsClustering.v2 .515 nan .565 .606 cl-nagoya/ruri-v3-130m False

Note: Content truncated due to GitHub API limits. See the full report in the workflow artifacts.

@KennethEnevoldsen

KennethEnevoldsen commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

Quite a few scores don't reproduce here. @ayush1298 can I ask you to run them as well just to see if it is just me getting a different score?

Samoed
Samoed reviewed Oct 19, 2026
View reviewed changes

@Samoed Samoed left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems biggest gap in

    microsoft/harrier-oss-v1-270m/STSBenchmarkMultilingualSTS: The difference between the current score (0.837302) and the previous (0.809034) exceeds threshold of 0.001
    microsoft/harrier-oss-v1-270m/STSBenchmarkMultilingualSTS: The difference between the current score (0.811503) and the previous (0.780971) exceeds threshold of 0.001
    Qwen/Qwen3-Embedding-8B/XQuADRetrieval: The difference between the current score (0.97014) and the previous (0.01663) exceeds threshold of 0.001
    Qwen/Qwen3-Embedding-8B/MTOPIntentClassification: The difference between the current score (0.865823) and the previous (0.018662) exceeds threshold of 0.001
    Qwen/Qwen3-Embedding-8B/MTOPIntentClassification: The difference between the current score (0.854638) and the previous (0.015739) exceeds threshold of 0.001
    Qwen/Qwen3-Embedding-4B/PublicHealthQA: The difference between the current score (0.84706) and the previous (0.86926) exceeds threshold of 0.001
    Qwen/Qwen3-Embedding-0.6B/STS22: The difference between the current score (0.708118) and the previous (0.711317) exceeds threshold of 0.001
    Qwen/Qwen3-Embedding-0.6B/TweetSentimentClassification: The difference between the current score (0.437109) and the previous (0.441406) exceeds threshold of 0.001

Other results seems close enough. Results for qwen3-8b model for MTOPIntentClassification were added in #428. Overall results for qwen 8b in that PR looks strange

@ayush1298

ayush1298 commented Oct 19, 2026

Copy link
Copy Markdown
Collaborator

Quite a few scores don't reproduce here. @ayush1298 can I ask you to run them as well just to see if it is just me getting a different score?

will run tomorrow or day after it.

@ayush1298

ayush1298 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

@KennethEnevoldsen I was able to reproduce every number that you had, which was different in previous runs, maybe bugs in older runs or something. So, we can merge with your results.

@KennethEnevoldsen

KennethEnevoldsen commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @ayush1298!

Should we consider removing all Qwen/Qwen3-Embedding-8B from that #428?

@Samoed are you fine with merging this?

Samoed
Samoed approved these changes Oct 20, 2026
View reviewed changes

@Samoed Samoed left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can just rerun other tasks from that pr. I think we can merge this

@ayush1298

ayush1298 commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

Should we consider removing all Qwen/Qwen3-Embedding-8B from that #428?

yes, maybe we can run fully.
Also, heads-up, I am running some of the Jina models, and will add PR soon

@KennethEnevoldsen
KennethEnevoldsen merged commit 54c7a49 into main Oct 21, 2026
3 of 4 checks passed
@KennethEnevoldsen
KennethEnevoldsen deleted the coverage-5589 branch October 21, 2026 20:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Samoed Samoed Samoed approved these changes
@ayush1298 ayush1298 Awaiting requested review from ayush1298

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@KennethEnevoldsen @ayush1298 @Samoed