Request for guidance reproducing Biomedical Multi-label Classification results (sem_join + optimization)

[impression393-hash]

Hi LOTUS team—thank you for the excellent work and for releasing the code! I’m trying to reproduce the Biomedical Multi-label Classification evaluation (Tables 3 & 4) and I may be misconfiguring something.

Bug Description

My run triggers many more sem_filter LLM calls than reported, and I’d be grateful for your guidance.
My setup
Data: 250 articles × ~24,300 reactions ≈ ~6.0M pairs
Operator: sem_join with optimization (§3.2)
LM: meta-llama/llama-3.3-70b-instruct
RM (embeddings): intfloat/e5-base-v2
map_instruction: default (not provided)
join_instruction: "The article {medical_article} indicates that the patient is experiencing the {reaction}"
cascade_args: CascadeArgs(recall_target=0.7, precision_target=0.7, sampling_percentage = 0.0001)

I first ran sem_join with optimization and completed threshold learning.
After thresholding, 24,852 low-confidence pairs remained, which then require sem_filter.
This implies 24,852 sem_filter LLM calls, whereas the paper reports 5,290 for the complete join process. I likely missed a configuration or optimization detail.

Expected Behavior

My understanding is that, after threshold learning and the approximation step, the number of low-confidence pairs should be much smaller than 24, 852—is that correct?

Questions

Could you please share the exact experimental settings used to reproduce the results in Tables 3, 4, and 5? I’d also appreciate any guidance on what I might be doing incorrectly, and whether there are additional steps to further reduce LLM calls in sem_filter.

Futher context

"msf_t_pos": 0.8734673261642456,
"msf_t_neg": 0.8317583799362183,
"num_helper_high_conf_neg": 6043330,
"join_optimization_cost": 857
Python Version:
3.10

Package Versions:

Checklist

• I have searched existing issues to avoid duplicates
• I have provided all required information
• I have tested with the latest version of the package
• I have included a minimal reproduction example (if applicable)
  Checked

[liana313]

Hi @impression393-hash, thanks for the question. I took a quick look at your setup -- I believe we used Llama-3-70B, accuracy targets of 0.9, and failure probabilities of 0.2. These are small details, but could account for some discrepancy. We will merge the original scripts for reproducibility

[impression393-hash]

Hi @impression393-hash, thanks for the question. I took a quick look at your setup -- I believe we used Llama-3-70B, accuracy targets of 0.9, and failure probabilities of 0.2. These are small details, but could account for some discrepancy. We will merge the original scripts for reproducibility

Thank you for your reply~

For my run, I set:
cascade_args = CascadeArgs(recall_target=0.9, precision_target=0.9, sampling_percentage = 0.0001)
From the code it looks like failure_probability defaults to 0.2, so I kept that unchanged. Below is the full CascadeArgs object passing to learn_join_cascade_threshold():
CascadeArgs(recall_target=0.9, precision_target=0.9, sampling_percentage=0.0001, failure_probability=0.2, map_instruction=None, map_examples=None, proxy_model=<ProxyModel.HELPER_LM: 'helper_lm'>, cascade_IS_weight=0.9, cascade_num_calibration_quantiles=50, min_join_cascade_size=100, cascade_IS_max_sample_range=200, cascade_IS_random_seed=None)

I’ll also retry with Llama-3-70B as suggested. May I kindly confirm the exact model ID you used — was it meta-llama/Meta-Llama-3-70B?

One small concern: I’m not sure the 70B llama model choice alone would account for a ~4–5× increase in LLM calls (24,852 for sem_filter + 250 for projection + 607 for threshold learning ≈ 25,709 vs. 5,290). Do you think there might be other configuration differences at play? May I also confirm whether your embedding model was intfloat/e5-base-v2, the same as mine?

[parkervg]

Related issue here: #227

To confirm, is sc/ranking_directory the branch we should be going to for the biomedical classification code used in the paper?