Pretraining loss used in the code

[Egiob]

Hello and thanks for making all this code public!
I have a question regarding the pretraining loss actually used in the training, I understand through looking at the pretraining code (on branch dev-temp) and at the paper that two losses are being used:

• a gene expression prediction MSE loss, from the output of the transformer stack
• a second gene expression prediction loss, only from the cell embedding and the original sequence embedding

What I don't get is, as far as I understand, a third loss is being added to these two ones after the variable global_iter gets higher than 1000. Here is the link to the code snippet i'm referring to: https://github.com/bowang-lab/scGPT/blob/4068d67caaac1e28d56964da68e0214817e38428/examples/pretrain.py#L894C1-L909C70.

Could you comment on this additional loss? Please correct me if I misunderstood the paper or the code and thanks again!

[subercui]

Hi, thank you for the question! The code piece you refer to runs the training for the "cell-prompt" generation as described in the paper. Intuitively, the model learns to reconstruct the gene expressions conditioned additionally on the cell embedding, in other words, conditioned on the information describing what the cell is. During training, we use the cell embedding output from the previous step. You can find the description of "gene-prompt" and "cell-prompt" generations in the manuscript section 4.3.2 and how we trained these two modes consecutively in section 4.3.3. I hope these explanations make it more clear to you

[Egiob]

Hello, and thanks for the answer and the clarification. There is still something I don't get. As far as I understand the MVC argument is set to True is the pretraining script. This means that 3 losses are actually being used: the first-step prediction error, the MVC prediction error and the second-step prediction error. I don't really get why we would need the second-step loss if we already have the MVC loss and vice-versa. Could you clarify the role of these two last losses please?

Thank you very much!

[chansigit]

Hi, thanks for the great work and for the earlier answers in this thread. Could you clarify, specifically for the pretraining stage (not fine-tuning), exactly which loss terms are used and how they are weighted relative to each other?

From examples/pretrain.py on the dev-temp branch, together with the released checkpoint config (MVC=true, USE_CLS=true, no_cce=true, USE_GENERATIVE_TRAINING=true), the total pretraining loss appears to be the sum of four terms, each added with unit weight:

1. the gene-prompt / first-step MSE (GEP),
2. the MVC MSE,
3. the cell-type classification cross-entropy (CLS), and
4. the cell-prompt / second-step MSE (GEN), added after global_iter > 1000.

My questions:

1. Is this the actual loss composition used to train the released whole-human checkpoint, or were some terms disabled or weighted differently?
2. What are the relative weights among these terms during pretraining? The code seems to add them 1:1 (CCE would be 10x but is disabled) — is that what was used?
3. The paper (Methods 4.4) lists MVC and CLS as fine-tuning objectives, yet they appear enabled during pretraining in the code. Were CLS and MVC really part of the pretraining objective?
4. As already raised above, what is the intended, separate role of the cell-prompt (GEN) loss and the MVC loss, given that both reconstruct expression from the cell embedding?

Thanks a lot!