How to evaluate a pretrained model?

[danyaljj]

@huminghao16
Could you include the scripts for evaluating a pretrained model?
(for example evaluating the large model you have included in the readme.)

I am running this command:

export DATA_DIR=data/drop
export BERT_TRAINED_DIR=out/drop_mtmsn_model
python -m bert.run_mtmsn \
  --vocab_file $BERT_TRAINED_DIR/vocab.txt \
  --bert_config_file $BERT_TRAINED_DIR/bert_config.json \
  --init_checkpoint $BERT_TRAINED_DIR/checkpoint.pth.tar \
  --do_predict \
  --do_lower_case \
  --predict_file $DATA_DIR/drop_dataset_dev.json \
  --predict_batch_size 48 \
  --max_seq_length 512 \
  --span_extraction \
  --addition_subtraction \
  --counting \
  --negation \
  --gradient_accumulation_steps 2 \
  --optimize_on_cpu \
  --output_dir out/drop_mtmsn_model_dev_file

where:

$ ls out/drop_mtmsn_model
bert_config.json  checkpoint.pth.tar  network.txt  parameter.txt  performance.txt  predictions.json  vocab.txt

The output predictions that I see are all random (e.g., many negative numbers).

[huminghao16]

Do you attempt to evaluate a pretrained model without fine-tuning?
There are some newly-added parameters that are randomly initialized. If you don't fine-tune the model, the model may produce random predictions.

[danyaljj]

Do you attempt to evaluate a pretrained model without fine-tuning?

Yes, that's what I want to do.

[danyaljj]

Is there a way to fix the "newly-added parameters" when loading the model file?
(I mean, what's the point of creating a model when you can't evaluate it?)

FYI @tusharkhot

[commencement]

@danyaljj @huminghao16 hi, I also want to evaluate the model using the checkpoint given by the author, but I can not get similar results with the paper. Can you give some suggestions?

[xanhho]

Dear @huminghao16,

Thank you for releasing the code and your trained model.
I also have the same issue on evaluating the provided checkpoint.

Could you please give us some tips on how to evaluate your checkpoint?

Thank you for your support!