About evaluate and activate_subnet

[shijian971]

When I use the following code.

trainer.compression_ctrl.multi_elasticity_handler.activate_subnet_for_config(config)
validate_model_fn(trainer.model, eval_loader)

The results are the same during the evaluation regardless of different config.

The same issue will also occur when using the following code for evaluation.

def evaluate_sst2(model, valid_loader):

  model.eval()
  correct = 0
  total = 0
  with torch.no_grad():
      for batch in valid_loader:
          input_ids_sentence = batch['input_ids_sentence'].to(model.device)
          attention_mask_sentence = batch['attention_mask_sentence'].to(model.device)
          labels = batch['label'].to(model.device)

          outputs = model(
              input_ids_sentence,
              attention_mask=attention_mask_sentence
          )
          _, predicted= torch.max(outputs.logits, 1)

          correct += (predicted == labels).sum().item()
          total += labels.size(0)

  acc = correct / total
  return acc
evaluate_sst(trainer.model, valid_loader)

How to solve the issue? or How to check if the activation of the subnet has been completed?

[Yuan0320]

Hi @shijian971, to better assist you, could you please provide a more complete version of your code? This will help us understand the details of your implementation.

Additionally, before calling trainer.compression_ctrl.multi_elasticity_handler.activate_subnet_for_config(config), did you call trainer.compression_ctrl.multi_elasticity_handler.enable_all() (as shown here)? If not, you can try this. Thanks!

[shijian971]

if training_args.do_search and nncf_config is not None:

    logger.info("*** Search ***")
    trainer.compression_ctrl.multi_elasticity_handler.enable_all()
    search_algo = BaseSearchAlgorithm.from_config(trainer.model, trainer.compression_ctrl, nncf_config)

    eval_loader = trainer.get_eval_dataloader(eval_dataset)
    key = "eval_accuracy"
    if data_args.task_name == "cola":
        key = "eval_matthews_correlation"
    elif data_args.task_name == "stsb":
        key = "eval_pearson"

    def validate_model_fn(model_, loader_):
        metrics = trainer.evaluation_loop(loader_, model=model_, description="search eval").metrics
        return metrics[key]

    trainer.compression_ctrl.multi_elasticity_handler.activate_maximum_subnet()
    if search_algo.bn_adaptation is not None:
        search_algo.bn_adaptation.run(trainer.model)
    top1_acc = validate_model_fn(trainer.model, eval_loader)
    logger.info(
        "Maximal subnet Top1 acc: {top1_acc}, Macs: {macs}".format(
            top1_acc=top1_acc,
            macs=trainer.compression_ctrl.multi_elasticity_handler.count_flops_and_weights_for_active_subnet()[0]
            / 2000000,
        )
    )

    import pandas as pd
    data = pd.read_csv('./lonas-bert-base-glue/lonas-bert-base-sst2/search_progression.csv',
                       header=None)
    width_column = data[0]
    accs = data[4]
    macs = data[2]

    from enum import Enum

    class ElasticityDim(Enum):
        """
        Defines elasticity dimension or type of elasticity applied to the model
        """

        KERNEL = "kernel"
        WIDTH = "width"
        DEPTH = "depth"

    def ODtoD(item):

        item = item.split('{', 1)[1].rsplit('})]', 1)[0]
        parsed_item = ast.literal_eval('{' + item + '}')
        return parsed_item

    #for i in range(len(width_column)):
    for i in [0,30,100]:
        subnet_config = ODtoD(width_column[i])
        subnet_config_1 = {ElasticityDim.WIDTH: subnet_config}
        print(f'subnet_config_1:{subnet_config_1}')
        print(f'subnet:{subnet_config}')
        #trainer.compression_ctrl.multi_elasticity_handler.enable_all()
        trainer.compression_ctrl.multi_elasticity_handler.activate_subnet_for_config(subnet_config_1)
        print(validate_model_fn(trainer.model, eval_loader))
        print(f"{i}_finished*******")

Thank you for your reply.
I have made modifications to the search section of run_glue.py. I want to activate the searched configurations for evaluation.

python run_glue.py --task_name sst2 --model_name_or_path bert-base-uncased --do_eval --do_search --per_device_eval_batch_size 64 --max_seq_length 128 --lora --lora_weights lonas-bert-base-glue/lonas-bert-base-sst2 --nncf_config nncf_config/glue/nncf_lonas_bert_base_sst2.json --output_dir lonas-bert-base-glue/lonas-bert-base-sst2/results_12_9

[shijian971]

@Yuan0320

[Yuan0320]

@shijian971 Thanks for providing the code.

After reviewing your code, the simplest fix to resolve the issue is to replace your custom ElasticityDim(Enum) with the following import:

from nncf.experimental.torch.nas.bootstrapNAS.elasticity.elasticity_dim import ElasticityDim

The reason for this change is that the ElasticityDim class you defined and the one provided by NNCF are different classes. The activate_subnet_for_config method only recognizes the ElasticityDim class from the NNCF library. By using the correct import, we need to ensure that the method can properly identify and activate the subnet configuration. Thanks.

[shijian971]

@Yuan0320 thanks，After debugging, it was found that the problem stemmed from the type of the config passed in. Currently, this problem has been solved.