Upgrading transformers version to 4.51.3 to support recent models

[filyp]

What does this PR do?

• It upgrades transformers to 4.51.3, to support newer models like gemma3 and qwen3
• Makes UnlearnTrainer implementation more future proof. (See Upgrading transformers version to support recent models #173)
• Makes other necessary changes to be compatible with new transformers version
  • Adds num_items_in_batch to compute_loss signature
  • Prevent exceptions from evaluating when eval_dataset=None
  • Uses trainer.processing_class instead of trainer.tokenizer (it's depracated and transformers==5 removes it)

Additionally it:

• Simplifies the installation of lm_eval (no need to have a special install group in setup.py)
• Minor fix to the leaderboard doc
• Makes .gitignore more comprehensive

Related issues: #173 and #155

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Have you gone through the contributions guide?
• Are your changes documented? Read documentation guidelines here.

Other notes

I manually tested the new evaluation, with removed prediction_step from UnlearninTrainer, and it works the same.

When I tested unlearning, the overall unlearning trajectory is very similar, but there are some tiny changes to the logged training loss, likely due to a difference in how new transformers aggregates it. (Not sure though, but the differences were small.)