RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad().

[Bartender-bit]

Hello, I am wondering if anyone has encountered this problem.

When I replace the "LIFSqueeze" in the code:

nn.Conv2d(1, 32, kernel_size=5, stride=2, padding=2),
LIFSqueeze(tau_mem=20.0, batch_size=batch_size),
nn.Dropout2d(p=dropout_p)

with "IAFSqueeze"

nn.Conv2d(1, 32, kernel_size=5, stride=2, padding=2),
IAFSqueeze(
           batch_size=batch_size,
           num_timesteps=1,
           spike_fn=spike_fn,
           surrogate_grad_fn=spike_grad,
           reset_fn=spike_reset,
           tau_syn=None,
           spike_threshold=1,
           min_v_mem=-1.0,
           record_states=True
),
nn.Dropout2d(p=dropout_p)

the following error occurs:

Traceback (most recent call last):
  File "/home/Projects/EventCamera/GENX320/scripts/chat_gpt.py", line 391, in <module>
    main(pretrained=False)
  File "/home/Projects/EventCamera/GENX320/scripts/chat_gpt.py", line 380, in main
    train_loss = train_epoch(model, train_loader, optimizer, loss_fn, device)
  File "/home/Projects/EventCamera/GENX320/scripts/chat_gpt.py", line 325, in train_epoch
    loss.backward()
  File "/home/anaconda3/envs/retina/lib/python3.10/site-packages/torch/_tensor.py", line 648, in backward
    torch.autograd.backward(
  File "/home/anaconda3/envs/retina/lib/python3.10/site-packages/torch/autograd/__init__.py", line 353, in backward
    _engine_run_backward(
  File "/home/anaconda3/envs/retina/lib/python3.10/site-packages/torch/autograd/graph.py", line 828, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "/home/anaconda3/envs/retina/lib/python3.10/site-packages/torch/autograd/function.py", line 307, in apply
    return user_fn(self, *args)
  File "/home/anaconda3/envs/retina/lib/python3.10/site-packages/sinabs/activation/spike_generation.py", line 11, in backward
    (v_mem,) = ctx.saved_tensors
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

I followed the instructions and added loss.backward(retain_graph=True), but another error occurred:

(Triggered internally at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:122.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/home/xixu/Projects/EventCamera/GENX320/scripts/chat_gpt.py", line 392, in <module>
    main(pretrained=False)
  File "/home/xixu/Projects/EventCamera/GENX320/scripts/chat_gpt.py", line 381, in main
    train_loss = train_epoch(model, train_loader, optimizer, loss_fn, device)
  File "/home/xixu/Projects/EventCamera/GENX320/scripts/chat_gpt.py", line 326, in train_epoch
    loss.backward(retain_graph=True)
  File "/home/xixu/anaconda3/envs/retina/lib/python3.10/site-packages/torch/_tensor.py", line 648, in backward
    torch.autograd.backward(
  File "/home/xixu/anaconda3/envs/retina/lib/python3.10/site-packages/torch/autograd/__init__.py", line 353, in backward
    _engine_run_backward(
  File "/home/xixu/anaconda3/envs/retina/lib/python3.10/site-packages/torch/autograd/graph.py", line 828, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [32, 1, 5, 5]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Thanks for any suggestions!!!