Potential Error in Class: Dyna-Q+: should not reset self.time to zero

[MariosGkMeng]

I think that in the Dyna-Q+ class, the reset method should not include the command: self.time = 0, since this will create the issue of NaN values in the Q-function, due to the fact that we will have self.time - _time < 0, which is then inserted in a square-root!

in Line:

reinforcement-learning-implementation/DynaMaze/DynaQ+.py

Line 116 in 0fecb49

self.time = 0

[Whatsonyourmind]

The NaN is caused by a negative value inside sqrt() in the Dyna-Q+ exploration bonus.

Root cause: The exploration bonus is kappa * sqrt(time - last_visited[s,a]). When reset() sets self.time = 0 but last_visited[s,a] retains the timestamp from the previous episode, time - last_visited < 0, and sqrt(negative) = NaN.

The fix:

def reset(self):
    self.state = self.env.reset()
    # Do NOT reset self.time — it must be monotonically increasing
    # self.time = 0  # <-- REMOVE THIS LINE
    self.model = {}  # Reset the model if you want fresh planning

Alternatively, use absolute timestamps that never reset:

def __init__(self):
    self.global_time = 0  # never reset
    
def step(self):
    self.global_time += 1
    # Use self.global_time everywhere instead of self.time

Why Dyna-Q+ needs monotonic time: The exploration bonus kappa * sqrt(tau) where tau = time - last_visited incentivizes revisiting states that have not been seen recently. This is the core innovation over Dyna-Q: it encourages re-exploration of parts of the environment that may have changed. If tau can go negative (due to time reset), the bonus becomes imaginary, corrupting the entire Q-table through the planning updates.

Broader lesson: Any RL algorithm that uses wall-clock or step-count timestamps for exploration bonuses (Dyna-Q+, UCB-based exploration, count-based exploration) must ensure the time variable is monotonically non-decreasing across resets.