Video-enabled find_avoid example project

[TarekSaati]

Video-enabled example from deepworlds repository

This is an example implementing video logging to tensorboard to monitor agent performance during trainig in Webots simulator.

• Here are the changes I did to training.py script in "find_and_avoid_v2" example project in deepworlds dev banch.

Todo:

1- Import Video from logger:
in training script:

from stable_baselines3.common.logger import HParam, Video

2- Add necessary variables to AdditionalInfoCallback class:
add in the constructor:

        self.frames = []  # List to store frames for GIF creation
        self.episode_cnt = 1
        self.record = False
        self.render_interval=render_interval

3- Implement the on_step() event handler to record frames periodically:

if self.env.done:
            if self.episode_cnt % self.render_interval == 0:
                self.env.camera.enable(self.env.timestep * 10) # basic time step = 32
                self.record = True
            else:
                self.env.camera.disable()
                self.record = False
            self.episode_cnt+=1
            self.frames = []
            print(f'Starting Episode {self.episode_cnt} ...')  

4- Add video creation and logging to tensorboard to on_rollout_end() event:
if self.record:

            # Save the frames to tensorboard
            frame = self.env.render(mode='rgb_array') # (c, h, w)
            self.frames.append(frame) 
            video = np.asarray([self.frames])
            self.logger.record("visualization",
                                Video(torch.from_numpy(video), fps=30),
                                exclude=("stdout", "log", "json", "csv"))

5- Add render parameters to run() function:

run(... ,
        log_interval=4,
        render_interval=100)

Results:

"Visualization" tab is added to Tensorboard monitoring & is updated after a predefined sequence of episodes, hence more ability to track the agent performance across different difficulty levels.

[tsampazk]

Hey @TarekSaati, thank you! Could you add a couple of screenshots that show the resulting visualization?

If you think it can prove generally useful for running and monitoring the find and avoid project, you could open a pull request with the suggested changes and integrate it in the dev branch for others to easily use. I suppose it can be very useful for people running webots in headless mode.

[TarekSaati]

Hey @tsampazk and thank you for highlighting this feature. Yes of course I am willing to open a pull request for it shortly. I have introduced a video-enabled DQN example based on find_and_avoid example in an RL workshop run by students from a local GPU server in headless mode a few weeks ago and the results were of a great help!
Kind regards.

[tsampazk]

Please correct me if i am wrong, so something similar to adding an RGB camera on the robot and using a convolutional model for the agent? Consequently, this feature adds the camera video stream on tensoboard.

It's amazing and i am very happy that deepbots and this specific example helped you and the students in their work! Do you have access to the accompanying paper?

@eakirtas

[TarekSaati]

Exactly @tsampazk , I actually implemented the self.env.render(mode='rgb_array') function to get image from a first-person-view camera I added to the robot. However, I used the same lidar data in DQN training but as you said it is straightforward now to apply CNN based visual navigation to solve the tasks. By the way, I am already working on vision & language navigation (VLN) in another simulator through my PhD!
Thank you for introducing DeepWorlds. I hope for it to be a standard in RL tutorials soon!

P.S. I did not come across the paper. If there is one, I am pleased to read it!

[tsampazk]

That's very interesting @TarekSaati! We were actually thinking towards extending this example with cameras, but never got around to doing much about it. You can find the related paper here. It details the methodology of the original work, including on how the action mask works and how it helps. There is also an example that uses an on board camera for target acquisition only (it is not used in the input of the agent).

Thank you very much for your kind words! There is some work to be done to update deepbots, which hopefully i will have the time to work on in the near future.

[TarekSaati]

That is great to hear! Early in 2020 in my MSc, I developed DQN agent in Matlab to navigate in dynamic indoor scenes with Lidar & odometry as inputs and tested my work in Webots (ResearchGate link). However, I wished Deepbots was there in that time so training would be easier and more effective.. What a time to revisit training in Webots! Thanks again for your work.