diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..19aec8e --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +English/RL\ Tutorial\ 2/Deep\ Q\ Learning-Update.ipynb diff --git a/English/RL Tutorial 2/.ipynb_checkpoints/Deep Q Learning-checkpoint.ipynb b/English/RL Tutorial 2/.ipynb_checkpoints/Deep Q Learning-checkpoint.ipynb deleted file mode 100644 index 9581bcb..0000000 --- a/English/RL Tutorial 2/.ipynb_checkpoints/Deep Q Learning-checkpoint.ipynb +++ /dev/null @@ -1,1658 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Reinforcement Learning Tutorial -2: DQN" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### MD Muhaimin Rahman\n", - "contact: sezan92[at]gmail[dot]com" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the last tutorial, I tried to explain Q-learning algorithm. The biggest problem with Q-learning is that,it only takes discrete inputs and outputs discrete values. In the Mountain Car problem, we solved this issue by discretizing states which are actually continuous. But this can't be done always. Specially when the states are multidimensional states or images. Deep learning comes here to solve this problem!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For example breakout game by atari" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![](Atari-breakout.jpg)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here , the states are the actual image itself! It is 210x160x3 size RGB numpy array. How will you make discrete for $Q learning$ ? Will that be efficient ? ***NO***! . DQN comes us to save us!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Intuition" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Deep Q Learning" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Q learning is a lookup table problem. i.e. You have the state , you just look at the table and see which action gives you best $Q$ value! That's it. But for continuous state - as mentioned above- you cannot make a lookup table! You need something like a regression model! Which will give you the Q values for given state and action! And the best regression model would be, Deep Neural Network!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "So, we will replace the Q table explained in the last tutorial with a Neural Network. i.e. some thing like the following picture" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![](Q2DQN.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***But there is one little problem!***" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In our mountain car problem, we have three discrete actions. $0,1 & 2$ . Using the above architecture, we will have to calculate $Q$ value for each action . Because , you need to take the action with best $Q$ value. To get the action of the best $Q$ value, you need to know the $Q(state,action)$ for each state! So in our case, we will have to run same feed forward process three times!" - ] - }, - { - "cell_type": "raw", - "metadata": {}, - "source": [ - "action_list =[0,1,2]\n", - "Q1 = model.predict(state,action_list[0])\n", - "Q2 = model.predict(state,action_list[1])\n", - "Q3 = model.predict(state,action_list[2])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "What if we have more 100 actions ? Will we feed forward 100 times! It is a bit inefficient!! Instead, we will use the following architecture." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![](NewDQN.png)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Meaning, our output layer will calculate $Q$ value for each action. As a result we can calculate Q values in one single forward pass each step! And then we will choose the action with maximum value" - ] - }, - { - "cell_type": "raw", - "metadata": {}, - "source": [ - "action = np.argmax(Q(state))" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Bellman Update Equation" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the Original Equation, the bellman update equation is \n", - "\\begin{equation}\n", - "Q(s_t,a) = Q(s_t,a) + \\alpha (Q'(s_{t+1},a)-Q(s_t,a))\n", - "\\end{equation}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For DQN, we will use similar equation, using Gradient descent\n", - "\\begin{equation}\n", - "\\theta_Q \\gets \\theta_Q - \\alpha \\frac{\\partial}{\\partial \\theta}(Q'(s_{t+1},a)-Q(s_t,a))^2\n", - "\\end{equation}" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "If you are intelligent enough, then you may ask , why there is a squareed part in the gradient descent equation but not in the actual bellman update equation? The reason might be, Mean squared errors are more sensitive to sudden spikes in the target data, which makes it most popular metric for regression models!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### The Concept of Experience Replay" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "One of the problems in Reinforcment learning is relearning Problem. That is , suppose, in the course of trial and error, one state $s_t$ comes only once or twice, and never comes back. What will happen? There is a chance that the Agent will forget that experience after some time- like us! . So we need to make the agent keep some kind of track of that memory as well. This problem was solved in 1993- yes 26 years ago- by Long Ji lin. In his paper, ***Reinforcement Learning for Robots Using Neural Networks*** , he introduced the concept of Experience Replay. What he did was, he initialized a buffer of a certain size . He stored the experiences of the agent, i.e. state $s_t$, action $a$,next state $s_{t+1}$, reward $r$ . Before training the agent, he you just sample randomly from the buffer . It also helps randomizing the data , which in turn, helps to converge the model faster, as mentioned by Yoshua Bengio in his paper ***Practical Recommendations for Gradient-Based Training of Deep\n", - "Architectures***,2012 " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### The concept of $\\epsilon$-greedy Policy" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "In the beginning of training, we will have to explore random actions. Because we dont know the value of each action for each state. So we will take some random actions. We will evaluate those actions and see which random action gives us the most rewards. We will try to increase those actions. It means, at first you just ***explore*** different actions , the more you take actions the less you explore and more use your previouse experience to guide you-known as ***exploit***. This thing can be done using a technique -with another freaking out name- $\\epsilon$-greedy policy. " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "The big idea is that, we will select a value of $\\epsilon$ , suppose $0.9$. Then we will generate a random floating point number . If the generated number is greater than $\\epsilon$ we will take action according to the DQN, otherwise a random action. After each episode , we will decrease the value of $\\epsilon$ . As a result , in the last episodes, the agent will take actions according to DQN model, not the random actions." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- Set $\\epsilon$\n", - "- Generate random number $n_{rand}$\n", - "- if $n_{rand} < \\epsilon$ ***do***\n", - "- - take random action\n", - "- else ***do***\n", - "- - take action according to DQN" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Coding!!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Okay, let's start the most juicy part!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***Importing Libraries***" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/home/sezan92/anaconda/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", - " from ._conv import register_converters as _register_converters\n", - "Using TensorFlow backend.\n" - ] - } - ], - "source": [ - "import gym\n", - "import numpy as np\n", - "from collections import deque\n", - "import random\n", - "import keras\n", - "from keras.models import Sequential\n", - "from keras.layers import Dense\n", - "from keras.optimizers import Adam\n", - "import matplotlib.pyplot as plt\n", - "from sklearn.preprocessing import OneHotEncoder" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***Initialization of Environment***" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n" - ] - } - ], - "source": [ - "env = gym.make('MountainCar-v0')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***Hyper parameters***" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- ```action_size``` number of actions\n", - "- ```actions``` the actions list\n", - "- ```gamma``` discount factor $\\gamma$\n", - "- ```lr``` learning rate $\\alpha$\n", - "- ```num_episodes``` number of episodes\n", - "- ```epsilon``` epsilon , to choose random actions for epsilon greedy policy \n", - "- ```epsilon_decay``` epsilon decay rate\n", - "- ```batch_size``` batch size for training" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "action_list = [0,1,2]\n", - "gamma =0.45\n", - "lr =0.001\n", - "num_episodes =1000\n", - "epsilon =1\n", - "epsilon_decay =0.995\n", - "memory_size =1000\n", - "batch_size=100\n", - "show=False\n", - "action_size=env.action_space.n\n", - "state_size=env.observation_space.shape[0]" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "factor=[1,100]" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Initializing Replay buffer for ***Experience Replay***" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- ```memory``` a deque -which is a special type of list with limited memory- the replay buffer\n", - "- ```s``` current state\n", - "- ```a``` action\n", - "- ```new_s``` new state\n", - "- ```r``` reward\n", - "- ```d``` terminal\n", - "- ```experience``` tuple of state,reward,action,next state\n" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***Psuedocode*** for experience replay" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- get initial state $s$\n", - "- for each iteration do\n", - "- - take a random action $a$\n", - "- - get next state $s_{next}$, reward $r$, terminal $d$ \n", - "- - $s \\gets s_{next}$\n", - "- - if environment is terminated do\n", - "- - - reward $\\gets$ -100\n", - "- - - reset environment\n", - "- - - add state,reward,action,next state into replay buffer\n" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [], - "source": [ - "memory=deque(maxlen=memory_size)\n", - "s=env.reset()\n", - "s = s.reshape((1,-1))\n", - "s = s*factor\n", - "for _ in range(memory_size):\n", - " a=env.action_space.sample()\n", - " new_s,r,d,_ =env.step(a)\n", - " new_s = new_s.reshape((1,-1))\n", - " new_s = new_s*factor\n", - " if show:\n", - " env.render()\n", - " if d:\n", - " r=-100\n", - " experience =(s,r,a,new_s,d)\n", - " s=env.reset()\n", - " s = s.reshape((1,-1))\n", - " else: \n", - " experience =(s,r,a,new_s,d)\n", - " memory.append(experience)\n", - " s = new_s\n", - "env.close() " - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***Model Definition***" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here , I have defined the model as a simple MLP neural network with 2 hidden layers of 100 nodes with ```relu``` activation function" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "WARNING:tensorflow:From /home/sezan92/anaconda/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n", - "Instructions for updating:\n", - "keep_dims is deprecated, use keepdims instead\n", - "_________________________________________________________________\n", - "Layer (type) Output Shape Param # \n", - "=================================================================\n", - "dense_1 (Dense) (None, 1, 100) 300 \n", - "_________________________________________________________________\n", - "dense_2 (Dense) (None, 1, 100) 10100 \n", - "_________________________________________________________________\n", - "dense_3 (Dense) (None, 1, 3) 303 \n", - "=================================================================\n", - "Total params: 10,703\n", - "Trainable params: 10,703\n", - "Non-trainable params: 0\n", - "_________________________________________________________________\n" - ] - } - ], - "source": [ - "model = Sequential()\n", - "model.add(Dense(100,activation='relu',input_shape=(1,state_size)))\n", - "model.add(Dense(100,activation='relu'))\n", - "model.add(Dense(action_size,activation='linear'))\n", - "model.compile(loss='mse',optimizer=Adam(lr=lr),)\n", - "model.summary()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Here ,\n", - "- ```ep_list``` list of episodes\n", - "- ```reward_list``` list of rewards\n", - "- ```total_rewards``` totatl reward" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "***Psuedocode***" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "- for each episode do\n", - "- - get initial state $s$\n", - "- - $rewards_{total} \\gets 0 $\n", - "- - set terminal $d$ to false \n", - "- - for each step do\n", - "- - - choose action based on epsilon greedy policy\n", - "- - - get next state $s_{next}$, reward $r$, terminal $d$ doing the action\n", - "- - - $rewards_{total} \\gets rewards_{total}+r$\n", - "- - - if $d$ is $True $ \n", - "- - - - if $rewards_{total}<-199$ \n", - "- - - - - then give punishment $r \\gets -100$\n", - "- - - - - break \n", - "- - - $s \\gets s_{next}$\n", - "- - take random samples of $s,r,a,s_{next}$ from replay buffer\n", - "- - get $Q(s_{next})$ \n", - "- - $Q_{target} \\gets r+\\gamma max(Q(s_{next})) $\n", - "- - $loss \\gets \\frac{1}{N}\\sum(Q_{target}-Q(s))^2$\n", - "- - train the network using this loss" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Episode 0, Failed! Reward -200\n", - "Episode 1, Failed! Reward -200\n", - "Episode 2, Failed! Reward -200\n", - "Episode 3, Failed! Reward -200\n", - "Episode 4, Failed! Reward -200\n", - "Episode 5, Failed! Reward -200\n", - "Episode 6, Failed! Reward -200\n", - "Episode 7, Failed! Reward -200\n", - "Episode 8, Failed! Reward -200\n", - "Episode 9, Failed! Reward -200\n", - "Episode 10, Failed! Reward -200\n", - "Episode 11, Failed! Reward -200\n", - "Episode 12, Failed! Reward -200\n", - "Episode 13, Failed! Reward -200\n", - "Episode 14, Failed! Reward -200\n", - "Episode 15, Failed! Reward -200\n", - "Episode 16, Failed! Reward -200\n", - "Episode 17, Failed! Reward -200\n", - "Episode 18, Failed! Reward -200\n", - "Episode 19, Failed! Reward -200\n", - "Episode 20, Failed! Reward -200\n", - "Episode 21, Failed! Reward -200\n", - "Episode 22, Failed! Reward -200\n", - "Episode 23, Failed! Reward -200\n", - "Episode 24, Failed! Reward -200\n", - "Episode 25, Failed! Reward -200\n", - "Episode 26, Failed! Reward -200\n", - "Episode 27, Failed! Reward -200\n", - "Episode 28, Failed! Reward -200\n", - "Episode 29, Failed! Reward -200\n", - "Episode 30, Failed! Reward -200\n", - "Episode 31, Failed! Reward -200\n", - "Episode 32, Failed! Reward -200\n", - "Episode 33, Failed! Reward -200\n", - "Episode 34, Failed! Reward -200\n", - "Episode 35, Failed! Reward -200\n", - "Episode 36, Failed! Reward -200\n", - "Episode 37, Failed! Reward -200\n", - "Episode 38, Failed! Reward -200\n", - "Episode 39, Failed! Reward -200\n", - "Episode 40, Failed! Reward -200\n", - "Episode 41, Failed! Reward -200\n", - "Episode 42, Failed! Reward -200\n", - "Episode 43, Failed! Reward -200\n", - "Episode 44, Failed! Reward -200\n", - "Episode 45, Failed! Reward -200\n", - "Episode 46, Failed! Reward -200\n", - "Episode 47, Failed! Reward -200\n", - "Episode 48, Failed! Reward -200\n", - "Episode 49, Failed! Reward -200\n", - "Episode 50, Failed! Reward -200\n", - "Episode 51, Failed! Reward -200\n", - "Episode 52, Failed! Reward -200\n", - "Episode 53, Failed! Reward -200\n", - "Episode 54, Failed! Reward -200\n", - "Episode 55, Failed! Reward -200\n", - "Episode 56, Failed! Reward -200\n", - "Episode 57, Failed! Reward -200\n", - "Episode 58, Failed! Reward -200\n", - "Episode 59, Failed! Reward -200\n", - "Episode 60, Failed! Reward -200\n", - "Episode 61, Failed! Reward -200\n", - "Episode 62, Failed! Reward -200\n", - "Episode 63, Failed! Reward -200\n", - "Episode 64, Failed! Reward -200\n", - "Episode 65, Failed! Reward -200\n", - "Episode 66, Failed! Reward -200\n", - "Episode 67, Failed! Reward -200\n", - "Episode 68, Failed! Reward -200\n", - "Episode 69, Failed! Reward -200\n", - "Episode 70, Failed! Reward -200\n", - "Episode 71, Failed! Reward -200\n", - "Episode 72, Failed! Reward -200\n", - "Episode 73, Failed! Reward -200\n", - "Episode 74, Failed! Reward -200\n", - "Episode 75, Failed! Reward -200\n", - "Episode 76, Failed! Reward -200\n", - "Episode 77, Failed! Reward -200\n", - "Episode 78, Failed! Reward -200\n", - "Episode 79, Failed! Reward -200\n", - "Episode 80, Failed! Reward -200\n", - "Episode 81, Failed! Reward -200\n", - "Episode 82, Failed! Reward -200\n", - "Episode 83, Failed! Reward -200\n", - "Episode 84, Failed! Reward -200\n", - "Episode 85, Failed! Reward -200\n", - "Episode 86, Failed! Reward -200\n", - "Episode 87, Failed! Reward -200\n", - "Episode 88, Failed! Reward -200\n", - "Episode 89, Failed! Reward -200\n", - "Episode 90, Failed! Reward -200\n", - "Episode 91, Failed! Reward -200\n", - "Episode 92, Failed! Reward -200\n", - "Episode 93, Failed! Reward -200\n", - "Episode 94, Failed! Reward -200\n", - "Episode 95, Failed! Reward -200\n", - "Episode 96, Failed! Reward -200\n", - "Episode 97, Failed! Reward -200\n", - "Episode 98, Failed! Reward -200\n", - "Episode 99, Failed! Reward -200\n", - "Episode 100, Failed! Reward -200\n", - "Episode 101, Failed! Reward -200\n", - "Episode 102, Failed! Reward -200\n", - "Episode 103, Failed! Reward -200\n", - "Episode 104, Failed! Reward -200\n", - "Episode 105, Failed! Reward -200\n", - "Episode 106, Failed! Reward -200\n", - "Episode 107, Failed! Reward -200\n", - "Episode 108, Failed! Reward -200\n", - "Episode 109, Failed! Reward -200\n", - "Episode 110, Failed! Reward -200\n", - "Episode 111, Failed! Reward -200\n", - "Episode 112, Failed! Reward -200\n", - "Episode 113, Failed! Reward -200\n", - "Episode 114, Failed! Reward -200\n", - "Episode 115, Failed! Reward -200\n", - "Episode 116, Failed! Reward -200\n", - "Episode 117, Failed! Reward -200\n", - "Episode 118, Failed! Reward -200\n", - "Episode 119, Failed! Reward -200\n", - "Episode 120, Failed! Reward -200\n", - "Episode 121, Failed! Reward -200\n", - "Episode 122, Failed! Reward -200\n", - "Episode 123, Failed! Reward -200\n", - "Episode 124, Failed! Reward -200\n", - "Episode 125, Failed! Reward -200\n", - "Episode 126, Failed! Reward -200\n", - "Episode 127, Failed! Reward -200\n", - "Episode 128, Failed! Reward -200\n", - "Episode 129, Failed! Reward -200\n", - "Episode 130, Failed! Reward -200\n", - "Episode 131, Failed! Reward -200\n", - "Episode 132, Failed! Reward -200\n", - "Episode 133, Failed! Reward -200\n", - "Episode 134, Failed! Reward -200\n", - "Episode 135, Failed! Reward -200\n", - "Episode 136, Failed! Reward -200\n", - "Episode 137, Failed! Reward -200\n", - "Episode 138, Failed! Reward -200\n", - "Episode 139, Failed! Reward -200\n", - "Episode 140, Failed! Reward -200\n", - "Episode 141, Failed! Reward -200\n", - "Episode 142, Failed! Reward -200\n", - "Episode 143, Failed! Reward -200\n", - "Episode 144, Failed! Reward -200\n", - "Episode 145, Failed! Reward -200\n", - "Episode 146, Failed! Reward -200\n", - "Episode 147, Failed! Reward -200\n", - "Episode 148, Failed! Reward -200\n", - "Episode 149, Failed! Reward -200\n", - "Episode 150, Failed! Reward -200\n", - "Episode 151, Failed! Reward -200\n", - "Episode 152, Failed! Reward -200\n", - "Episode 153, Failed! Reward -200\n", - "Episode 154, Failed! Reward -200\n", - "Episode 155, Failed! Reward -200\n", - "Episode 156, Failed! Reward -200\n", - "Episode 157, Failed! Reward -200\n", - "Episode 158, Failed! Reward -200\n", - "Episode 159, Failed! Reward -200\n", - "Episode 160, Failed! Reward -200\n", - "Episode 161, Failed! Reward -200\n", - "Episode 162, Failed! Reward -200\n", - "Episode 163, Failed! Reward -200\n", - "Episode 164, Failed! Reward -200\n", - "Episode 165, Failed! Reward -200\n", - "Episode 166, Failed! Reward -200\n", - "Episode 167, Failed! Reward -200\n", - "Episode 168, Failed! Reward -200\n", - "Episode 169, Failed! Reward -200\n", - "Episode 170, Failed! Reward -200\n", - "Episode 171, Failed! Reward -200\n", - "Episode 172, Failed! Reward -200\n", - "Episode 173, Failed! Reward -200\n", - "Episode 174, Failed! Reward -200\n", - "Episode 175, Failed! Reward -200\n", - "Episode 176, Failed! Reward -200\n", - "Episode 177, Failed! Reward -200\n", - "Episode 178, Failed! Reward -200\n", - "Episode 179, Failed! Reward -200\n", - "Episode 180, Failed! Reward -200\n", - "Episode 181, Failed! Reward -200\n", - "Episode 182, Failed! Reward -200\n", - "Episode 183, Failed! Reward -200\n", - "Episode 184, Failed! Reward -200\n", - "Episode 185, Failed! Reward -200\n", - "Episode 186, Failed! Reward -200\n", - "Episode 187, Failed! Reward -200\n", - "Episode 188, Failed! Reward -200\n", - "Episode 189, Failed! Reward -200\n", - "Episode 190, Failed! Reward -200\n", - "Episode 191, Failed! Reward -200\n", - "Episode 192, Failed! Reward -200\n", - "Episode 193, Failed! Reward -200\n", - "Episode 194, Failed! Reward -200\n", - "Episode 195, Failed! Reward -200\n", - "Episode 196, Failed! Reward -200\n", - "Episode 197, Failed! Reward -200\n", - "Episode 198, Failed! Reward -200\n", - "Episode 199, Failed! Reward -200\n", - "Episode 200, Failed! Reward -200\n", - "Episode 201, Failed! Reward -200\n", - "Episode 202, Failed! Reward -200\n", - "Episode 203, Failed! Reward -200\n", - "Episode 204, Failed! Reward -200\n", - "Episode 205, Failed! Reward -200\n", - "Episode 206, Failed! Reward -200\n", - "Episode 207, Failed! Reward -200\n", - "Episode 208, Failed! Reward -200\n", - "Episode 209, Failed! Reward -200\n", - "Episode 210, Failed! Reward -200\n", - "Episode 211, Failed! Reward -200\n", - "Episode 212, Failed! Reward -200\n", - "Episode 213, Failed! Reward -200\n", - "Episode 214, Failed! Reward -200\n", - "Episode 215, Failed! Reward -200\n", - "Episode 216, Failed! Reward -200\n", - "Episode 217, Failed! Reward -200\n", - "Episode 218, Failed! Reward -200\n", - "Episode 219, Failed! Reward -200\n", - "Episode 220, Failed! Reward -200\n", - "Episode 221, Failed! Reward -200\n", - "Episode 222, Failed! Reward -200\n", - "Episode 223, Better! Reward -123\n", - "Episode 224, Failed! Reward -200\n", - "Episode 225, Failed! Reward -200\n", - "Episode 226, Failed! Reward -200\n", - "Episode 227, Failed! Reward -200\n", - "Episode 228, Failed! Reward -200\n", - "Episode 229, Failed! Reward -200\n", - "Episode 230, Failed! Reward -200\n", - "Episode 231, Failed! Reward -200\n", - "Episode 232, Failed! Reward -200\n", - "Episode 233, Failed! Reward -200\n", - "Episode 234, Failed! Reward -200\n", - "Episode 235, Failed! Reward -200\n", - "Episode 236, Failed! Reward -200\n", - "Episode 237, Failed! Reward -200\n", - "Episode 238, Failed! Reward -200\n", - "Episode 239, Failed! Reward -200\n", - "Episode 240, Failed! Reward -200\n", - "Episode 241, Failed! Reward -200\n", - "Episode 242, Failed! Reward -200\n", - "Episode 243, Failed! Reward -200\n", - "Episode 244, Failed! Reward -200\n", - "Episode 245, Failed! Reward -200\n", - "Episode 246, Failed! Reward -200\n", - "Episode 247, Failed! Reward -200\n", - "Episode 248, Failed! Reward -200\n", - "Episode 249, Failed! Reward -200\n", - "Episode 250, Failed! Reward -200\n", - "Episode 251, Failed! Reward -200\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Episode 252, Failed! Reward -200\n", - "Episode 253, Failed! Reward -200\n", - "Episode 254, Failed! Reward -200\n", - "Episode 255, Failed! Reward -200\n", - "Episode 256, Failed! Reward -200\n", - "Episode 257, Failed! Reward -200\n", - "Episode 258, Failed! Reward -200\n", - "Episode 259, Failed! Reward -200\n", - "Episode 260, Failed! Reward -200\n", - "Episode 261, Failed! Reward -200\n", - "Episode 262, Failed! Reward -200\n", - "Episode 263, Failed! Reward -200\n", - "Episode 264, Failed! Reward -200\n", - "Episode 265, Failed! Reward -200\n", - "Episode 266, Failed! Reward -200\n", - "Episode 267, Failed! Reward -200\n", - "Episode 268, Failed! Reward -200\n", - "Episode 269, Failed! Reward -200\n", - "Episode 270, Failed! Reward -200\n", - "Episode 271, Failed! Reward -200\n", - "Episode 272, Failed! Reward -200\n", - "Episode 273, Failed! Reward -200\n", - "Episode 274, Failed! Reward -200\n", - "Episode 275, Failed! Reward -200\n", - "Episode 276, Failed! Reward -200\n", - "Episode 277, Failed! Reward -200\n", - "Episode 278, Failed! Reward -200\n", - "Episode 279, Failed! Reward -200\n", - "Episode 280, Failed! Reward -200\n", - "Episode 281, Failed! Reward -200\n", - "Episode 282, Failed! Reward -200\n", - "Episode 283, Failed! Reward -200\n", - "Episode 284, Failed! Reward -200\n", - "Episode 285, Failed! Reward -200\n", - "Episode 286, Failed! Reward -200\n", - "Episode 287, Failed! Reward -200\n", - "Episode 288, Failed! Reward -200\n", - "Episode 289, Failed! Reward -200\n", - "Episode 290, Failed! Reward -200\n", - "Episode 291, Failed! Reward -200\n", - "Episode 292, Failed! Reward -200\n", - "Episode 293, Failed! Reward -200\n", - "Episode 294, Failed! Reward -200\n", - "Episode 295, Failed! Reward -200\n", - "Episode 296, Failed! Reward -200\n", - "Episode 297, Failed! Reward -200\n", - "Episode 298, Failed! Reward -200\n", - "Episode 299, Failed! Reward -200\n", - "Episode 300, Failed! Reward -200\n", - "Episode 301, Failed! Reward -200\n", - "Episode 302, Failed! Reward -200\n", - "Episode 303, Failed! Reward -200\n", - "Episode 304, Failed! Reward -200\n", - "Episode 305, Failed! Reward -200\n", - "Episode 306, Failed! Reward -200\n", - "Episode 307, Failed! Reward -200\n", - "Episode 308, Failed! Reward -200\n", - "Episode 309, Failed! Reward -200\n", - "Episode 310, Failed! Reward -200\n", - "Episode 311, Failed! Reward -200\n", - "Episode 312, Failed! Reward -200\n", - "Episode 313, Failed! Reward -200\n", - "Episode 314, Failed! Reward -200\n", - "Episode 315, Failed! Reward -200\n", - "Episode 316, Failed! Reward -200\n", - "Episode 317, Failed! Reward -200\n", - "Episode 318, Failed! Reward -200\n", - "Episode 319, Failed! Reward -200\n", - "Episode 320, Failed! Reward -200\n", - "Episode 321, Failed! Reward -200\n", - "Episode 322, Failed! Reward -200\n", - "Episode 323, Failed! Reward -200\n", - "Episode 324, Failed! Reward -200\n", - "Episode 325, Failed! Reward -200\n", - "Episode 326, Failed! Reward -200\n", - "Episode 327, Failed! Reward -200\n", - "Episode 328, Failed! Reward -200\n", - "Episode 329, Failed! Reward -200\n", - "Episode 330, Failed! Reward -200\n", - "Episode 331, Failed! Reward -200\n", - "Episode 332, Failed! Reward -200\n", - "Episode 333, Failed! Reward -200\n", - "Episode 334, Failed! Reward -200\n", - "Episode 335, Failed! Reward -200\n", - "Episode 336, Failed! Reward -200\n", - "Episode 337, Failed! Reward -200\n", - "Episode 338, Failed! Reward -200\n", - "Episode 339, Failed! Reward -200\n", - "Episode 340, Failed! Reward -200\n", - "Episode 341, Failed! Reward -200\n", - "Episode 342, Failed! Reward -200\n", - "Episode 343, Failed! Reward -200\n", - "Episode 344, Failed! Reward -200\n", - "Episode 345, Failed! Reward -200\n", - "Episode 347, Failed! Reward -200\n", - "Episode 348, Failed! Reward -200\n", - "Episode 349, Failed! Reward -200\n", - "Episode 350, Failed! Reward -200\n", - "Episode 351, Failed! Reward -200\n", - "Episode 352, Failed! Reward -200\n", - "Episode 353, Failed! Reward -200\n", - "Episode 354, Failed! Reward -200\n", - "Episode 355, Failed! Reward -200\n", - "Episode 356, Failed! Reward -200\n", - "Episode 357, Failed! Reward -200\n", - "Episode 358, Failed! Reward -200\n", - "Episode 359, Better! Reward -181\n", - "Episode 360, Better! Reward -196\n", - "Episode 361, Failed! Reward -200\n", - "Episode 362, Failed! Reward -200\n", - "Episode 363, Failed! Reward -200\n", - "Episode 364, Failed! Reward -200\n", - "Episode 365, Failed! Reward -200\n", - "Episode 366, Failed! Reward -200\n", - "Episode 367, Failed! Reward -200\n", - "Episode 368, Failed! Reward -200\n", - "Episode 369, Failed! Reward -200\n", - "Episode 370, Failed! Reward -200\n", - "Episode 371, Failed! Reward -200\n", - "Episode 372, Failed! Reward -200\n", - "Episode 373, Failed! Reward -200\n", - "Episode 374, Failed! Reward -200\n", - "Episode 375, Failed! Reward -200\n", - "Episode 376, Failed! Reward -200\n", - "Episode 377, Failed! Reward -200\n", - "Episode 378, Failed! Reward -200\n", - "Episode 379, Failed! Reward -200\n", - "Episode 380, Failed! Reward -200\n", - "Episode 381, Failed! Reward -200\n", - "Episode 382, Failed! Reward -200\n", - "Episode 383, Failed! Reward -200\n", - "Episode 384, Better! Reward -197\n", - "Episode 385, Failed! Reward -200\n", - "Episode 386, Failed! Reward -200\n", - "Episode 387, Failed! Reward -200\n", - "Episode 388, Failed! Reward -200\n", - "Episode 389, Failed! Reward -200\n", - "Episode 390, Failed! Reward -200\n", - "Episode 391, Failed! Reward -200\n", - "Episode 392, Failed! Reward -200\n", - "Episode 393, Failed! Reward -200\n", - "Episode 394, Failed! Reward -200\n", - "Episode 395, Failed! Reward -200\n", - "Episode 396, Failed! Reward -200\n", - "Episode 397, Better! Reward -165\n", - "Episode 398, Failed! Reward -200\n", - "Episode 399, Failed! Reward -200\n", - "Episode 400, Failed! Reward -200\n", - "Episode 401, Better! Reward -196\n", - "Episode 402, Failed! Reward -200\n", - "Episode 403, Failed! Reward -200\n", - "Episode 404, Failed! Reward -200\n", - "Episode 405, Failed! Reward -200\n", - "Episode 406, Failed! Reward -200\n", - "Episode 407, Failed! Reward -200\n", - "Episode 408, Failed! Reward -200\n", - "Episode 409, Failed! Reward -200\n", - "Episode 410, Failed! Reward -200\n", - "Episode 411, Failed! Reward -200\n", - "Episode 412, Failed! Reward -200\n", - "Episode 413, Failed! Reward -200\n", - "Episode 414, Failed! Reward -200\n", - "Episode 415, Failed! Reward -200\n", - "Episode 416, Failed! Reward -200\n", - "Episode 417, Failed! Reward -200\n", - "Episode 418, Failed! Reward -200\n", - "Episode 419, Failed! Reward -200\n", - "Episode 420, Failed! Reward -200\n", - "Episode 421, Failed! Reward -200\n", - "Episode 422, Failed! Reward -200\n", - "Episode 423, Failed! Reward -200\n", - "Episode 424, Failed! Reward -200\n", - "Episode 425, Failed! Reward -200\n", - "Episode 426, Failed! Reward -200\n", - "Episode 427, Failed! Reward -200\n", - "Episode 428, Failed! Reward -200\n", - "Episode 429, Failed! Reward -200\n", - "Episode 430, Failed! Reward -200\n", - "Episode 431, Failed! Reward -200\n", - "Episode 432, Failed! Reward -200\n", - "Episode 433, Failed! Reward -200\n", - "Episode 434, Failed! Reward -200\n", - "Episode 435, Failed! Reward -200\n", - "Episode 436, Failed! Reward -200\n", - "Episode 437, Failed! Reward -200\n", - "Episode 438, Failed! Reward -200\n", - "Episode 439, Failed! Reward -200\n", - "Episode 440, Failed! Reward -200\n", - "Episode 441, Failed! Reward -200\n", - "Episode 442, Failed! Reward -200\n", - "Episode 443, Failed! Reward -200\n", - "Episode 444, Failed! Reward -200\n", - "Episode 445, Failed! Reward -200\n", - "Episode 446, Failed! Reward -200\n", - "Episode 447, Failed! Reward -200\n", - "Episode 448, Failed! Reward -200\n", - "Episode 449, Failed! Reward -200\n", - "Episode 450, Failed! Reward -200\n", - "Episode 451, Failed! Reward -200\n", - "Episode 452, Failed! Reward -200\n", - "Episode 453, Failed! Reward -200\n", - "Episode 454, Failed! Reward -200\n", - "Episode 455, Failed! Reward -200\n", - "Episode 456, Failed! Reward -200\n", - "Episode 457, Failed! Reward -200\n", - "Episode 458, Failed! Reward -200\n", - "Episode 459, Failed! Reward -200\n", - "Episode 460, Failed! Reward -200\n", - "Episode 461, Failed! Reward -200\n", - "Episode 462, Failed! Reward -200\n", - "Episode 463, Failed! Reward -200\n", - "Episode 464, Failed! Reward -200\n", - "Episode 465, Failed! Reward -200\n", - "Episode 466, Failed! Reward -200\n", - "Episode 467, Failed! Reward -200\n", - "Episode 468, Failed! Reward -200\n", - "Episode 469, Failed! Reward -200\n", - "Episode 470, Failed! Reward -200\n", - "Episode 471, Failed! Reward -200\n", - "Episode 472, Failed! Reward -200\n", - "Episode 473, Failed! Reward -200\n", - "Episode 474, Failed! Reward -200\n", - "Episode 475, Failed! Reward -200\n", - "Episode 476, Failed! Reward -200\n", - "Episode 477, Failed! Reward -200\n", - "Episode 478, Failed! Reward -200\n", - "Episode 479, Failed! Reward -200\n", - "Episode 480, Failed! Reward -200\n", - "Episode 481, Failed! Reward -200\n", - "Episode 482, Failed! Reward -200\n", - "Episode 483, Failed! Reward -200\n", - "Episode 484, Failed! Reward -200\n", - "Episode 485, Failed! Reward -200\n", - "Episode 486, Failed! Reward -200\n", - "Episode 487, Failed! Reward -200\n", - "Episode 488, Failed! Reward -200\n", - "Episode 489, Failed! Reward -200\n", - "Episode 490, Failed! Reward -200\n", - "Episode 491, Failed! Reward -200\n", - "Episode 492, Failed! Reward -200\n", - "Episode 493, Failed! Reward -200\n", - "Episode 494, Failed! Reward -200\n", - "Episode 495, Failed! Reward -200\n", - "Episode 496, Failed! Reward -200\n", - "Episode 497, Failed! Reward -200\n", - "Episode 498, Failed! Reward -200\n", - "Episode 499, Failed! Reward -200\n", - "Episode 500, Failed! Reward -200\n", - "Episode 501, Failed! Reward -200\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Episode 502, Failed! Reward -200\n", - "Episode 503, Failed! Reward -200\n", - "Episode 504, Failed! Reward -200\n", - "Episode 505, Failed! Reward -200\n", - "Episode 506, Failed! Reward -200\n", - "Episode 507, Failed! Reward -200\n", - "Episode 508, Failed! Reward -200\n", - "Episode 509, Failed! Reward -200\n", - "Episode 510, Failed! Reward -200\n", - "Episode 511, Failed! Reward -200\n", - "Episode 512, Failed! Reward -200\n", - "Episode 513, Failed! Reward -200\n", - "Episode 514, Failed! Reward -200\n", - "Episode 515, Failed! Reward -200\n", - "Episode 516, Failed! Reward -200\n", - "Episode 517, Failed! Reward -200\n", - "Episode 518, Failed! Reward -200\n", - "Episode 519, Failed! Reward -200\n", - "Episode 520, Failed! Reward -200\n", - "Episode 521, Failed! Reward -200\n", - "Episode 522, Failed! Reward -200\n", - "Episode 523, Failed! Reward -200\n", - "Episode 524, Failed! Reward -200\n", - "Episode 525, Failed! Reward -200\n", - "Episode 526, Failed! Reward -200\n", - "Episode 527, Failed! Reward -200\n", - "Episode 528, Failed! Reward -200\n", - "Episode 529, Failed! Reward -200\n", - "Episode 530, Failed! Reward -200\n", - "Episode 531, Failed! Reward -200\n", - "Episode 532, Failed! Reward -200\n", - "Episode 533, Failed! Reward -200\n", - "Episode 534, Failed! Reward -200\n", - "Episode 535, Failed! Reward -200\n", - "Episode 536, Failed! Reward -200\n", - "Episode 537, Failed! Reward -200\n", - "Episode 538, Failed! Reward -200\n", - "Episode 539, Failed! Reward -200\n", - "Episode 540, Failed! Reward -200\n", - "Episode 541, Failed! Reward -200\n", - "Episode 542, Failed! Reward -200\n", - "Episode 543, Failed! Reward -200\n", - "Episode 544, Failed! Reward -200\n", - "Episode 545, Failed! Reward -200\n", - "Episode 546, Failed! Reward -200\n", - "Episode 547, Failed! Reward -200\n", - "Episode 548, Failed! Reward -200\n", - "Episode 549, Failed! Reward -200\n", - "Episode 550, Failed! Reward -200\n", - "Episode 551, Failed! Reward -200\n", - "Episode 552, Failed! Reward -200\n", - "Episode 553, Failed! Reward -200\n", - "Episode 554, Failed! Reward -200\n", - "Episode 555, Failed! Reward -200\n", - "Episode 556, Failed! Reward -200\n", - "Episode 557, Failed! Reward -200\n", - "Episode 558, Failed! Reward -200\n", - "Episode 559, Failed! Reward -200\n", - "Episode 560, Failed! Reward -200\n", - "Episode 561, Failed! Reward -200\n", - "Episode 562, Failed! Reward -200\n", - "Episode 563, Failed! Reward -200\n", - "Episode 564, Failed! Reward -200\n", - "Episode 565, Failed! Reward -200\n", - "Episode 566, Failed! Reward -200\n", - "Episode 567, Failed! Reward -200\n", - "Episode 568, Failed! Reward -200\n", - "Episode 569, Failed! Reward -200\n", - "Episode 570, Failed! Reward -200\n", - "Episode 571, Failed! Reward -200\n", - "Episode 572, Failed! Reward -200\n", - "Episode 573, Failed! Reward -200\n", - "Episode 574, Failed! Reward -200\n", - "Episode 575, Failed! Reward -200\n", - "Episode 576, Failed! Reward -200\n", - "Episode 577, Failed! Reward -200\n", - "Episode 578, Failed! Reward -200\n", - "Episode 579, Failed! Reward -200\n", - "Episode 580, Failed! Reward -200\n", - "Episode 581, Failed! Reward -200\n", - "Episode 582, Failed! Reward -200\n", - "Episode 583, Failed! Reward -200\n", - "Episode 584, Failed! Reward -200\n", - "Episode 585, Failed! Reward -200\n", - "Episode 586, Failed! Reward -200\n", - "Episode 587, Failed! Reward -200\n", - "Episode 588, Failed! Reward -200\n", - "Episode 589, Failed! Reward -200\n", - "Episode 590, Failed! Reward -200\n", - "Episode 591, Failed! Reward -200\n", - "Episode 592, Failed! Reward -200\n", - "Episode 593, Failed! Reward -200\n", - "Episode 594, Failed! Reward -200\n", - "Episode 595, Failed! Reward -200\n", - "Episode 596, Failed! Reward -200\n", - "Episode 597, Failed! Reward -200\n", - "Episode 598, Failed! Reward -200\n", - "Episode 599, Failed! Reward -200\n", - "Episode 600, Failed! Reward -200\n", - "Episode 601, Failed! Reward -200\n", - "Episode 602, Failed! Reward -200\n", - "Episode 603, Failed! Reward -200\n", - "Episode 604, Failed! Reward -200\n", - "Episode 605, Failed! Reward -200\n", - "Episode 606, Failed! Reward -200\n", - "Episode 607, Failed! Reward -200\n", - "Episode 608, Failed! Reward -200\n", - "Episode 609, Failed! Reward -200\n", - "Episode 610, Failed! Reward -200\n", - "Episode 611, Failed! Reward -200\n", - "Episode 612, Failed! Reward -200\n", - "Episode 613, Failed! Reward -200\n", - "Episode 614, Failed! Reward -200\n", - "Episode 615, Failed! Reward -200\n", - "Episode 616, Failed! Reward -200\n", - "Episode 617, Failed! Reward -200\n", - "Episode 618, Failed! Reward -200\n", - "Episode 619, Failed! Reward -200\n", - "Episode 620, Better! Reward -191\n", - "Episode 621, Better! Reward -168\n", - "Episode 622, Better! Reward -169\n", - "Episode 623, Better! Reward -158\n", - "Episode 624, Better! Reward -161\n", - "Episode 625, Better! Reward -123\n", - "Episode 626, Better! Reward -126\n", - "Episode 627, Better! Reward -156\n", - "Episode 628, Failed! Reward -200\n", - "Episode 629, Better! Reward -155\n", - "Episode 630, Failed! Reward -200\n", - "Episode 631, Better! Reward -123\n", - "Episode 632, Better! Reward -157\n", - "Episode 633, Failed! Reward -200\n", - "Episode 634, Failed! Reward -200\n", - "Episode 635, Failed! Reward -200\n", - "Episode 636, Failed! Reward -200\n", - "Episode 637, Better! Reward -165\n", - "Episode 638, Failed! Reward -200\n", - "Episode 639, Passed! Reward -97\n", - "Episode 640, Failed! Reward -200\n", - "Episode 641, Failed! Reward -200\n", - "Episode 642, Failed! Reward -200\n", - "Episode 643, Failed! Reward -200\n", - "Episode 644, Failed! Reward -200\n", - "Episode 645, Failed! Reward -200\n", - "Episode 646, Failed! Reward -200\n", - "Episode 647, Failed! Reward -200\n", - "Episode 648, Failed! Reward -200\n", - "Episode 649, Failed! Reward -200\n", - "Episode 650, Failed! Reward -200\n", - "Episode 651, Better! Reward -190\n", - "Episode 652, Failed! Reward -200\n", - "Episode 653, Failed! Reward -200\n", - "Episode 654, Failed! Reward -200\n", - "Episode 655, Failed! Reward -200\n", - "Episode 656, Failed! Reward -200\n", - "Episode 657, Failed! Reward -200\n", - "Episode 658, Better! Reward -173\n", - "Episode 659, Failed! Reward -200\n", - "Episode 660, Failed! Reward -200\n", - "Episode 661, Failed! Reward -200\n", - "Episode 662, Failed! Reward -200\n", - "Episode 663, Failed! Reward -200\n", - "Episode 664, Failed! Reward -200\n", - "Episode 665, Failed! Reward -200\n", - "Episode 666, Failed! Reward -200\n", - "Episode 667, Failed! Reward -200\n", - "Episode 668, Failed! Reward -200\n", - "Episode 669, Failed! Reward -200\n", - "Episode 670, Failed! Reward -200\n", - "Episode 671, Failed! Reward -200\n", - "Episode 672, Failed! Reward -200\n", - "Episode 673, Failed! Reward -200\n", - "Episode 675, Failed! Reward -200\n", - "Episode 676, Better! Reward -164\n", - "Episode 677, Failed! Reward -200\n", - "Episode 678, Failed! Reward -200\n", - "Episode 679, Better! Reward -169\n", - "Episode 680, Better! Reward -139\n", - "Episode 681, Failed! Reward -200\n", - "Episode 682, Failed! Reward -200\n", - "Episode 683, Failed! Reward -200\n", - "Episode 684, Failed! Reward -200\n", - "Episode 685, Failed! Reward -200\n", - "Episode 686, Failed! Reward -200\n", - "Episode 687, Failed! Reward -200\n", - "Episode 688, Failed! Reward -200\n", - "Episode 689, Failed! Reward -200\n", - "Episode 690, Failed! Reward -200\n", - "Episode 691, Failed! Reward -200\n", - "Episode 693, Failed! Reward -200\n", - "Episode 694, Failed! Reward -200\n", - "Episode 695, Failed! Reward -200\n", - "Episode 696, Failed! Reward -200\n", - "Episode 697, Failed! Reward -200\n", - "Episode 698, Failed! Reward -200\n", - "Episode 699, Failed! Reward -200\n", - "Episode 700, Failed! Reward -200\n", - "Episode 701, Failed! Reward -200\n", - "Episode 702, Failed! Reward -200\n", - "Episode 703, Failed! Reward -200\n", - "Episode 704, Failed! Reward -200\n", - "Episode 705, Failed! Reward -200\n", - "Episode 706, Failed! Reward -200\n", - "Episode 707, Failed! Reward -200\n", - "Episode 708, Failed! Reward -200\n", - "Episode 709, Failed! Reward -200\n", - "Episode 710, Failed! Reward -200\n", - "Episode 711, Failed! Reward -200\n", - "Episode 712, Failed! Reward -200\n", - "Episode 713, Failed! Reward -200\n", - "Episode 714, Failed! Reward -200\n", - "Episode 715, Failed! Reward -200\n", - "Episode 716, Failed! Reward -200\n", - "Episode 717, Failed! Reward -200\n", - "Episode 718, Failed! Reward -200\n", - "Episode 719, Failed! Reward -200\n", - "Episode 720, Failed! Reward -200\n", - "Episode 721, Failed! Reward -200\n", - "Episode 722, Failed! Reward -200\n", - "Episode 723, Failed! Reward -200\n", - "Episode 724, Failed! Reward -200\n", - "Episode 725, Failed! Reward -200\n", - "Episode 726, Failed! Reward -200\n", - "Episode 727, Failed! Reward -200\n", - "Episode 728, Failed! Reward -200\n", - "Episode 729, Failed! Reward -200\n", - "Episode 730, Failed! Reward -200\n", - "Episode 731, Failed! Reward -200\n", - "Episode 732, Failed! Reward -200\n", - "Episode 733, Failed! Reward -200\n", - "Episode 734, Failed! Reward -200\n", - "Episode 735, Failed! Reward -200\n", - "Episode 736, Failed! Reward -200\n", - "Episode 737, Failed! Reward -200\n", - "Episode 738, Failed! Reward -200\n", - "Episode 739, Failed! Reward -200\n", - "Episode 740, Failed! Reward -200\n", - "Episode 741, Failed! Reward -200\n", - "Episode 742, Failed! Reward -200\n", - "Episode 743, Failed! Reward -200\n", - "Episode 744, Failed! Reward -200\n", - "Episode 745, Failed! Reward -200\n", - "Episode 746, Failed! Reward -200\n", - "Episode 747, Failed! Reward -200\n", - "Episode 748, Failed! Reward -200\n", - "Episode 749, Failed! Reward -200\n", - "Episode 750, Failed! Reward -200\n", - "Episode 751, Failed! Reward -200\n", - "Episode 752, Failed! Reward -200\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Episode 753, Failed! Reward -200\n", - "Episode 754, Failed! Reward -200\n", - "Episode 755, Failed! Reward -200\n", - "Episode 756, Failed! Reward -200\n", - "Episode 757, Failed! Reward -200\n", - "Episode 758, Failed! Reward -200\n", - "Episode 759, Failed! Reward -200\n", - "Episode 760, Failed! Reward -200\n", - "Episode 761, Failed! Reward -200\n", - "Episode 762, Failed! Reward -200\n", - "Episode 763, Failed! Reward -200\n", - "Episode 764, Failed! Reward -200\n", - "Episode 765, Better! Reward -125\n", - "Episode 766, Better! Reward -117\n", - "Episode 767, Failed! Reward -200\n", - "Episode 768, Failed! Reward -200\n", - "Episode 769, Failed! Reward -200\n", - "Episode 770, Failed! Reward -200\n", - "Episode 771, Better! Reward -134\n", - "Episode 772, Failed! Reward -200\n", - "Episode 773, Better! Reward -119\n", - "Episode 774, Better! Reward -121\n", - "Episode 775, Better! Reward -119\n", - "Episode 776, Better! Reward -120\n", - "Episode 777, Better! Reward -119\n", - "Episode 778, Better! Reward -123\n", - "Episode 779, Better! Reward -116\n", - "Episode 780, Failed! Reward -200\n", - "Episode 781, Better! Reward -122\n", - "Episode 782, Better! Reward -125\n", - "Episode 783, Better! Reward -125\n", - "Episode 784, Better! Reward -124\n", - "Episode 785, Better! Reward -133\n", - "Episode 786, Better! Reward -127\n", - "Episode 787, Better! Reward -120\n", - "Episode 788, Better! Reward -114\n", - "Episode 789, Better! Reward -117\n", - "Episode 790, Better! Reward -116\n", - "Episode 791, Failed! Reward -200\n", - "Episode 792, Better! Reward -118\n", - "Episode 793, Better! Reward -113\n", - "Episode 794, Better! Reward -119\n", - "Episode 795, Failed! Reward -200\n", - "Episode 796, Better! Reward -125\n", - "Episode 797, Better! Reward -118\n", - "Episode 798, Better! Reward -117\n", - "Episode 799, Better! Reward -114\n", - "Episode 800, Better! Reward -115\n", - "Episode 801, Better! Reward -117\n", - "Episode 802, Better! Reward -117\n", - "Episode 803, Failed! Reward -200\n", - "Episode 804, Better! Reward -115\n", - "Episode 805, Better! Reward -113\n", - "Episode 806, Better! Reward -115\n", - "Episode 807, Better! Reward -115\n", - "Episode 808, Better! Reward -159\n", - "Episode 809, Better! Reward -115\n", - "Episode 810, Better! Reward -113\n", - "Episode 811, Failed! Reward -200\n", - "Episode 812, Failed! Reward -200\n", - "Episode 813, Better! Reward -112\n", - "Episode 814, Passed! Reward -110\n", - "Episode 815, Better! Reward -117\n", - "Episode 816, Better! Reward -116\n", - "Episode 817, Better! Reward -115\n", - "Episode 818, Better! Reward -116\n", - "Episode 819, Better! Reward -115\n", - "Episode 820, Better! Reward -112\n", - "Episode 821, Better! Reward -116\n", - "Episode 822, Better! Reward -114\n", - "Episode 823, Better! Reward -115\n", - "Episode 824, Better! Reward -114\n", - "Episode 825, Better! Reward -115\n", - "Episode 826, Better! Reward -196\n", - "Episode 827, Better! Reward -115\n", - "Episode 828, Better! Reward -115\n", - "Episode 829, Better! Reward -113\n", - "Episode 830, Better! Reward -151\n", - "Episode 831, Better! Reward -114\n", - "Episode 832, Better! Reward -112\n", - "Episode 833, Better! Reward -113\n", - "Episode 834, Better! Reward -114\n", - "Episode 835, Better! Reward -113\n", - "Episode 836, Better! Reward -112\n", - "Episode 837, Failed! Reward -200\n", - "Episode 838, Better! Reward -113\n", - "Episode 839, Failed! Reward -200\n", - "Episode 840, Better! Reward -113\n", - "Episode 841, Better! Reward -118\n", - "Episode 842, Better! Reward -117\n", - "Episode 843, Better! Reward -115\n", - "Episode 844, Better! Reward -118\n", - "Episode 845, Better! Reward -120\n", - "Episode 846, Failed! Reward -200\n", - "Episode 847, Better! Reward -118\n", - "Episode 848, Failed! Reward -200\n", - "Episode 849, Better! Reward -118\n", - "Episode 850, Better! Reward -119\n", - "Episode 851, Better! Reward -113\n", - "Episode 852, Better! Reward -119\n", - "Episode 853, Better! Reward -117\n", - "Episode 854, Better! Reward -113\n", - "Episode 855, Better! Reward -129\n", - "Episode 856, Better! Reward -117\n", - "Episode 857, Better! Reward -123\n", - "Episode 858, Better! Reward -114\n", - "Episode 859, Better! Reward -133\n", - "Episode 860, Better! Reward -116\n", - "Episode 861, Better! Reward -115\n", - "Episode 862, Better! Reward -114\n", - "Episode 863, Better! Reward -117\n", - "Episode 864, Better! Reward -116\n", - "Episode 865, Better! Reward -116\n", - "Episode 866, Better! Reward -122\n", - "Episode 867, Better! Reward -118\n", - "Episode 868, Better! Reward -125\n", - "Episode 869, Better! Reward -116\n", - "Episode 870, Better! Reward -113\n", - "Episode 871, Better! Reward -115\n", - "Episode 872, Better! Reward -117\n", - "Episode 873, Better! Reward -119\n", - "Episode 874, Better! Reward -116\n", - "Episode 875, Better! Reward -113\n", - "Episode 876, Better! Reward -115\n", - "Episode 877, Better! Reward -124\n", - "Episode 878, Better! Reward -113\n", - "Episode 879, Better! Reward -114\n", - "Episode 880, Better! Reward -116\n", - "Episode 881, Better! Reward -117\n", - "Episode 882, Better! Reward -112\n", - "Episode 883, Better! Reward -116\n", - "Episode 884, Better! Reward -115\n", - "Episode 885, Better! Reward -116\n", - "Episode 886, Better! Reward -113\n", - "Episode 887, Better! Reward -116\n", - "Episode 888, Better! Reward -114\n", - "Episode 889, Better! Reward -128\n", - "Episode 890, Better! Reward -113\n", - "Episode 891, Better! Reward -119\n", - "Episode 892, Better! Reward -116\n", - "Episode 893, Better! Reward -145\n", - "Episode 894, Better! Reward -117\n", - "Episode 895, Better! Reward -113\n", - "Episode 896, Better! Reward -123\n", - "Episode 897, Better! Reward -117\n", - "Episode 898, Better! Reward -113\n", - "Episode 899, Better! Reward -116\n", - "Episode 900, Better! Reward -117\n", - "Episode 901, Better! Reward -116\n", - "Episode 902, Better! Reward -115\n", - "Episode 903, Better! Reward -114\n", - "Episode 904, Better! Reward -115\n", - "Episode 905, Better! Reward -116\n", - "Episode 906, Better! Reward -118\n", - "Episode 907, Better! Reward -116\n", - "Episode 908, Better! Reward -115\n", - "Episode 909, Better! Reward -116\n", - "Episode 910, Better! Reward -117\n", - "Episode 911, Better! Reward -115\n", - "Episode 912, Better! Reward -116\n", - "Episode 913, Better! Reward -113\n", - "Episode 914, Better! Reward -115\n", - "Episode 915, Better! Reward -134\n", - "Episode 916, Failed! Reward -200\n", - "Episode 917, Better! Reward -119\n", - "Episode 918, Passed! Reward -85\n", - "Episode 919, Better! Reward -147\n", - "Episode 920, Better! Reward -115\n", - "Episode 921, Better! Reward -113\n", - "Episode 922, Better! Reward -117\n", - "Episode 923, Better! Reward -121\n", - "Episode 924, Failed! Reward -200\n", - "Episode 925, Better! Reward -168\n", - "Episode 926, Better! Reward -155\n", - "Episode 927, Better! Reward -174\n", - "Episode 928, Better! Reward -157\n", - "Episode 929, Better! Reward -160\n", - "Episode 930, Passed! Reward -96\n", - "Episode 931, Better! Reward -163\n", - "Episode 932, Better! Reward -188\n", - "Episode 933, Better! Reward -160\n", - "Episode 934, Better! Reward -158\n", - "Episode 935, Passed! Reward -93\n", - "Episode 936, Better! Reward -154\n", - "Episode 937, Better! Reward -160\n", - "Episode 938, Better! Reward -157\n", - "Episode 939, Better! Reward -173\n", - "Episode 940, Better! Reward -155\n", - "Episode 941, Better! Reward -153\n", - "Episode 942, Better! Reward -174\n", - "Episode 943, Better! Reward -166\n", - "Episode 944, Better! Reward -115\n", - "Episode 945, Better! Reward -158\n", - "Episode 946, Better! Reward -161\n", - "Episode 947, Better! Reward -157\n", - "Episode 948, Better! Reward -157\n", - "Episode 949, Better! Reward -153\n", - "Episode 950, Better! Reward -160\n", - "Episode 951, Better! Reward -156\n", - "Episode 952, Better! Reward -157\n", - "Episode 953, Failed! Reward -200\n", - "Episode 954, Better! Reward -154\n", - "Episode 955, Better! Reward -161\n", - "Episode 956, Better! Reward -168\n", - "Episode 957, Failed! Reward -200\n", - "Episode 958, Failed! Reward -200\n", - "Episode 959, Failed! Reward -200\n", - "Episode 960, Better! Reward -163\n", - "Episode 961, Failed! Reward -200\n", - "Episode 962, Passed! Reward -100\n", - "Episode 963, Better! Reward -144\n", - "Episode 964, Better! Reward -164\n", - "Episode 965, Failed! Reward -200\n", - "Episode 966, Better! Reward -152\n", - "Episode 967, Failed! Reward -200\n", - "Episode 968, Failed! Reward -200\n", - "Episode 969, Better! Reward -162\n", - "Episode 970, Better! Reward -154\n", - "Episode 971, Failed! Reward -200\n", - "Episode 972, Failed! Reward -200\n", - "Episode 973, Better! Reward -114\n", - "Episode 974, Passed! Reward -92\n", - "Episode 975, Passed! Reward -87\n", - "Episode 976, Failed! Reward -200\n", - "Episode 977, Failed! Reward -200\n", - "Episode 978, Failed! Reward -200\n", - "Episode 979, Failed! Reward -200\n", - "Episode 980, Failed! Reward -200\n", - "Episode 981, Failed! Reward -200\n", - "Episode 982, Failed! Reward -200\n", - "Episode 983, Better! Reward -131\n", - "Episode 984, Better! Reward -127\n", - "Episode 985, Better! Reward -130\n", - "Episode 986, Better! Reward -129\n", - "Episode 987, Better! Reward -135\n", - "Episode 988, Better! Reward -129\n", - "Episode 989, Better! Reward -125\n", - "Episode 990, Failed! Reward -200\n", - "Episode 991, Better! Reward -133\n", - "Episode 992, Better! Reward -136\n", - "Episode 993, Better! Reward -142\n", - "Episode 994, Better! Reward -146\n", - "Episode 995, Failed! Reward -200\n", - "Episode 996, Better! Reward -118\n", - "Episode 997, Better! Reward -121\n", - "Episode 998, Better! Reward -121\n", - "Episode 999, Better! Reward -125\n" - ] - } - ], - "source": [ - "ep_list =[]\n", - "reward_list =[] \n", - "index=0 \n", - "oh = OneHotEncoder(n_values=3)\n", - "for ep in range(num_episodes):\n", - " s= env.reset()\n", - " s=s.reshape((1,-1))\n", - " s = s*factor\n", - " total_rewards =0\n", - " d = False\n", - " j = 0\n", - " for j in range(200):\n", - " if np.random.random()< epsilon:\n", - " a = np.random.randint(0,len(action_list))\n", - " else:\n", - " Q = model.predict(s.reshape(-1,s.shape[0],s.shape[1]))\n", - " a =np.argmax(Q)\n", - " new_s,r,d,_ = env.step(a)\n", - " new_s = new_s.reshape((1,-1))\n", - " new_s = new_s*factor\n", - " total_rewards=total_rewards+r\n", - " if show:\n", - " env.render()\n", - " if d:\n", - " if total_rewards<-199:\n", - " r =-100\n", - " experience = (s,r,a,new_s,d)\n", - " memory.append(experience)\n", - " print(\"Episode %d, Failed! Reward %d\"%(ep,total_rewards))\n", - " elif total_rewards<-110 and total_rewards>-199:\n", - " r=10\n", - " d=True\n", - " experience = (s,r,a,new_s,d)\n", - " memory.append(experience)\n", - " print(\"Episode %d, Better! Reward %d\"%(ep,total_rewards))\n", - " elif total_rewards>=-110:\n", - " r=100\n", - " experience = (s,r,a,new_s,d)\n", - " memory.append(experience)\n", - "\n", - " print(\"Episode %d, Passed! Reward %d\"%(ep,total_rewards))\n", - " ep_list.append(ep)\n", - " reward_list.append(total_rewards)\n", - " break\n", - " \n", - " experience = (s,r,a,new_s,d)\n", - " memory.append(experience)\n", - " if j==199:\n", - " print(\"Reward %d after full episode\"%(total_rewards))\n", - " \n", - " s = new_s\n", - " batches=random.sample(memory,batch_size)\n", - " states= np.array([batch[0] for batch in batches])\n", - " rewards= np.array([batch[1] for batch in batches])\n", - " actions= np.array([batch[2] for batch in batches])\n", - " actions=oh.fit_transform(actions.reshape(-1,1)).toarray()\n", - " actions = actions.reshape(-1,1,action_size)\n", - " new_states= np.array([batch[3] for batch in batches])\n", - " dones= np.array([batch[4] for batch in batches])\n", - " Qs =model.predict(states)\n", - " new_Qs = model.predict(new_states)\n", - " target_Qs=rewards.reshape(-1,1)+gamma*(np.max(new_Qs,axis=2)*(~dones.reshape(-1,1)))\n", - " Qs[actions==1]=target_Qs.reshape(-1,)\n", - " model.fit(states,Qs,verbose=0)\n", - " epsilon=epsilon*epsilon_decay\n", - "env.close()" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "Text(0,0.5,'Rewards')" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEWCAYAAACnlKo3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztvXm8HGWV///+5CYkIWQh7ITEsEQUWfWyKTqoqKAyjAgKOoPjqAwz4jb6U5BRYH7jhrxcUHRgcAMUZEQWJRIWJYCyJbKFQDAkQEISSIAkhCQk997z/aOqb6r7VndXV1d3Vfc979erc6ufp+qpU1Wd59RzznnOIzPDcRzHcdIwIm8BHMdxnM7FlYjjOI6TGlcijuM4TmpciTiO4zipcSXiOI7jpMaViOM4jpMaVyKOkwJJP5f033nL0SiSvizpkozbPFLS0izbdDoHVyJOoZD0pKQNktZJWhF21tvkLVeRCO/JpvAelT4PJjnWzL5uZh9vtYzO8MGViFNEjjWzbYADgYOAM/MSRNLIvM5dh/PMbJvI54C8BXKGJ65EnMJiZiuAWQTKBABJoyWdL+lpSc9K+h9JY8O62ZLeH24fIckkvTv8fpSkB8LtPSX9UdLzklZJ+qWkSZFzPCnpS5IeAl6WNFLSQZL+KuklSb8GxkT2317S7yWtlvSCpDskDfm/Fcp6fkXZdZL+I9z+kqRnwnMskPT2Ru+ZpOnhdZ8qaZmk5ZI+H6k/R9Ll4fYYSZeH92G1pPsk7RTW7Srp+vB6Fkr6RKSNseFo6EVJ84GDK2TYVdLVklZKWizp041eh9M5uBJxCouk3YBjgIWR4m8BryZQLHsBU4CvhnWzgSPD7bcAi4C/i3yfXWoa+AawK/BaYCpwTsXpTwbeA0wi+H9yLXAZMBn4P+D9kX0/DywFdgB2Ar4MxOUT+hXwQUkKr29b4J3AlZL2Bk4HDjaz8cC7gCdjb0wy3grMCNs/Q9JRMft8BJhIcP3bAacBG8K6K8Jr2hU4Afh6RKmdDewZft4VtkN4TSOA3wEPEjybtwOflfSuJq7FKTCuRJwicq2kl4AlwHMEnRZh5/sJ4HNm9oKZvQR8HTgpPG425UrjG5HvfxfWY2YLzexmM3vFzFYC34nsV+ICM1tiZhuAw4BRwPfMbLOZ/Qa4L7LvZmAX4FVh/R0Wn5TuDgLl8ubw+wnAXWa2DOgHRgP7SBplZk+a2RM17tEXwtFD6fOLivpzzexlM3sY+BmBUqxkM4Hy2MvM+s1srpmtlTQVOAL4kpltNLMHgEuAfwqP+wDwtfAZLAEuiLR5MLCDmf2XmW0ys0XA/7LlGTldhisRp4j8Q/g2fiTwGmD7sHwHYGtgbqnzBG4MywHuAl4dmmQOBC4FpkraHjgEuB1A0o6SrgxNR2uByyPnKLEksr0r8EyFYngqsv1tgtHSTZIWSToj7qLC469kS4f+IeCXYd1C4LMEI6LnQvl2rXGPzjezSZHPRyrqo/I/FV5DJZcRmAuvDE1f50kaFe5bUtLRNqaE27vGtF/iVcCuUQVHMDLbqca1OB2MKxGnsJjZbODnQMmPsIrA3PK6SOc5MXTCY2brgbnAZ4B5ZrYJ+AvwH8ATZrYqbOcbBCOC/c1sAvCPBCaustNHtpcDU0pmqJBpETlfMrPPm9kewLHAf9TwZ1wBnCDpVcChwNWRdn5lZkcQdMRGYLpLy9QKWZdV7hCOms41s32ANwLvBU4J950saXxFG8+E28tj2i+xBFhcoeDGm9m7m7gWp8C4EnGKzveAd0g60MwGCEwj35W0I4CkKRX29tkEvoWS/+O2iu8A44F1wGpJU4D/r44MdwF9wKdDJ/vxBCMbQhneK2mvUMmsJTBN9cc1ZGb3AysJzEOzzGx12Mbekt4maTSwkUBZxraRkK9I2lrS64CPAr+u3EHSWyXtJ6knlHsz0B+aqP4CfCN0vu8PfIxw1ARcBZwpadvQb/WpSLP3AmvDIIGxknok7SupzPnudA+uRJxCE/osLgW+EhZ9icB0dHdoiroF2DtyyGwCJXF7le8A5wKvB9YANwC/rSPDJuB44J+BF4EPVhwzI5RjHYHC+ZGZ3VajySuAowgc7SVGA98kGG2tAHYkMANV44sqnyeyqqJ+NsF9upXA9HVTTBs7A78hUCCPhsdcHtadDEwnGJVcA5xtZjeHdecSmLAWAzcRmMUAMLN+gtHYgWH9KgKFObHGtTgdjHxRKsfpHiRNJ+i8R5lZX77SOMMBH4k4juM4qXEl4jiO46SmcEpE0oGS7pb0gKQ5kg4JyyXpgnD27EOSXp+3rI5TNML5JXJTltMuCqdEgPMIJkodSDAT+byw/BgCB+YM4FTgx/mI5ziO45QoYnI5AyaE2xPZEt9+HHBpOGHrbkmTJO1iZstrNbb99tvb9OnTWyas4zhOtzF37txVZrZD/T2LqUQ+C8wKE9WNIJgEBcFs2egs2aVhWU0lMn36dObMmdMKOR3HcboSSU/V3ysgFyUi6RaCGPVKziJI2PY5M7ta0geAnxDE1FfOKIb4JHdIOpXA5MW0adPidnEcx3EyoHDzRCStASaZmYUzgNeY2QRJFwG3mdkV4X4LgCPrmbN6e3vNRyKO4zjJkTTXzHqT7FtEx/oytmRUfRvwt3D7euCUMErrMALlUlOBOI7jOK2liD6RTwDfV7Ci3EZCsxQwE3g3QSqH9QT5gBzHcZwcKZwSMbM7gTfElBvwyfZL5DiO41SjiOYsx3Ecp0NwJeI4juOkxpWI4zhOG+nrH+CqOUsYGChWZGxaCucTcRzH6WYuuXMx3/zDY5gZHzy48+ex+UjEcRynjTy/7hUA1mzYnLMk2eBKxHEcx0mNKxHHcZw2UkoSothMTsmZfsYN/MevH8hAouZwJeI4jtNGSu50NadDAPjt/c8030iTuBJxHMdxUuNKxHEcp41kkfO2v0Dhwa5EHMdx2oiFBi01Yc/auLk/K3GaxpWI4zhOh+FKxHEcZ5iShTlrY98AACMycM43iysRx3GcHGim/y+NRMaM6slGmCZwJeI4jpMDzYT4uhJxHMcZpmSxJPmm0Jw1qid/e5YrEcdxnByIdv/9A8aXr3mYRSvXJTq2FOE7IosZi03iSsRxHKeNxI1DHl2+ll/d8zSn/+r+ZG2EoxlXIo7jOMOMwdxZTSiAgcE2MhCoSVyJOI7j5EAzCmDAShMWMxKmCVyJOI7jtBGLNWg1xoCbsxzHcYYnW1LBB8x7Zg1/mLe8qTbyxJfHdRzHyYNwFPHeH9zZ8KE+EnEcpyN4ctXLvPEbt7Jizca8RRkWJDV0lbL4FkCHuBJxHKc6l9/9FMvWbOR3Dy7LW5SuYXBRqmba8HkijuM4wxOrEZ6bVCUMe3OWpBMlPSJpQFJvRd2ZkhZKWiDpXZHyo8OyhZLOaL/UjuM4rSWpOcvnicA84Hjg9mihpH2Ak4DXAUcDP5LUI6kHuBA4BtgHODnc13Ecp8PILsS3mQmLWZGLEjGzR81sQUzVccCVZvaKmS0GFgKHhJ+FZrbIzDYBV4b7Oo7jdBRbwnMbUwC/+MuT3P74yrCNkjkrU9FSUTSfyBRgSeT70rCsWnkskk6VNEfSnJUrV7ZEUMdxnGZo1Cdy9vWPcMpP7wWKZc5q2TwRSbcAO8dUnWVm11U7LKbMiFd2VceEZnYxcDFAb29vcVa0dxxn2FMrE3xyn0hxHOstUyJmdlSKw5YCUyPfdwNKsYXVyh3HcTqOZrr/gQLNWC+aOet64CRJoyXtDswA7gXuA2ZI2l3SVgTO9+tzlNNxHCcVWeTOsgI51nNJeyLpfcAPgB2AGyQ9YGbvMrNHJF0FzAf6gE+aWX94zOnALKAH+KmZPZKH7I7jOM1Qa55IUoqUxTcXJWJm1wDXVKn7GvC1mPKZwMwWi+Y4jtMWGo3OijIQrI5bCJ9I0cxZjuM4XU0WkT4DHuLrOI4zPLEMkmelnWvSClyJOI7j5EBz0VnF8Ym4EnEcx+kwijTZ0JWI4zhOGymF+DYTnttfoMmGrkQcx3HaSQYTBc2ViOM4nUQWE+Sc7BjwlQ0dx3GGJ9mE+AZ/fSTiOE5HUIRQ0m7BakRWWa3sjBE8OstxHGeY04wC8DXWHcdxcuTGeSt4+vn1uZy71lgjacSWz1h3HMfJiYEB47TL53L8j/+cqxxxJsLk5qwtreSNKxHHcYYVqzdsBmDVuk25nD+hnqiJj0Qcx3Fy4vl1rwAwfvTIsjd/M6N/wAbDZxsl6SjCBv+m1ya1nPPtxpWI4zjDiudfDkYg6zb1sfuZM7l67lIAfjz7Cfb88kz2+PJM5j2zpqE273/6RXY/cyb3LHo+8TFxOie5TyT46451x3E6gk6cbLi5f6CsU39m9QYWr3qZPy9cBWzpxK+5/xk29Q1w3o0LBvc9b9YCXn6lD4DlazZw24LnWPpiuSP+sRVrWbZ6A3967Dku/NMTAHz/1r8x75k1PLN6A7c/vpL7nnyh7JjFq17mmbCdexeX1wUyNRbiWwQlksuiVI7jOK3m/FkLuOj2Rfzu9CPYb7eJvOmbf4zdr2eE+M9rHy4ru/3xlZx2+Vwu+9ihHP6NLcc9+c33DG4f/b07hrT1lyee570/uLOs7PefOoJ9p0wE4K3n3zZYfuV9Szjl8OmNXhYQdaznjysRx3Hq0mmTDa+as4RL73oKgDsXruKexdXNTLMfXxlbHjdS+P4tf2PD5n6+8M5XJ5blwj8t5EOHTuPNM3YYUvfypr7E7URJOmJpB65EHMfpOr74m4cGt79142Op2nilb2BI2XdveRyAQ/eYnLidP8xbwR/mrSgbxTRLyZxVBDOj+0Qcx3Gq8JcnVsWWl/wljTD9jBsStXPPoufZ88szeeHl6iHIBRqIuBJxHKc72Li5nzXrN2fa5gW3/i22/J5FQ01dadiwqX9I2f/MfoL+AeOBJS9mco5W40rEcZyu4L0/uJMD/uumTNsc1RPfRV5291OZtP9yjBIpUWu0YQn2aReuRBzH6Uiefn49d/5ti7lp4XPrABqaLPj7Tx3BMfvuXLV+ZINTwo8/aEpD+2+Icaw3s+JhHrgScRynI3nLt//EP/7kniHlm/qHOsSrse+UiUwcO6pqfbWRSBw7TRjNka/ZcUj5EXttX/WY9TVGIrUojUB8JOI4jpMxcVFVcfz2399Yd5+b5j87pGzq5LGx+97z5aPoiRlFXPaxQwa3z3r3a8vqapmz0jLnyRf4yrXzMm+3GrkoEUknSnpE0oCk3kj5OyTNlfRw+Pdtkbo3hOULJV2gThvzOY7TFjbFKJE4s1TaDmT0yJ6qdXEDF0l88ei9ufLUwxhRIUecOSsJtUJ7T7zoLi67+yle6cteQcWR10hkHnA8cHtF+SrgWDPbD/gIcFmk7sfAqcCM8HN0G+R0HKfg/OzPi8vCZ+PMWVuNrN7VNWoSGl2jrWppSP79yL04bI/thmTdbXYkEqdMthkdTP97aWM6BdUouSgRM3vUzBbElN9vZsvCr48AYySNlrQLMMHM7rJgqualwD+0UWTHcQrKL+95uux73EgkTomkNWZElcg5x+5TVjeyp3abPUNGIimVSA3FN2FM4OPpaiWSkPcD95vZK8AUYGmkbmlYFoukUyXNkTRn5cr4lAaO43QH225d7hiPM+PUcpBX0yVjR8WbraIK6U0VTvN6CRErFdf6lOasEnGjqPFjSiORbOfMVKNlSkTSLZLmxXyOS3Ds64BvAf9aKorZraouNrOLzazXzHp32GFovhrHcbqHyo45diQSo0TqjUOqRfeOHLGlrZEV7VaONOq1mTo6q0ZdyZz1lyeSp6VvhpblzjKzo9IcJ2k34BrgFDN7IixeCuwW2W03YFnlsY7jDD8O3X1yWbLEpOaselQzdw1EXv8rHfZx0Vm16mtFkqUN352x0zbMeerFwXkzraZQ5ixJk4AbgDPNbHABZDNbDrwk6bAwKusU4LqcxHQcp0CMG13+Lvx/c5YO2Sd2JFJnKFKtPjraqDST1R+JVNTHLUxVW6zgsMEEjEPZduutgCb8LQ2SV4jv+yQtBQ4HbpA0K6w6HdgL+IqkB8JPafbOvwGXAAuBJ4A/tFvu4cyaDZvpa2ASl9OZbO4fYO3GzWzc3N+0vb5dVL6x/3rOkiH7jBrZuBO92hGltUFgqCO9rhJpw6LopdvRrueXSyp4M7uGwGRVWf7fwH9XOWYOsG+LRXNi6Osf4IBzb+Kkg6fyzffvn7c4Tgv5zJX3M/PhFUwYM5K1G/v4+BG75y1SXZKkQ49zrNdbI6WaOStaOmpEebuVx/S+atuy75U6JG0q9yQz1tP6WxqlUOYsp5j0hbmIrrn/mZwlcVrNzIdXALC2TeGhWXLQtElV6+LMWfVIEgFcL6T3F/9ySNn3RpazTZvRpKRYXIk4jtN2RtXpFItIqdOcNnnrqvvEzxOp3W6SOzGyR+w3ZSIHT982tr7SX1NpzoobSSTRM7UUTGl009XmLMdxism40SNZnfGaHEUg3UikijkrUjxqxAh+96kjYuviyN4lskWd3DL/WZ58/uXB7+1yrLsScRxnkG26VImkcWjXOuLaT76JG+etaLjdyhDfZk1WUT5+6RwA/vUtewCw324Th+7UAlyJOI4zyDajO7dLqNWdx/X1aUN8AQ6cOokDp1b3wVRvs9KcldKxHqqfaoePGTWCi/6pN74yY9wn4jjOIPUcxUUkbUdcn2rRWenvUSMDl0aua/6ytVuOa0SgDHAl4jhOXdKGoraDUl9bK6FiXMdfP8S3KbFiqZxHEn9X6584ql8eXLKad19wR0UL7XsZcCXi1GUwJj1fMRynJrW6zTQKIU03XD8fV7aduwHL12woL2vzcoeuRBzHqUs732wbZbDLbFDEZnwi1dh75/GMGZU8Y3AW/X3PiMbDl7PElYhTl5Ipo7jdiJMVRVYWzZBuJFI/xLeSrbcayb1nVc89Wy8tSqPEjTrave66KxHHcTqatJ1mIyORn/5zL5PHbZWs3Rp1leas9CG+W46snFRodWTIGlciTl3a/Wbj5Ec7zSBZU2sUlWaEFT1i2uSt+cjh0xMdV8vvMTSLb/P/ueLSm6RdtTENrkQcx+loEkWOxc0TSZmAse6pahyWlTXLIn8rlYibs5zC4QMRpxPI+uW7vL3kjdcciSQK8W2MuOWAC2fOkrSnpNHh9pGSPh0uIOUMA9odMujkRydas5L8POOuq9HorCwCTLLK4htNBV95/e2e05N0JHI10C9pL+AnwO7Ar1omleM4haLQkw3Dv1krwKi5q5FRTm2fSPn3tFl861LAEN8BM+sD3gd8z8w+B+zSOrGcIlHc7sPJnA72rNf6ncb5N+pdaeUh24XRWfWitGr5PYbOWG8udxYMtRRcetdTqdpMS9Jsa5slnQx8BDg2LBvVGpEcxykaRZ4/8nQk/TnAe/bfhTfvtT1n/PbhwbJmZ58L+NChr2LMqB6Of/1utY9rJDqrgkYtxxZzTP+AFc8nAnyUYD30r5nZYkm7A5e3TiynSLhLxCky1z6wrOz7QVMn8dpdJtQ9rr5PZOj66Sf2Tq2/jnqN6kYifNPWtZtEIxEzmw98OvJ9MfDNVgnlFIwC/WAdpx6xpqsmc2c1Eu7baCLI8mMbGzWZxRvE2jlPpKYSkfQwNboQM9s/c4kcx8mN4hqt6lPe6Td6RExtpDqr+1JvJJJ0hBHdrWXO+YTUG4m8N/z7yfDvZeHfDwPrWyKRUziKHJnjOCWiUVqVb/ypfCIFDjKIUxzHHrArv3tw2dCKFlNTiZjZUwCS3mRmb4pUnSHpz8B/tVI4x3HaS4H7zcTEXUMaE1fjI5v6DJ170jxxL3lFdKyPkzS4Gr2kNwLjWiOSUzSK5MRznCRk0em3QqFWjpAqw3PLz1lzumFM2/mQNMT3X4CfSZpIIP2asMwZBljFX2f40UkmzdjZ6QnLyusVu91KzBpTXtEZ62U+nKI41gEkjQD2MrMDJE0AZGZrWi+a4zhORqSJzmrFSCSrBIwRnW4xZe2krjnLzAaA08PttVkoEEknSnpE0oCk3pj6aZLWSfpCpOxoSQskLZR0RrMyOMkpDbm7wFzu1KHaM27kTfx3Dy7jlvnPZiNQCpK+hTcWtptWmop2smlmkGojxCL6RG6W9AVJUyVNLn2aOO884Hjg9ir13wX+UPoiqQe4EDgG2Ac4WdI+TZzfcZwW8akr7ufjl87JVYZsfCKt74rTjh4sJsY3r6CIRnwisCXUF4JR1B5pTmpmj0K1iAn9A7AIiOYyOARYaGaLwn2uBI4D5qc5v9MYnWMNd5zknWl9n0j2NOrvSLRf2G6ciasdJJ2xvnurBQGQNA74EvAO4AuRqinAksj3pcChNdo5FTgVYNq0adkLOswYTDudrxiOk4i4eSKJj1U1R3XzcoUtlX2rFbBQOzYrqB2UtaL+lc1D1xhpFUlHIkjal8CUNKZUZmaX1tj/FmDnmKqzzOy6KoedC3zXzNZVjFLiHmGtmfQXAxcD9Pb2et/nOAkp8gS7rGl0PZFWkEV+rLj9XukbSCdQChIpEUlnA0cSKJGZBL6JO4GqSsTMjkohz6HACZLOAyYBA5I2AnOBqZH9dgPaPzVzmJLFYjxOZ9NJIb5I2fhEytYTyebX35A5q9YoJbooFTZEvr6B9j2vpCORE4ADgPvN7KOSdgIuyVoYM3tzaVvSOcA6M/uhpJHAjDB78DPAScCHsj6/4wx3uuVFIUlnHWfyEpH0KS2ZbFhOktUL07TbTpJGZ20IQ337wrkiz5HSqQ4g6X2SlhKkl79B0qxa+4cLYp0OzAIeBa4ys0fSnt9pkA56CXWcZjrU6Bt95Xoi7aKk3JLOV887o0TSkciccE31/yUwLa0D7k17UjO7Brimzj7nVHyfSWBKc9qMz1h3OokgnXplAsZkubPSpn9PSmWblWlPqtVV289Co1eerqyk0Vn/Hm7+j6QbgQlm9lDrxHIcp0gUeWXDVtGK6Kys7mKlThHK7SUvqWP9UuAO4A4ze6y1IjlFo1oYodN9dENwlsjGsd4KGsnim8RMFc2dlRdJfSI/B3YBfiDpCUlXS/pM68RyHMdpLfEp4yPb0fIUr1BH7LV9/Z1qLCg1UGbOqjzMyrcbXBExS5Kas/4oaTZwMPBW4DTgdcD3WyibUxA6KrzTGfY0usRs2bFl8VnlbTbCwq8dw4i4jBwNSNZodFbRzVm3EqwfcheBWetgM3uulYI5juO0kljHeUbp1Ef2xBt5GjJn1dovmkWiQ8xZDwGbgH2B/YF9JY1tmVROocjb5uq0j2pvyp0wGo367rL2ibTLVFRmpqoVuVWx3QnRWZ8DkLQN8FHgZwQpTUa3TjSnKBS/+3CcLcR1qGmWkG11v3zA1EmsWb+pan35XJAq/wsHFWd+WiSpOet04M3AG4CngJ8SmLUcx+kmqvRFnRfimzIBY7bNDW0m0s707bbmwQolUnafa+bVSjZiaQdJJxuOBb4DzA1njzvDiLx/pI7TCEkVXt3orHYkYIwpK523PAIrckzF/8dGl9TNmkQ+ETP7NjAK+CcASTuEeawcx3EKQdYKoBVrrFemVakZulvl3a1/YMteRUiOmkiJhFl8vwScGRaNAi5vlVBOsfCByDCig5/14O80JsQ3PtlicyG4aWgkrUq1/FiVCXrzfmRJo7PeB/w94WqDZrYMGN8qoZxikveP1Wk9lQ5of+YBrTIXVd7vqBKr9vI2YFaeCt7yXQcmqRLZZIEhzmBwBULHcbqMIeaVDtUiSTrVuF323DHStVWZvd4M1WbElxgyEz2GgZiHUnhzFnCVpIuASZI+AdxCC9YTcYqJ584aPlR2T3EdVrdyzL47850PHNjSc1Say5KubBhVKE+uWl8xTyTfZ5TUsX4+8BvgamBv4KtmdkErBXMcp/1UUxp5d1SNkuSFp3Kft7x6B8aNHhlb3xJzUZ0mq93xd19wx9B9ij7ZEMDMbgZuBpDUI+nDZvbLlknmFIZO60Cc9Aw1Zw3fZ98SvVGZ9qTi9m7cPMDMh1cAMGveCvbeaTyH7D655nMxK7A5S9IESWdK+qGkdyrgdGAR8IH2iOjkzaATL18xnDZQ+Yy3BDwVy5hZT7klUgB1FqVKUt4o9TIDP/3C+sHtOxeu4gMX3VWzvSIo+XrmrMsIzFcPAx8HbgJOBI4zs+NaLJvjOO0mZiJbEaklV3YdfmtNWElHOq///29m5UuvlJX9/qHl5c3mGJ1Vz5y1h5ntByDpEmAVMM3MXmq5ZE5hiITfO13O0JFIQbVIHZIogKRL5tYqb5aXN9VPAPLCy5u4af6zVevNDCm/UUm9kcjm0oaZ9QOLXYE4TvdS6VivnNhWFNohVmt86eUz1lev31x95wib+wdiywfnXeRIvZHIAZLWhtsCxobfgxn7ZhNaKp1TCIpgd3XaQ/fME0m3T7URTHZpT9Idt7kvXokMtkt+Jq2aSsTMetoliFNcrOKv070MVRoW/lusp98NLzaN9PlVRyKRGetFNWc5jjOMGOITKWhfnXRFwFrE9eFVO/asUsGnPG5Tf60FqjpgsqEzvPEZ68OHuDTjnUgjb/lJrjErS1F5Ft/kjfZVGYlsaSs/XIk4jgPAc2s38tiK8riZoqY9ySLEN86H0M7OOBNzFh2ynkjWSDpR0iOSBiT1VtTtL+musP5hSWPC8jeE3xdKukB5BkYPO4rZkTjZUqlAoLiTDevRSPeQyAnfhCxZtFPbnJUveY1E5gHHA7dHCyWNJFin5DQzex1wJFvCjH8MnArMCD9Ht0vY4Y7PWB8ebL3V0Diagg5EMvEDxHboLdaVaRfOqjYSKVtEJSdyUSJm9qiZLYipeifwkJk9GO73vJn1S9oFmGBmd4Up6S8F/qGNIjtO1zN6ZIwSKeirQz3lVq1LPfvYfVKdL2/DR1UlwjA1Z9Xg1YBJmiXpr5K+GJZPAZZG9lsalsUi6VRJcyTNWblyZQvFHR4UsxtxsqY/rmfusocfHW01Mk8kK8rbT36uWkpkSxh2PiTO4tsokm4Bdo6pOsvMrqshzxHAwcB64FZJc4G1MftWvWdmdjFwMUBvb2+X/TdwnNbQPzC0o2r0P0+e8zeio6as38wzay5lQ5vugewbAAAWZklEQVSqTDbM35jVQiViZkelOGwpMNvMVgFImgm8nsBPsltkv92AZU0L6STCQ3yHB30xzttSdFZSs1ZRfSglytOOJM+d1RJZ6pzrZ/98MB/9+X1AHcd6zv8/i2bOmgXsL2nr0Mn+d8B8M1sOvCTpsDAq6xSg2mjGcZwU9MUkympUKbRLh9T3iaQLuao61zCzeSLJ933ra3Yc3K6W9mTLjPX8zFl5hfi+T9JS4HDgBkmzAMzsReA7wH3AA8BfzeyG8LB/I1iSdyHwBPCHtgs+TCmqc9XJllgl0mAb7ZpX0u7fZGa5syLbjdyq519+pWZ9niHYLTNn1cLMrgGuqVJ3OYH5qrJ8DrBvi0VzYvAQ3+FBrE+kQaWQpzmrzFSVIH1JrGO9xfasaPsDDaRIfvzZdVXr8n7JK5o5y3GcnIjziTQ62fB7tzyeoUTVSRvim5ZW6JbYaLgGMcxDfJ3ik7fjzmkP/bE+keQdnZnxo9ueyFKklhKbgLGN52xkJNJIu+3GlYgzLDEzpp9xA5fcsShvUQpDnE9k5sMrEh/fzgWs4k5VZtaptkJhS6RJTnTEUDkS6RnRuHRm+ZuZXYk4dcnb5toKSh3e12c+mq8gBSJuJNII7ZwjksW5YhMwtlHLVN7vnpQnL60nkheuRJy6dKNjvRsWNcqauJFIIxTpjmYdrZRZiG9ErspItk5NKetKxBmWFKnDKwq11qxIMhptZ9r4LM4U7xNpdXTWlu3KkciIFFokMGd5dJbTIXToi1IsRV0nI0+aHokU6JZmn/Yk+19/pc5uRmaPznKcNlOkDq8o/M/szomsqrkoVcIeNXa3ag75FnTSWbzIWPiPKxGn0HiHOzxY+uKGpo5v6++kzfNEsiLa2Wfll8v7v6crEacuJZtr3j/WLHFzVmMkMed0wj2NSthIAsZWKKXKu9XM7RP5Jc9yJeIMSzqgv2s7e+80nvGj02dCau9ApPbZ8l5EqhpRxZXFb9DMMDM3ZznFphs73C68pKYZM2oEb5i+berjOy1sOn5Rqmr7ZpSAMWrOqqhLG2U1eFROisSViDMs6bQOrx0MWLow0xJtHYnUOJmZFdYnEiXL36DAzVlOcSnC6mlZ084UHZ3CQJOdr9VawTVj6j2+tLqw2ogjq99+rXbShhH7jHWn8JTemLqq3+2qi8mGgRqdURJTS96T3ko02qG2c1Aala3yvG7OcpwOoigdXpEwM5LmANzUN8CzazdWHN8CoapQzxSU5K2+MZ9IAqESkHU/X5Zh281ZTlHpxu7WzVlDsQZ8Ip//vwc59Ou3lqVK6dRbmuSSW2EuqnyRSauEzSxXW7MrEWdY4o71oQyYMSJhjzBrXpAiPprOvFC5sxLM92hknkhWlE82HFo/5z+Paqg9K8CYOpflcZ3OohsXpcr7P14RGTBryp/QXnNW+86VJbV8IhJsv81ott16FJ94yx6NtZuFcCnxkYiTgO5zrHfC7Op2U8ucleR25f5OHF2TKpGJKqasjd1x6X598ei9g++h/Pd/9Z38+5F7JWvDCHNneXSW47QX1yFDGGjAsV6iTLm0dWXDJiOZCkDp3qVdjGqwHU8F7xSdbnxp78JLappakw2r3a+BMp9IC4RKSZJuOTaJbw4v9GmWxa3EzVmO02a6UTE2y0CKHExRJdLWN+IueH6lS2g2S4B5Knin6HTnjPUu6IUyJo1PxBLs0wrqz1hPMk8k5190yZzV5Egk75+yKxFnWOIqZCipfCKRVCd+TxujNHIrKZE0968Uqt7OgIBKclEikk6U9IikAUm9kfJRkn4h6WFJj0o6M1J3tKQFkhZKOiMPuYcreb/ptAKfJzKUQIkk64xKHWCZT6SNTpHYx6fYzark7ROxrEYiDM9U8POA44HbK8pPBEab2X7AG4B/lTRdUg9wIXAMsA9wsqR92inwcKYbc2e5DhlKmtxZhTILNhjimzclcTvdnJXLZEMzexRif7AGjJM0EhgLbALWAocAC81sUXjclcBxwPx2yex0F3n/xysijeTOCswnlqNPpPmT5T5PJLxhgyG+KS6pPMI6nx910XwivwFeBpYDTwPnm9kLwBRgSWS/pWGZ0wa60bGed2x9EakZ4lsl42xe0Vn1FFaePoJGad6clW+QQMtGIpJuAXaOqTrLzK6rctghQD+wK7AtcEfYTtwdqvozknQqcCrAtGnTGhHb6WCWvLCep19Yz5v22r7uvkWa01AUmp1sWOR7GjvqiClsq08k/DuidNPTnLsA97xlSsTMGsskFvAh4EYz2ww8J+nPQC/BKGRqZL/dgGU1zn0xcDFAb29vAW5zZ9Mppp+3fPtPmMGT33xP3X3dsT6UgYHkubMGzVllubOKk4Ax9aJU6Q5LxZAZ66mz+PpkwyhPA29TwDjgMOAx4D5ghqTdJW0FnARcn6Ocw4qSmaLo3W4jfVjRryUPGkkFX6LcnFUMOsWQlZVjneEYnSXpfZKWAocDN0iaFVZdCGxDEL11H/AzM3vIzPqA04FZwKPAVWb2SA6iO12CD0SGUsucVW2UUaZE2jkSafBcSXePjsRa3jGXHOuD80Qav39F+BnnFZ11DXBNTPk6gjDfuGNmAjNbLJoTRxF+qRnj5qyhDFjEPp8QTwWfni0jkSbb8bQnjtN+uqAPypw0ubPKlEi24jRFJ/lESibENBFlZvnHGboScerSlSG+ef/PKyBpcmeVm7NaIVV+tOt6mjFnlRh2aU+czqL0n6mb+ohCzbQuCKlCfCuOLwJG+k61vSG+5T6R1O2kGEFmiSsRp+tI4u8oSH9XKGrlzkqynoj7RBoji0WpjC3PJq974krEqUv+VtfG6E8w663Trqkd1MqdVY2owm7rjPUa5xJN+ETa+EpfmYCxGSXg80QcJ0P6fSTSMCVlUD3Et355ke5pJ/jvhsxYT9OGhfddys2k5UrEqUupc+iE/5gAAwP19ylSh1cESoO3xicbbtkusjmryFl9m1nZENyc5XQAVvG36CQaiXTM1bSHgXojkQSp4It0T7MwS7Va8dQb/TWCm7McJ0MS+USK098VgpIyaNwnEr/darrp8ZV8Ip95+4yGjzUs9+isXGasO51Fp83uTrLCXlHCUYtC5cS3pOSVO6veb7LA1qtBovc8SdLQouIjEafrSGbOcqLUNWdV3LDBpJxlqeC76662+nJK97CZUUQR/JU+EnHq0mkz1pOMRLqsv2uapI71L/7mQW6a/2zkuJzmidQpK7IjvcQWBdD88rhduSiV00V02Iz1ZCG+nXI17WGLTyS+vnS3rpqzFICtekYw1JXu97QRBl/Omuz/8w5ocHOW03Ukm2zoRLEwLDrpG23s8rgFCvGtvI4ivzM0o0OKYM5yJeLUJe83nUbxeSKNszm8aaN6GpttGB3RtXd53NafrF0hvs2346ngHSdT3JzVOH39wf0YOSJZl1Cy4+e1PG4cHeAGKSMzc1bOP2VXIk5d8v6RNkoSc1Z735qLz+b+YCQysspIpPJ2bTFnVd+nlcT9JjvukQ4KnMHESE8F7zjZkSTUtNNMdK2mL9QGVc1ZVeiU9USKGK2VqWPdzVlOkSmC864RkoxEXIeU01caiSQ0Z5XIy5zVTY+vOce6bVnvx3NnOUWl43JnuTmrYTb31x6JVHZQpZFLeSr49tGODrPlkw1TppqJw6OzHCdD3JzVOH2D0VnJuoTSLc4ri283kMUkXgs/Un4mO1ciTl3yjrpplEYSMA4Y3DhveYslKj6lkcjIKkqkmtKNlrcz7Uk7XgJaH+Jb7PaS4krE6Tpe6as/USTa4Z12+V9bKU5HUIrOGtVgXvJa0Vmd8vLx2aNezdtfsyPHHrBrW8+bRe6ssCGPznKKTaflztqwqT+2fL9zZvGj2xYCnePfaRd99UYiVW5YeXRWG0ciNU7VaKe888Qx/OSfD2ab0e3NApVF7iyzQBm5OcspNIPRH/mKkZj1MUqkr3+Alzb2cd6NC4KCTrmYNlGasV5tnkhVao5EmpOp5mm74PkNKpGMOn83ZzlOk5QW91m/qW9I3bpXysvcsV5OaSQyqkqIb7W7VWsk4ne49QSLUg3DtCeSvi3pMUkPSbpG0qRI3ZmSFkpaIOldkfKjw7KFks7IQ+7hSzG7g0Ur1/G7B5cNfh8zMvg5x41EXtpYrkSS5NcaTvTVmbFejahP5NHlL5XVBXMYjEvvepI16zc3K2J52234Tbbrzb75yYbD0ydyM7Cvme0PPA6cCSBpH+Ak4HXA0cCPJPVI6gEuBI4B9gFODvd1hjHH//gvfOqK+9m4OVAaY0b1APFKZM2G8k6smGoxPzZnMGP927MWDKmf+9SLfPW6R/jq9fOaE7ALyXKeSJ7ksp6Imd0U+Xo3cEK4fRxwpZm9AiyWtBA4JKxbaGaLACRdGe47v1UyHvuDOwc7p+FOyRS0qW+Ad3xnds7SbGF1+HZ79PduZ1TPCFaHiuJ/71jEb/+6tGzfDZFn+Y7vzB5i3irSdeXB2o3Bvas2Y/3Ke5/mlshiVCXOuf4Rzo9RHgBHf/8O1of3+boHljF/2dqMpIWNfUP/b24VBgX0xESYbRWOUkc2EH02NnwpaRVjtgrab0aFrFq3idXrN3PI7pMZPSqfMUERFqX6F+DX4fYUAqVSYmlYBrCkovzQag1KOhU4FWDatGmphNpzh3Fs6nebR4lNi17goGmTBv8zFoGdJ47h0eUvsc+uEwB49U7jeXH9JiZtPSp2/019A7x2lwmMGx385928+AXGjxnFhDEjmbLt2LbJXVS2Gzea3bYdy1X/ejhn/vYhdp00lmWrN7DkhQ0cvud2AEwetxUr1m5kxo7b8MCS1Rw0bdASzY4TRvPnhc/T+6pt2WnimME37dULVvKWGTvQYEaVuuy903j6B4z5y9dy7t/vy+F7bMekcaN4z367AHD2sfvw0sY+XrPzeA7dYzvGjxnJsQfsyksb+8peKir53FGvZu7TL3Lhh1/PKT+5l3e+bqdM5T7vhP3ZY/tx7Dh+DNc/+Ay7TBxTdd+r/+1wbluwksWrXmb+srUcvud2zH3qRbbdeisOmDqJJS+sxzCO3X9XDttjO3adNJa9dxrPdttslanMtVCrwvIk3QLsHFN1lpldF+5zFtALHG9mJulC4C4zuzys/wkwk8Ds9i4z+3hY/k/AIWb2qXpy9Pb22pw5czK5JsdxnOGApLlm1ptk35aNRMzsqFr1kj4CvBd4u23RZEuBqZHddgNKntNq5Y7jOE5O5BWddTTwJeDvzWx9pOp64CRJoyXtDswA7gXuA2ZI2l3SVgTO9+vbLbfjOI5TTl4+kR8Co4Gbw8iEu83sNDN7RNJVBA7zPuCTZtYPIOl0YBbQA/zUzB7JR3THcRynRMt8IkXBfSKO4ziN0YhPpDihNo7jOE7H4UrEcRzHSY0rEcdxHCc1rkQcx3Gc1HS9Y13SSuCplIdvD6zKUJxOwK95eODX3P00c72vMrMdkuzY9UqkGSTNSRqh0C34NQ8P/Jq7n3Zdr5uzHMdxnNS4EnEcx3FS40qkNhfnLUAO+DUPD/yau5+2XK/7RBzHcZzU+EjEcRzHSY0rEcdxHCc1rkRikHS0pAWSFko6I295skLSVEl/kvSopEckfSYsnyzpZkl/C/9uG5ZL0gXhfXhI0uvzvYL0SOqRdL+k34ffd5d0T3jNvw6XGCBchuDX4TXfI2l6nnKnRdIkSb+R9Fj4vA/v9ucs6XPh73qepCskjem25yzpp5KekzQvUtbwc5X0kXD/v4VrO6XGlUgFknqAC4FjgH2AkyXtk69UmdEHfN7MXgscBnwyvLYzgFvNbAZwa/gdgnswI/ycCvy4/SJnxmeARyPfvwV8N7zmF4GPheUfA140s72A74b7dSLfB240s9cABxBce9c+Z0lTgE8DvWa2L8GSESfRfc/558DRFWUNPVdJk4GzCZYYPwQ4u6R4UmFm/ol8gMOBWZHvZwJn5i1Xi671OuAdwAJgl7BsF2BBuH0RcHJk/8H9OulDsBLmrcDbgN8DIpjJO7LymROsWXN4uD0y3E95X0OD1zsBWFwpdzc/Z2AKsASYHD633wPv6sbnDEwH5qV9rsDJwEWR8rL9Gv34SGQopR9jiaVhWVcRDt8PAu4BdjKz5QDh3x3D3brlXnwP+CIwEH7fDlhtZn3h9+h1DV5zWL8m3L+T2ANYCfwsNOFdImkcXfyczewZ4HzgaWA5wXObS3c/5xKNPtdMn7crkaEopqyr4qAlbQNcDXzWzNbW2jWmrKPuhaT3As+Z2dxoccyulqCuUxgJvB74sZkdBLzMFhNHHB1/zaE55jhgd2BXYByBOaeSbnrO9ah2jZleuyuRoSwFpka+7wYsy0mWzJE0ikCB/NLMfhsWPytpl7B+F+C5sLwb7sWbgL+X9CRwJYFJ63vAJEml5aGj1zV4zWH9ROCFdgqcAUuBpWZ2T/j9NwRKpZuf81HAYjNbaWabgd8Cb6S7n3OJRp9rps/blchQ7gNmhFEdWxE4567PWaZMkCTgJ8CjZvadSNX1QClC4yMEvpJS+SlhlMdhwJrSsLlTMLMzzWw3M5tO8Cz/aGYfBv4EnBDuVnnNpXtxQrh/R72hmtkKYImkvcOitwPz6eLnTGDGOkzS1uHvvHTNXfucIzT6XGcB75S0bTiCe2dYlo68nURF/ADvBh4HngDOylueDK/rCIJh60PAA+Hn3QS24FuBv4V/J4f7iyBS7QngYYLIl9yvo4nrPxL4fbi9B3AvsBD4P2B0WD4m/L4wrN8jb7lTXuuBwJzwWV8LbNvtzxk4F3gMmAdcBozutucMXEHg89lMMKL4WJrnCvxLeO0LgY82I5OnPXEcx3FS4+Ysx3EcJzWuRBzHcZzUuBJxHMdxUuNKxHEcx0mNKxHHcRwnNa5EHCcBkvolPRD51MzuLOk0SadkcN4nJW3fbDuO0yo8xNdxEiBpnZltk8N5nySI71/V7nM7ThJ8JOI4TRCOFL4l6d7ws1dYfo6kL4Tbn5Y0P1zT4cqwbLKka8OyuyXtH5ZvJ+mmMHHiRUTyHEn6x/AcD0i6SMEaKT2Sfh6uofGwpM/lcBucYYwrEcdJxtgKc9YHI3VrzewQ4IcEebkqOQM4yMz2B04Ly84F7g/LvgxcGpafDdxpQeLE64FpAJJeC3wQeJOZHQj0Ax8mmJk+xcz2NbP9gJ9leM2OU5eR9XdxHAfYEHbecVwR+fvdmPqHgF9KupYgBQkEKWjeD2BmfwxHIBOBtwDHh+U3SHox3P/twBuA+4LUUIwlSLT3O2APST8AbgBuSn+JjtM4PhJxnOaxKtsl3kOQw+gNwNwwa2ytdNxxbQj4hZkdGH72NrNzzOxFgpULbwM+CVyS8hocJxWuRByneT4Y+XtXtELSCGCqmf2JYGGsScA2wO0E5igkHQmssmBtl2j5MQSJEyFIrHeCpB3DusmSXhVGbo0ws6uBrxCkfHectuHmLMdJxlhJD0S+32hmpTDf0ZLuIXgpO7niuB7g8tBUJYL1vldLOodg5cGHgPVsSeV9LnCFpL8CswlSnGNm8yX9J3BTqJg2E4w8NoTtlF4Iz8zukh2nPh7i6zhN4CG4znDHzVmO4zhOanwk4jiO46TGRyKO4zhOalyJOI7jOKlxJeI4juOkxpWI4ziOkxpXIo7jOE5q/h9T0xIv00XLXgAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "plt.plot(reward_list)\n", - "plt.title(\"Rewards vs Episode\")\n", - "plt.xlabel(\"Episodes\")\n", - "plt.ylabel(\"Rewards\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 2", - "language": "python", - "name": "python2" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 2 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython2", - "version": "2.7.15" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/English/RL Tutorial 3/DDPG.ipynb b/English/RL Tutorial 3/DDPG.ipynb new file mode 100644 index 0000000..50f8245 --- /dev/null +++ b/English/RL Tutorial 3/DDPG.ipynb @@ -0,0 +1,714 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Reinforcement Learning Tutorial -3: DDPG" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### MD Muhaimin Rahman\n", + "contact: sezan92[at]gmail[dot]com" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the last tutorial, I tried to Explain DQN. DQN solves one problem, that is it can deal with continuous state space. But it cannot output continuous action. To solve that problem, here comes DDPG! It means Deep Deterministic Policy Gradient" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- [Importing Libraries](#libraries)\n", + "- [Algorithm](#algorithm)\n", + "- [Model Definition](#model)\n", + "- [Replay Buffer](#buffer)\n", + "- [Noise Class](#noise)\n", + "- [Training](#training)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Importing Libraries\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/sezan92/anaconda/lib/python2.7/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n", + " from ._conv import register_converters as _register_converters\n", + "Using TensorFlow backend.\n" + ] + } + ], + "source": [ + "from __future__ import print_function,division\n", + "import gym\n", + "import keras\n", + "from keras import layers\n", + "from keras import backend as K\n", + "from collections import deque\n", + "from tqdm import tqdm\n", + "import random\n", + "import numpy as np\n", + "import copy\n", + "SEED =123\n", + "np.random.seed(SEED)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Important constants" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "num_episodes = 100\n", + "steps_per_episode=500\n", + "BATCH_SIZE=256\n", + "TAU=0.001\n", + "GAMMA=0.95\n", + "actor_lr=0.0001\n", + "critic_lr=0.001\n", + "SHOW= False" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "from keras.models import Model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Algorithm" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The actual algorithm was developed by Timothy lilicap et al. The algorithm is an actor-critic based algorithm.. Which means, it has two networks to train- an actor network, which predicts action based on the current state. The other networ- known as Critic network- evaluates the state and action. This is the case for all actor-critic networks. The critic network is updated using Bellman Equation like DQN. The difference is the training of actor network. In DDPG , we train Actor network by trying to get the maximum value of gradient of $Q(s,a)$ for given action $a$ in a state $s$. In a normal machine learning classification and regression algorithm, our target is to get the value with minimum loss. Then we train the network by gradient descent technique using the gradient of Loss ." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\\begin{equation}\n", + "\\theta \\gets \\theta - \\alpha \\frac{\\partial L}{\\partial \\theta}\n", + "\\end{equation}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Here, $\\theta$ is the weight parameter of the network, and $L$ is loss" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "But in our case, we have to get the maximize the $Q$ value. So we have to set the weight parameters such that we get the maximum $Q$ value. This technique is known as Gradient Ascent, as it does the exact opposite of Gradient Descent" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\\begin{equation}\n", + "\\theta_a \\gets \\theta_a - \\alpha (-\\frac{\\partial Q(s,a) }{\\partial \\theta_a}) \n", + "\\end{equation}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The above equation looks like the actual Gradient Descent equation. Only difference is , the minus sign. It makes the equation to minimize the negative value of $Q$ , which in turn maximizes $Q$ value.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "So the training is as following" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- 1) Define Actor network $actor$ and Critic Network $critic$\n", + "- 2) Define Target Actor and Critic Networks - $actor_{target}$ and $critic_{target}$ with exact same weights\n", + "- 3) Initialize Replay Buffer \n", + "- 4) Get the initial state , $state$\n", + "- 5) Get the action $a$ from , $a \\gets actor(state)$ + Noise .[Here Noise is given to make the process stochastic and not so deterministic. The paper uses ornstein uhlenbeck noise process , so we will as well]\n", + "- 6) Get Next state $state_{next}$ , Reward $r$ , Terminal from agent for given $state$ and $action$\n", + "- 7) Add the experience , $state$,$action$,$reward$,$state_{next}$,$terminal$ to replay buffer\n", + "- 8) Get sample minibatch from Replay buffer\n", + "- 9) Train Critic Network Using Bellman Equation. Like DQN\n", + "- 10) Train Actor Network using Gradient Ascent with gradients of $Q$ . $\\theta_a \\gets \\theta_a - \\alpha (-\\frac{\\partial Q(s,a) }{\\partial \\theta_a}) $\n", + "- 11) Update weights of $actor_{target}$ and $critic_{target}$ using the equation $ \\theta \\gets \\tau \\theta + (1-\\tau)\\theta_{target}$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Model Definition" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "After some trials and errors, I have selected this network. The Actor Network is 3 layer MLP with 320 hidden nodes in each layer. The critic network is also a 3 layer MLP with 640 hidden nodes in each layer.Notice that the return arguments of function ```create_critic_network```." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "def create_actor_network(state_shape,action_shape):\n", + " in1=layers.Input(shape=state_shape,name=\"state\")\n", + " l1 =layers.Dense(320,activation=\"relu\")(in1)\n", + " l2 =layers.Dense(320,activation=\"relu\")(l1)\n", + " l3 =layers.Dense(320,activation=\"relu\")(l2)\n", + " action =layers.Dense(action_shape,activation=\"tanh\")(l3)\n", + " actor= Model(in1,action)\n", + " return actor\n", + "\n", + "def create_critic_network(state_shape,action_shape):\n", + " in1 = layers.Input(shape=state_shape,name=\"state\")\n", + " in2 = layers.Input(shape=action_shape,name=\"action\")\n", + " l1 = layers.concatenate([in1,in2])\n", + " l2 = layers.Dense(640,activation=\"relu\")(l1)\n", + " l3 = layers.Dense(640,activation=\"relu\")(l2)\n", + " l4 = layers.Dense(640,activation=\"relu\")(l3)\n", + " value = layers.Dense(1)(l4)\n", + " critic = Model(inputs=[in1,in2],outputs=value)\n", + " return critic,in1,in2" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "I am chosing ```MountainCarContinuous-v0``` game. Mainly because my GPU is not that good to work on higher dimensional state space" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n", + "\u001b[33mWARN: gym.spaces.Box autodetected dtype as . Please provide explicit dtype.\u001b[0m\n" + ] + } + ], + "source": [ + "env = gym.make(\"MountainCarContinuous-v0\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "state_shape= env.observation_space.sample().shape" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "action_shape=env.action_space.sample().shape" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "actor = create_actor_network(state_shape,action_shape[0])\n", + "critic,state_tensor,action_tensor = create_critic_network(state_shape,action_shape)\n", + "target_actor=create_actor_network(state_shape,action_shape[0])\n", + "target_critic,_,_ = create_critic_network(state_shape,action_shape)\n", + "target_actor.set_weights(actor.get_weights())\n", + "target_critic.set_weights(critic.get_weights())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "I have chosen ```RMSProp``` optimizer, due to more stability compared to Adam . I found it after trials and errors, no theoritical background on chosing this optimizer" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "actor_optimizer = keras.optimizers.RMSprop(actor_lr)\n", + "\n", + "critic_optimizer = keras.optimizers.RMSprop(critic_lr)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "WARNING:tensorflow:From /home/sezan92/anaconda/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:1290: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n", + "Instructions for updating:\n", + "keep_dims is deprecated, use keepdims instead\n" + ] + } + ], + "source": [ + "critic.compile(loss=\"mse\",optimizer=critic_optimizer)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Actor training" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "I think this is the most critical part of ddpg in keras. The object ```critic``` and ```actor``` has a ```__call__``` method inside it, which will give output tensor if you give input a tensor. So to get the tensor object of ```Q``` we will use this functionality." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [], + "source": [ + "CriticValues = critic([state_tensor,actor(state_tensor)])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now it is time to get the gradient value of $-\\frac{\\partial Q(s,a)}{\\theta_a}$" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [], + "source": [ + "updates = actor_optimizer.get_updates(\n", + " params=actor.trainable_weights,loss=-K.mean(CriticValues))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now we will create a function which will train the actor network." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [], + "source": [ + "actor_train = K.function(inputs=[state_tensor],outputs=[actor(state_tensor),CriticValues],\n", + " updates=updates)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Replay Buffer" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [], + "source": [ + "memory = deque(maxlen=10000)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 10000/10000 [00:00<00:00, 21304.88it/s]\n" + ] + } + ], + "source": [ + "state = env.reset()\n", + "state = state.reshape(-1,)\n", + "for _ in tqdm(range(memory.maxlen)):\n", + " action = env.action_space.sample()\n", + " next_state,reward,terminal,_=env.step(action)\n", + " \n", + " state=next_state\n", + " if terminal:\n", + " reward=-100\n", + " state= env.reset()\n", + " state = state.reshape(-1,)\n", + " memory.append((state,action,reward,next_state,terminal))" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([-0.49533114, -0.00344986])" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "next_state" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Noise class" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The use of Noise is to make the process Stochastic and to help the agent explore different actions. The paper used Orstein Uhlenbeck Noise class" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [], + "source": [ + "class OrnsteinUhlenbeckProcess(object):\n", + " def __init__(self, theta, mu=0, sigma=1, x0=0, dt=1e-2, n_steps_annealing=10, size=1):\n", + " self.theta = theta\n", + " self.sigma = sigma\n", + " self.n_steps_annealing = n_steps_annealing\n", + " self.sigma_step = - self.sigma / float(self.n_steps_annealing)\n", + " self.x0 = x0\n", + " self.mu = mu\n", + " self.dt = dt\n", + " self.size = size\n", + " def restart(self):\n", + " self.x0=copy.copy(self.mu)\n", + " def generate(self, step):\n", + " #sigma = max(0, self.sigma_step * step + self.sigma)\n", + " x = self.x0 + self.theta * (self.mu - self.x0) * self.dt + self.sigma * np.sqrt(self.dt) * np.random.normal(size=self.size)\n", + " self.x0 = x\n", + " return x" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "### Training" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Failed! Episode 0 Total Reward -121.829750\n", + "Failed! Episode 1 Total Reward -118.666569\n", + "Failed! Episode 2 Total Reward -162.041720\n", + "Failed! Episode 3 Total Reward -87.619343\n", + "Failed! Episode 4 Total Reward -96.425173\n", + "Failed! Episode 5 Total Reward -143.686660\n", + "Failed! Episode 6 Total Reward -38.429055\n", + "Passed! Episode 7 Total Reward 76.095012\n", + "Passed! Episode 8 Total Reward 78.345259\n", + "Failed! Episode 9 Total Reward -0.019448\n", + "Passed! Episode 10 Total Reward 48.135478\n", + "Passed! Episode 11 Total Reward 54.750747\n", + "Passed! Episode 12 Total Reward 79.410100\n", + "Passed! ward 83.0033361Episode 13 Total Reward 83.003336\n", + "Passed! Episode 14 Total Reward 83.174427\n", + "Failed! Episode 15 Total Reward -48.217783\n", + "Passed! Episode 16 Total Reward 87.284775\n", + "Passed! Episode 17 Total Reward 32.635075\n", + "Passed! Episode 18 Total Reward 84.165725\n", + "Passed! Episode 19 Total Reward 89.819676\n", + "Passed! Episode 20 Total Reward 76.334761\n", + "Passed! Episode 21 Total Reward 58.299554\n", + "Passed! Episode 22 Total Reward 81.035049\n", + "Failed! Episode 23 Total Reward -39.691471\n", + "Failed! Episode 24 Total Reward -161.331777\n", + "Passed! Episode 25 Total Reward 86.132543\n", + "Failed! Episode 26 Total Reward -104.653697\n", + "Passed! Episode 27 Total Reward 85.415800\n", + "Failed! Episode 28 Total Reward -74.066867\n", + "Failed! Episode 29 Total Reward -29.747882\n", + "Passed! Episode 30 Total Reward 59.700877\n", + "Passed! Episode 31 Total Reward 64.003951\n", + "Failed! Episode 32 Total Reward -43.807664\n", + "Passed! Episode 33 Total Reward 81.749323\n", + "Passed! Episode 34 Total Reward 66.484130\n", + "Failed! Episode 35 Total Reward -120.525069\n", + "Failed! Episode 36 Total Reward -24.451582\n", + "Failed! Episode 37 Total Reward -207.239527\n", + "Failed! Episode 38 Total Reward -43.106546\n", + "Passed! Episode 39 Total Reward 34.880121\n", + "Passed! Episode 40 Total Reward 66.188821\n", + "Passed! Episode 41 Total Reward 29.939728\n", + "Failed! Episode 42 Total Reward -86.516192\n", + "Failed! Episode 43 Total Reward -225.143733\n", + "Passed! Episode 44 Total Reward 7.568837\n", + "Passed! Episode 45 Total Reward 84.050707\n", + "Failed! Episode 46 Total Reward -86.705966\n", + "Passed! Episode 47 Total Reward 51.108435\n", + "Failed! Episode 48 Total Reward -88.531423\n", + "Passed! Episode 49 Total Reward 79.210735\n", + "Failed! Episode 50 Total Reward -64.247977\n", + "Failed! Episode 51 Total Reward -28.334167\n", + "Failed! Episode 52 Total Reward -53.942703\n", + "Passed! Episode 53 Total Reward 56.541394\n", + "Failed! Episode 54 Total Reward -30.573394\n", + "Failed! Episode 55 Total Reward -43.900060\n", + "Failed! Episode 56 Total Reward -31.793103\n", + "Passed! Episode 57 Total Reward 49.367744\n", + "Failed! Episode 58 Total Reward -59.277307\n", + "Failed! Episode 59 Total Reward -91.342287\n", + "Passed! Episode 60 Total Reward 72.884442\n", + "Passed! Episode 61 Total Reward 81.967582\n", + "Failed! Episode 62 Total Reward -69.682571\n", + "Failed! Episode 63 Total Reward -21.655243\n", + "Failed! Episode 64 Total Reward -102.427871\n", + "Failed! Episode 65 Total Reward -18.727795\n", + "Failed! Episode 66 Total Reward -106.180673\n", + "Passed! Episode 67 Total Reward 66.997363\n", + "Failed! Episode 68 Total Reward -98.338890\n", + "Failed! Episode 69 Total Reward -17.488416\n", + "Failed! Episode 70 Total Reward -126.062955\n", + "Passed! Episode 71 Total Reward 91.446696\n", + "Failed! Episode 72 Total Reward -153.244886\n", + "Passed! Episode 73 Total Reward 32.589752\n", + "Failed! Episode 74 Total Reward -106.034746\n", + "Failed! Episode 75 Total Reward -46.197340\n", + "Failed! Episode 76 Total Reward -64.983000\n", + "Failed! Episode 77 Total Reward -50.082150\n", + "Failed! Episode 78 Total Reward -44.506584\n", + "Failed! Episode 79 Total Reward -130.709335\n", + "Passed! Episode 80 Total Reward 65.267968\n", + "Failed! Episode 81 Total Reward -32.770421\n", + "Passed! Episode 82 Total Reward 68.083454\n", + "Failed! Episode 83 Total Reward -81.669658\n", + "Failed! Episode 84 Total Reward -40.234104\n", + "Passed! Episode 85 Total Reward 89.699236\n", + "Passed! Episode 86 Total Reward 76.233450\n", + "Failed! Episode 87 Total Reward -60.663254\n", + "Passed! Episode 88 Total Reward 75.066797\n", + "Failed! Episode 89 Total Reward -46.776051\n", + "Failed! Episode 90 Total Reward -25.376468\n", + "Passed! Episode 91 Total Reward 81.885317\n", + "Passed! Episode 92 Total Reward 61.040792\n", + "Failed! Episode 93 Total Reward -59.524774\n", + "Passed! Episode 94 Total Reward 80.259871\n", + "Passed! Episode 95 Total Reward 71.395997\n", + "Passed! Episode 96 Total Reward 70.046112\n", + "Passed! Episode 97 Total Reward 87.850163\n", + "Passed! Episode 98 Total Reward 86.638371\n", + "Failed! Episode 99 Total Reward -37.902022\n" + ] + } + ], + "source": [ + "steps_per_episodes=5000\n", + "ou = OrnsteinUhlenbeckProcess(theta=0.35,mu=0.8,sigma=0.4,n_steps_annealing=10)\n", + "max_total_reward=0\n", + "for episode in range(num_episodes):\n", + " state= env.reset()\n", + " state = state.reshape(-1,)\n", + " total_reward=0\n", + " ou.restart()\n", + " for step in range(steps_per_episodes):\n", + " action= actor.predict(state.reshape(1,-1))+ou.generate(episode)\n", + " next_state,reward,done,_ = env.step(action)\n", + " total_reward=total_reward+reward\n", + " #random minibatch from buffer\n", + " \n", + " batches=random.sample(memory,BATCH_SIZE)\n", + " states= np.array([batch[0].reshape((-1,)) for batch in batches])\n", + " actions= np.array([batch[1] for batch in batches])\n", + " actions=actions.reshape(-1,1)\n", + " rewards=np.array([batch[2] for batch in batches])\n", + " rewards = rewards.reshape((-1,1))\n", + " new_states=np.array([batch[3].reshape((-1,)) for batch in batches])\n", + " terminals=np.array([batch[4] for batch in batches])\n", + " terminals = terminals.reshape((-1,1))\n", + " #training\n", + " \n", + " target_actions = target_actor.predict(new_states)\n", + " target_Qs = target_critic.predict([new_states,target_actions])\n", + " \n", + " new_Qs = rewards+GAMMA*target_Qs*terminals\n", + " critic.fit([states,actions],new_Qs,verbose=False)\n", + " _,critic_values=actor_train(inputs=[states])\n", + " target_critic_weights=[TAU*weight+(1-TAU)*target_weight for weight,target_weight in zip(critic.get_weights(),target_critic.get_weights())]\n", + " target_actor_weights=[TAU*weight+(1-TAU)*target_weight for weight,target_weight in zip(actor.get_weights(),target_actor.get_weights())]\n", + " target_critic.set_weights(target_critic_weights)\n", + " target_actor.set_weights(target_actor_weights)\n", + " print(\"Total Reward %f\"%total_reward,end=\"\\r\")\n", + " if SHOW:\n", + " env.render()\n", + " if done or step==(steps_per_episodes-1):\n", + " \n", + " if total_reward<0:\n", + " print(\"Failed!\",end=\" \")\n", + " reward=-100\n", + " elif total_reward>0:\n", + " print(\"Passed!\",end=\" \")\n", + " reward=100\n", + " memory.append((state,action,reward,next_state,done))\n", + " break\n", + " \n", + " memory.append((state,action,reward,next_state,done))\n", + " state=next_state\n", + " if total_reward>max_total_reward:\n", + " actor.save_weights(\"MC_DDPG_Weights/Actor_Best_weights episode %d_GAMMA_%f_TAU%f_lr_%f.h5\"%(episode,GAMMA,TAU,actor_lr))\n", + " critic.save_weights(\"MC_DDPG_Weights/Critic_Best_weights episode %d_GAMMA_%f_TAU%f_lr_%f.h5\"%(episode,GAMMA,TAU,critic_lr))\n", + " max_total_reward=total_reward\n", + " print(\"Episode %d Total Reward %f\"%(episode,total_reward))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Video" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Please watch at 2x speed. I changed some simple mistakes after the video so the rewards are not exactly the same" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "[![](http://img.youtube.com/vi/9Fe_n-ovIaA/0.jpg)](http://www.youtube.com/watch?v=9Fe_n-ovIaA \"Keras tutorial DDPG\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 2", + "language": "python", + "name": "python2" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 2 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython2", + "version": "2.7.15" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/README.md b/README.md index 1e9de0b..f5c02d2 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,11 @@ # Reinforcement Learning Tutorials using Keras ### MD Muhaimin Rahman + sezan9[at]gmail[dot]com + + In this repository, I am trying to write tutorials on popular Reinforcement Learning Algorithms. I will design the models using Keras, as it is popular now and most importantly, it's very easy for beginners to learn. As there aren't enough Deep Learning Tutorials in Bangla, I am planning to write in both English and Bangla. As for now, I have selected the following ones for the topic of my tutorials: - Q Learning - Deep Q Network