From c1e008f4811aa8014158ebd3281bd48856c836e5 Mon Sep 17 00:00:00 2001 From: sjsmiths <69963888+sjsmiths@users.noreply.github.com> Date: Fri, 28 Aug 2020 10:34:48 +0200 Subject: [PATCH] Update DirectFeedbackAlignment.ipynb fix to Iris (plant) url --- DirectFeedbackAlignment.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DirectFeedbackAlignment.ipynb b/DirectFeedbackAlignment.ipynb index ad179e7..d3a88d1 100644 --- a/DirectFeedbackAlignment.ipynb +++ b/DirectFeedbackAlignment.ipynb @@ -1 +1 @@ -{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DirectFeedbackAlignment.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.2"}},"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"4FI8TOmPj9Bf"},"source":["# Direct Feedback Alignment\n","\n","Although backpropagation remains the most popular choice to train a neural network, other methods exist. In backpropagation, the error is propagated backwards from the output layer through all hidden layers to the input layer, and the weights defining the layers are changed according to the propagated error.\n","\n","[\"Direct Feedback Alignment\"](https://arxiv.org/abs/1609.01596), building on work by [Lillicrap et al.](https://www.nature.com/articles/ncomms13276), takes a different approach. Instead of using the weight matrix from the layer downstream of the current layer that needs to be trained, a random (but fixed) matrix is used.\n","\n","This means for a small network given by:\n","(Input Layer: x) $\\rightarrow W_0 \\rightarrow$ (Hidden Layer: h) $\\rightarrow W \\rightarrow$ (Output Layer: y)\n","\n","in backpropagation:\n","* $\\Delta W \\propto eh^T$\n","* $\\Delta W_0 \\propto - W^T ex^T$\n","\n","where $e$ is the error in the output layer $e = y - \\hat{y}$, and $h=W_0x$ is the output of the hidden layer (prior to the activation function).\n","\n","Instead, in Direct Feedback Alignment, a random (but fixed) matrix $B$ is used instead of $W$ in the second step and each hidden layer is trained using its own fixed random matrix. This way, the output error is used to directly update the weights of the hidden layers instead of propagating the error backwards.\n","\n","*Note*: \n","The following network is a direct copy of the notebook \"neural network from scratch\" in this repository, where the minimal change required to replace backpropagation with Direct Feedback Aligment was done."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"kDKBsH37j0L6","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"6d2319d5-1dd4-4d9d-f2ef-fef57f36fe45"},"source":["import matplotlib.pyplot as plt\n","import numpy as np\n","\n","#for logistic sigmoid function (a.ka. expit)\n","from scipy.special import expit\n","\n","# import of sigmoid and crossentropy\n","from scipy.special import expit\n","from sklearn.metrics import log_loss\n","\n","\n","import matplotlib.pyplot as plt\n","import numpy as np\n","from keras.datasets import mnist\n","from keras.utils.np_utils import to_categorical\n","np.random.seed(1234)\n","%matplotlib inline\n","import sys"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"hMKhoL9jkEme"},"source":["## Iris Dataset\n","This example will use the popular \"iris\" dataset.\n","\n","The [Iris dataset](http://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html) was [originally introduced](http://en.wikipedia.org/wiki/Iris_flower_data_set) by Sir Robert Fisher in 1936 as an example for discriminant analysis.\n","The data focus on how to discriminate between three different types of the [iris flower](http://en.wikipedia.org/wiki/Iris_(plant) ):\n","\n","* Setosa, \n","* Versicolour and\n","* Virginica\n","\n","Each row in the dataset contains the following features (measured in cm):\n","\n","* Sepal Length, \n","* Sepal Width, \n","* Petal Length and \n","* Petal Width.\n","\n","The labels (true values) are mapped as integers in $[0,1,2]$ for the three different flower types. We will transform this via one-hot encoding to make the computations regarding the loss function easier later.\n","\n","As this is a popular dataset, it is contained in various machine learning packages.\n","Here we use the data from the [SciKit-Learn](https://scikit-learn.org/stable/) machine learning suite.\n","\n","Note that the dataset is quite small with only 150 observations. However, it has the benefit that the subsequent processing and network training is very fast, and it serves us well to illustrate the principles."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"o3uI8HzokHq9","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"a04da136-fcdb-4872-f649-07496ed8bd58"},"source":["from sklearn.datasets import load_iris\n","iris = load_iris()\n","print('number of samples: {}'.format(len(iris.data)))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["number of samples: 150\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab_type":"code","id":"Iz6lZielkIxY","colab":{}},"source":["\n","# access the data and split into helper arrays.\n","x1 = iris.data\n","y1 = iris.target\n","\n","# The samples are ordered by target class # in the original dataset. \n","# As a first step, we shuffle the order before splitting the data into\n","# training and test data\n","perm = np.random.permutation(x1.shape[0])\n","y2 = y1[perm]\n","x2 = x1[perm,:]\n","\n","# now we take 90% of the data for trainig and 10% for testing\n","frac_train = 0.9\n","train_index = int(round(len(iris.data)*frac_train))\n","X_train = x2[0:train_index]\n","X_test = x2[train_index+1 :]\n","\n","y_train = y2[0:train_index]\n","y_test = y2[train_index+1 :]\n","\n","# convert the class number into one-hot encoding.\n","# e.g. target class 2 -> [0,0,1]\n","n_classes = 3\n","y_train = to_categorical(y_train, n_classes)\n","y_test = to_categorical(y_test, n_classes)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"tbxJDBzFkK4H"},"source":["## Forward and DFA Pass\n","\n","This is the core of the neural network training. During the forward\n","pass we use the current settings of the weights and calculate the network response. During the backward pass in backpropagation, we compare the output to the desired output and calculate the change of the weights.\n","\n","Note that we train the network such that during each weight update or learning step we present a batch of input signals to the network instead of just one.\n","For example, if we have four input variables, we do not just present $(x_1,x_2,x_3,x_4)$ to the network to learn from this pattern, but each of $x_1,x_2,x_3,$ and $x_4$ is a vector itself that containts $n$ training samples.\n","\n","### Forward Pass\n","We need to calculate the response of the network.\n","In our small network, we only need to compute the path from the input to the hidden layer and from the hidden to the output layer.\n","\n","* Input to hidden layer: \\\n","$a_1 = W_1 x + b_1$, $h_1 = f(a_1)$ where $f(.)$ is the activation function for the hidden layer. We will use the tanh(x) function here.\n","\n","* Hidden to output layer: \\\n","$a_2 = W_2 h_1 + b_2$, $\\hat{y} = f_y(a_2)$ where $f_y(.)$ is the activation function for the output layer. We will use the sigmoid function here because it is more convenient in the backward pass.\n","The sigmoid function is given by: $ \\sigma(x) = \\frac{1}{1+e^{-x}}$\n","\n","### DFA Pass\n","\n","The random matrix $B_1$ is used to update the weights in the hidden layer.\n","This is the only change to switch from backpropagation to Direct Feedback Alignment."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"SFnsSm9FkyuX","colab":{}},"source":["#\n","# Forward pass: calculate network response\n","# N.B. need to append as many components to the bias b1 and b2\n","# as we have elements in each training batch using np.tile \n","def forward_pass(W1, W2, b1, b2, x):\n"," #input to hidden layer\n"," a1 = np.matmul(W1,x) + np.tile(b1, x.shape[1]) \n"," h1 = np.tanh(a1)\n","\n"," #hidden to output layer\n"," a2 = np.matmul(W2, h1) + np.tile(b2, x.shape[1])\n","\n"," #output layer has sigmoid activation\n"," y_hat = expit(a2)\n","\n"," return a1, h1, a2, y_hat\n","\n","#\n","# DFA pass: calculate changes to weights\n","# using random matrix B1\n","#\n","def backward_pass_DFA(e, h1, B1, a1, x):\n"," dW2 = -np.matmul(e, np.transpose(h1)) \n"," da1 = np.matmul(B1, e)*(1-np.tanh(a1)**2) # <--- only change w.r.t.\n"," # backpropagation\n","\n"," dW1 = -np.matmul(da1, np.transpose(x)) \n","\n"," # change in bias b1 and b2:\n"," db2 = -np.sum(e, axis=1)\n"," db1 = -np.sum(da1, axis=1)\n","\n"," return dW1, dW2, db1[:, np.newaxis], db2[:,np.newaxis]\n","\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"8Oq-hHJFkmdX"},"source":["## Network training"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"rx2wmPeTkl30","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"9ca077af-69a4-47ce-bd7d-9ac55ad8846a"},"source":["#\n","# Network parameters\n","#\n","n_input = 4\n","n_hidden = 100\n","n_output = n_classes\n","\n","batch_size = 32\n","learn_rate = 1e-4\n","n_epochs = 100\n","\n","# we need to change the shape of the data from a long array \n","# with four variables per observation to an array with the four\n","# input variables, each with the observations (per variable)\n","# and accordingly for the output for the calculations of the weights.\n","x = np.transpose(X_train)\n","y = np.transpose(y_train)\n","\n","#\n","# initialize network weights\n","#\n","W1 = np.random.randn(n_hidden, n_input)\n","W2 = np.random.randn(n_output, n_hidden)\n","\n","b1 = np.random.randn(n_hidden, 1)\n","b2 = np.random.randn(n_output, 1)\n","\n","\n","# DFA matrix: random but fixed matrix\n","# the rest of this function is exactly the same as in the case of \n","# training with backpropagation\n","B1 = np.random.randn(n_hidden, n_output)\n","\n","dataset_size = x.shape[1]\n","n_batches = dataset_size//batch_size\n","print('Dataset size: {}, batch_size: {}, number of batches {}'.format(dataset_size, batch_size, n_batches))\n","\n","# save the total error for each epoch, summed over all batches\n","train_error_epoch = []\n","\n","\n","\n","for i in range(n_epochs):\n"," #shuffle training data\n"," perm = np.random.permutation(x.shape[1])\n"," x = x[:, perm]\n"," y = y[:, perm]\n","\n"," # value of loss function and training error\n"," loss = 0.0\n"," train_error = 0.0\n","\n"," #\n"," # loop over batches\n"," #\n"," for j in range(n_batches):\n"," train_data = x[:, j*batch_size:(j+1)*batch_size]\n"," targets = y[:, j*batch_size:(j+1)*batch_size]\n"," #\n"," # forward pass\n"," #\n"," a1, h1, a2, y_hat = forward_pass(W1=W1, W2=W2, b1=b1, b2=b2, x=train_data)\n","\n"," # metrics\n"," error = y_hat - targets\n","\n"," # due to one-hot encoding, each prediction and true label is an array\n"," # recover the class number via np.argmax\n"," preds = np.argmax(y_hat, axis=0) \n"," truth = np.argmax(targets, axis=0)\n","\n"," # error is the difference between prediction and true, summed over\n"," # all elements in the batch\n"," train_error += np.sum(preds!=truth)\n","\n"," # cross entropy loss\n"," loss_on_batch = log_loss(targets, y_hat)\n","\n"," #\n"," # calculate change of weights - backward pass\n"," #\n"," dW1, dW2, db1, db2 = backward_pass_DFA(e=error, h1=h1, B1=B1, a1=a1, x=train_data)\n","\n"," #weight update\n"," W1 += learn_rate * dW1\n"," W2 += learn_rate * dW2\n"," b1 += learn_rate * db1\n"," b2 += learn_rate * db2\n"," loss += loss_on_batch\n","\n"," #update errors per epoch\n"," training_error = 100.*train_error/x.shape[1]\n"," train_error_epoch.append(training_error)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Dataset size: 135, batch_size: 32, number of batches 4\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"xPWIdDdtk2Rn"},"source":["Visualize the training error per epoch"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"L1t2uPk3k3ho","colab":{"base_uri":"https://localhost:8080/","height":295},"outputId":"c82f802c-c4d9-473a-df21-0d4245b7e1b4"},"source":["x = range(0,n_epochs)\n","plt.plot(x, train_error_epoch, label='training error')\n","plt.title('Network training error')\n","plt.xlabel('Epochs')\n","plt.ylabel('Training error %')\n","#plt.legend(loc='best')\n","plt.show()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd5xU5dn/8c+1vbLsskvvRRGwUERArKgxUWOJsQRLjD22RE0eze/xMRoTk5jEqDGJJpoAGnvFWEHFXgARpUmR3pa2sMD26/fHnF0X2F1my+wwO9/36zWvndOvMweuuec+97lvc3dERCR+JEQ7ABERaV1K/CIicUaJX0Qkzijxi4jEGSV+EZE4o8QvIhJnlPilzTOzt83sklY83t/N7JaWXlekpSRFOwCJTWa2FMgA+rj79mDeJcB57n50GNv/G1jp7v8bwTAbLTivS9x9SlP34e5XRGJdkZaiEr80RyJwXbSDqI+FtOi/cTNrc4Wluj6nxp5nW/xc2jIlfmmOu4Abzax9XQvNbKCZvWFmm8xsgZmdFcy/DBgP/NzMis1sspldZGaTa2270MyeqjW9wswOCd6PMbNPzawo+Dum1npvm9mvzex9YAfQd7eYupjZbDP7WR3xTgJ6ApODuH5uZr3NzM3sYjNbDrwZrPuUma0NYnjHzAbX2s+/zeyO4P3RZrbSzG4ws/VmtsbMLmriuh2Cz2prcN53mNl79V0cMxtlZh+Y2RYz+9zMjm7ocwrO8yozWwgsDNa71MwWBdfwRTPrWmsfe6wvMcLd9dKr0S9gKXAc8CxwRzDvEuDt4H0msAK4iFCV4lBgAzAoWP7v6u2C6b7AFkKFka7AMkJVQdXLNgfL8oL35wf7PTeY7hCs+zawHBgcLE8O5l0C9AG+Ai7b23nVmu4NODAxOKf0YP6PgGwgFfgzMKvWNjXnBhwNVAC3B7F8h1CizW3Cuo8HrwxgUPD5vlfPeXQDNgb7SACOD6YLGvicHHgj+IzTgWODazYsOM/7gHdqHWOX9aP9b1Kv8F8q8Utz/R9wjZkV7Db/ZGCpu//L3Svc/TPgGeD7de3E3ZcA24BDgCOB14DVZjYQOAp4192rgJOAhe4+KdjvY8B84JRau/u3u88JlpcH8wYBbwG3uvuDTTjPX7r7dnffGcT7sLtvc/dS4JfAwWaWU8+25cDt7l7u7i8DxcD+jVnXzBKB7wXx73D3ucCEBuI9D3jZ3V929yp3fwOYTuiLoFpdn9Od7r4pOM/xwMPuPjM4z5uB0WbWu9Y+aq8vMUKJX5rF3b8EXgJu2m1RL+CwoJphi5ltIZRIOjewu2mESr1HBu/fJpT0jwqm4ZtfA7UtI1TCrbaijn2PB1YBTzd8RvWq2aeZJZrZb81ssZltJfQrASC/nm03untFrekdQFYj1y0gVDKvfW51nWe1XsD3d/v8xwJd9rJ97Xm7fNbuXkzoV8PePmvZxynxS0u4FbiUPRPCNHdvX+uV5e5XBsvr6ha2OvEfEbyfxp6JfzWhpFZbT0JJvVpd+/4loWqL/wSl5/rU111t7fk/AE4lVNWVQ6g6CMAa2G9zFRKqBupea16PBtZfAUza7fPPdPff1lqnrnOtPW+Xz9rMMoEO7P2zln2cEr80m7svAp4Arq01+yVgPzM738ySg9ehZnZAsHwdu914JZTcjyFUX7wSeBc4kVCy+SxY5+Vgvz8wsyQzO5tQNc5LewmznFA1UyYwsYHWPnXFtbtsoJRQ6TcD+M1e1m82d68kdD/ll2aWEVSBXdDAJo8Ap5jZt4JfKGnBzePuDWyzu8eAi8zsEDNLJXSeH7v70qaeh+wblPilpdxOKKkC4O7bgBOAcwiVHNcCvyN0kxDgIWBQUA3xfLDNV4TqtN8NprcCS4D3g8SHu28kdP/gBkKJ9+fAye6+YW8BunsZcAbQCXi4nuR/J/C/QVw31rOriYSqQFYBc4GP9nbsFnI1oV8Ya4FJhBJzaV0ruvsKQr9KfkHo18IK4Gc04v+8h55luIXQvZk1QD9C11NinLnrl5pILDKz3wGd3f3CaMcisUUlfpEYETwXcZCFjAQuBp6LdlwSe/S0nUjsyCZUvdOV0L2IPwIvRDUiiUmq6hERiTOq6hERiTMxUdWTn5/vvXv3jnYYIiIxZcaMGRvcffen6mMj8ffu3Zvp06dHOwwRkZhiZrs/5Q6oqkdEJO4o8YuIxBklfhGROKPELyISZ5T4RUTijBK/iEicUeIXEYkzMdGOvyXMWV3Ea1+urXNZ55x0zj60B4kJTR9Ho6KyiqdmrGRU3w70ya/pnRh35+Uv1jKgUxb7dcpu8v5FRFpKXCR+d+dnT81m7pqtWB253R1e+XIN954zlNzMlEbvf2NxKVf/5zM+XLKR7NQk7j77EI4b1ImdZZX84rkveO6zVXTPTef1nx5JRkpcfOQisg+Liyw0c/lm5q7Zyq9PH8L4w3YftQ+e+HQ5tzw/h1P+8h4PnD+cwV33HDO7orKKpMQ9a8Zmr9zCFZNmsGF7GbecPIjnP1vFJROnc/mRfXl34Qbmrd3K94d356kZK7lnykJu/s4BNduGRryHhL380nB3rK5vLBGRJoiLOv6JHy4jOzWJ0w7pVufysw/tyZNXjKayyvne3z7g+c++GVK0qsq5Z8pCBt36Gve/tYjavZk+NX0FZ/79Q8yMZ64Yw8Vj+/DUFaM5c3h3HnhnCSs37+DhCw/lru8fzNkjevDP975mzuoiAL7esJ1v3/Mu5z30MVVVdfeQurWknCsmzeD4u99BvaiKSEuJiW6ZR4wY4U3tq6dwWyljfjuV80b14tZTBu913av+M5NPvt7Ejw7vwzXH9udnT89myrx19CvIZHHhdr49pDN3nnEgf3z9KyZ9tIwx/Tpw37lD6ZCVWrMfd+fN+evZr1M2PfIyANiyo4zj/jSNru3TuebYAVz/5CzKK6soKa/iV6cN4fxRu/4SWbR+G5dNnMGSDdsBmHnL8eQ1oRpKROKXmc1w9xG7z2/zJf7HP1lOeaXvkVjrUpCdyqOXHMZFh/fm4fe/ZtSdU3l7wXp+ecogplx/FP970gG8Pncdh/1mKpM+WsZlR/Zl4o9G7pL0AcyMcQd0qkn6AO0zUrjl5EHMXlnEpROn0zMvgzd+ehSH9+/A71+Zz7qtJTXrvvrlGk79y/tsLSnn0iP6ALB6y84W+kREJN616cRfUVnFox8v54gB+fQtyAprm+TEBG49ZTB3n30wB3Rpx6OXHMYPD++DmXHJEX2ZdPFIBnZpx73nDuUX3zmgznr/+nz34K6cO7IH4w/ryTNXjqFHXga/Pu1AyiqruG3yHCqrnN+/Op8rHplJ/07ZTL5mLKcc3BVQ4heRltOmb+6+MXcda7eW8KvThjR629OHduf0od33mD+mXz4vXJXfpHjMjDvPOGiXeb3zM7l23ADuem0Byza+x5zVWznn0B7cdupgUpMSSUoIfbGsKSqpa5ciIo0W0RK/mbU3s6fNbL6ZzTOz0WaWZ2ZvmNnC4G9upI4/8cNldGufzrEDO0bqEC3i0iP6sn+nbL5at43fnH4gv/3eQaQmJQLQITOFlKQElfhFpMVEusR/D/Cqu59pZilABvALYKq7/9bMbgJuAv4nEgf/1WlDWFtU0qwHs1pDSlICj156GMUlFfSu9fAXhJp6dslJY7VK/CLSQiKW+M0sBzgS+CGAu5cBZWZ2KnB0sNoE4G0ilPj7d8yif8fw6vajLT8rlfzdbhJX65KTxhqV+EWkhUSyqqcPUAj8y8w+M7N/mlkm0Mnd1wTrrAU61bWxmV1mZtPNbHphYWEEw9z3dW2frqoeEWkxkUz8ScAw4G/uPhTYTqhap4aHHiKo80ECd3/Q3Ue4+4iCgj3GCo4rXXPSWbetlMp6HvQSEWmMSCb+lcBKd/84mH6a0BfBOjPrAhD8XR/BGNqELu3TqKxy1m9TPb+INF/EEr+7rwVWmNn+waxxwFzgReDCYN6FwAuRiqGt6No+HVBbfhFpGZFu1XMN8GjQomcJcBGhL5snzexiYBlwVoRjiHldc6oTfwnD9/4AsohIgyKa+N19FrBHPxGESv8Spi7t0wBYU6QSv4g0X5vusqGtaJeWTHZqEqu3qI5fRJpPiT9GdGmfpjp+EWkRSvwxoktOuvrrEZEWocQfI/QQl4i0FCX+GNE1J42N28soKa+MdigiEuOU+GNEdVv+taruEZFmUuKPEdVNOlXdIyLNpcQfI2oe4lKJX0SaSYk/RnTOCR7iUolfRJpJiT9GpCUnkp+Vwmo9vSsizaTEH0O65KTr6V0RaTYl/hjStX2a+usRkWZT4o8hKvGLSEtQ4o8hXdunUVxawbaS8miHIiIxTIk/huRmpACwebsSv4g0nRJ/DKlJ/DvKohyJiMQyJf4YkpuZDCjxi0jzKPHHkPZBiX/LDlX1iEjTKfHHEFX1iEhLUOKPITnpyZjBZpX4RaQZlPhjSGKC0S4tmS0q8YtIMyjxx5jcjGSV+EWkWZT4Y0z7jBSV+EWkWZT4Y0xuRrJa9YhIs0Q08ZvZUjP7wsxmmdn0YF6emb1hZguDv7mRjKGtyc1IUaseEWmW1ijxH+Puh7j7iGD6JmCquw8ApgbTEqZQVY9K/CLSdNGo6jkVmBC8nwCcFoUYYlZuRjLFpRWUVVRFOxQRiVGRTvwOvG5mM8zssmBeJ3dfE7xfC3Sqa0Mzu8zMppvZ9MLCwgiHGTvaZ4S6bdiyU9U9ItI0kU78Y919GPBt4CozO7L2Qnd3Ql8Oe3D3B919hLuPKCgoiHCYsUPdNohIc0U08bv7quDveuA5YCSwzsy6AAR/10cyhrbmm66ZVeIXkaaJWOI3s0wzy65+D5wAfAm8CFwYrHYh8EKkYmiLqqt69BCXiDRVUgT33Ql4zsyqj/Mfd3/VzD4FnjSzi4FlwFkRjKHNyc2srupRiV9EmiZiid/dlwAH1zF/IzAuUsdt63JV4heRZtKTuzEmPTmRlKQElfhFpMmU+GOMmQUdtSnxi0jTKPHHoFC3DarqEZGmUeKPQe0z1Ce/iDSdEn8MUolfRJpDiT8GqU9+EWkOJf4YVN0nf6jHCxGRxlHij0G5GSlUVDnbSiuiHYqIxCAl/hhU00PndtXzi0jjKfHHoJqO2lTPLyJNoMQfg3Izq7ttUOIXkcZT4o9B6pNfRJpDiT8GqapHRJoj7MRvZqPM7FUze9vMNE5uFLVLC3Wqqoe4RKQp6u2W2cw6u/vaWrOuB04HDPgYeD7CsUk9khITaJeWpIe4RKRJGuqP/+9mNhP4vbuXAFuAM4EqYGtrBCf1y81Utw0i0jT1VvW4+2nAZ8BLZnYB8BMgFegAqKonytRtg4g0VYN1/O4+GfgWkENosPSv3P1edy9sjeCkfuqTX0Saqt7Eb2bfNbO3gFcJDZJ+NnCqmT1uZv1aK0CpW25GippzikiTNFTHfwcwEkgHXnP3kcANZjYA+DVwTivEJ/VoH3TUJiLSWA0l/iLgDCADWF89090XoqQfdbkZKRSXVlBWUUVKkh7HEJHwNZQxTid0IzcJ+EHrhCPhyq3uqG2n6vlFpHHqLfG7+wbgvlaMRRohLzMVgI3FZXTMTotyNCISS1RHEKPys0LdNmwoLo1yJCISayKe+M0s0cw+M7OXguk+ZvaxmS0ysyfMLCXSMbRF+dmhEr8Sv4g0VoOJP0jabzXzGNcB82pN/w642937A5uBi5u5/7iUnxUk/m2q4xeRxtnbA1yVQJWZ5TRl52bWHTgJ+GcwbcCxwNPBKhPQU8BN0i4tiZTEBJX4RaTRGmrOWa0Y+MLM3gC2V89092vD2PbPwM+B7GC6A7DF3asHi10JdKtrQzO7DLgMoGfPnmEcKr6YGflZKRQq8YtII4WT+J8NXo1iZicD6919hpkd3djt3f1B4EGAESNGeGO3jwf52alsKFZVj4g0zl4Tv7tPCG7A7hfMWuDu4TwyejjwXTP7DpAGtAPuAdqbWVJQ6u8OrGpa6JKflcraopJohyEiMWavrXqC0vpC4H7gr8BXZnbk3rZz95vdvbu79yb0pO+b7j4eeItQ984AFwIvNC10yc9KUR2/iDRaOM05/wic4O5HufuRhHrrvLsZx/wf4HozW0Sozv+hZuwrruVnpbJxexlVVaoJE5HwhVPHn+zuC6on3P0rM0tuzEHc/W3g7eD9EkKdv0kz5WelUlnlbNlZTl6mHocQkfCEk/hnmNk/gUeC6fHA9MiFJOGq/RCXEr+IhCucqp4rgLnAtcFrLnBlJIOS8NR027BN9fwiEr4GS/xmlgh87u4DgT+1TkgSroLg6V215ReRxgjnyd0FZqYnqPZBNd02qC2/iDRCOHX8ucAcM/uEXZ/c/W7EopKw5KQnk5RgatIpIo0STuK/JeJRSJMkJBgdslJUxy8ijRJOHf8DQR2/7IPys1JV4heRRlEdf4wrUH89ItJIquOPcflZqSxYuy3aYYhIDFEdf4zLz0plY3EZ7k5ouAMRkYaF0zvnNDPrBQxw9ylmlgEkRj40CUd+VgpllVVs3VlBTkajetIQkTgVTu+clxIaMeuBYFY34PlIBiXhK8jWQ1wi0jjhdNlwFaG+9bcCuPtCoGMkg5LwffMQlxK/iIQnnMRf6u41zUbMLAlQP8D7CCV+EWmscBL/NDP7BZBuZscDTwGTIxuWhEsdtYlIY4WT+G8CCoEvgMuBl4H/jWRQEr7cjBQSE0xt+UUkbOG06qkC/hG8ZB+TkGDkZWoIRhEJXzglftnHqdsGEWkMJf42ID8rhUJV9YhImJT424CCrFTd3BWRsO21jt/MJrNn880iQuPuPuDuJZEITMKXnx2q6lG3DSISjnBK/EuAYr65wbsV2Absh2747hPys1IoraiiuLQi2qGISAwIp5O2Me5+aK3pyWb2qbsfamZzIhWYhK/2EIzZaeqvR0QaFk6JP6t2f/zB+6xgUncU9wGdc9IAWLV5Z5QjEZFYEE7ivwF4z8zeMrO3gXeBG80sE5hQ30ZmlmZmn5jZ52Y2x8xuC+b3MbOPzWyRmT1hZiktcSLxrH9B6Ht4cWFxlCMRkVgQzgNcL5vZAKB6+MUFtW7o/rmBTUuBY9292MySCX15vAJcD9zt7o+b2d+Bi4G/Nf0UpCA7lezUJCV+EQlLuM05hwODgYOBs8zsgr1t4CHVmSg5eDlwLKFuniH0i+G0RkUsezAz+nbMUuIXkbCE05xzEtAPmAVUBrMdmBjGtonADKA/cD+wGNji7tXNT1YS6t+/rm0vAy4D6NlTQ/7uTf+CLN5bVBjtMEQkBoTTqmcEMMjdG90VczBY+yFm1h54jm+qi8LZ9kHgQYARI0aoG+i96Ncxk2dmrmRbSbla9ohIg8Kp6vkS6Nycg7j7FuAtYDTQPujTH6A7sKo5+5aQfsEN3iWF26MciYjs68JJ/PnAXDN7zcxerH7tbSMzKwhK+phZOnA8MI/QF8CZwWoXAi80LXSprX/HUOJftF71/CLSsHCqen7ZxH13ASYE9fwJwJPu/pKZzQUeN7M7gM+Ah5q4f6mlZ14GSQmmG7wislfhNOec1pQdu/tsYGgd85cAI5uyT6lfcmICvTpkqMQvIntVb1WPmb0X/N1mZltrvbaZ2dbWC1HC1V9NOkUkDPWW+N19bPA3u/XCkeboV5DF1HnrKa+sIjlRPW6LSN3Cyg5mlmhmXc2sZ/Ur0oFJ4/UryKKiylm2cUe0QxGRfVg4D3BdA9wKrAOqgtkOHBTBuKQJqlv2LC4srnkvIrK7cFr1XAfs7+4bIx2MNE/fgkxAnbWJSMPCqepZQWjELdnHZacl06ldqlr2iEiDwinxLwHeNrP/EupxEwB3/1PEopIm61eQxWI9vSsiDQinxL8ceANIAbJrvWQf1L9jFkvWF9OErpVEJE6E8wDXba0RiLSMfgVZbCutYP22Ujq1S4t2OCKyD6o38ZvZn939J2Y2mVArnl24+3cjGpk0SXVnbYvWFyvxi0idGirxTwr+/qE1ApGWUd2Mc0lhMYf3z49yNCKyL2royd0Zwd8m9dUj0dGpXSqZKYlq2SMi9QrnAa4BwJ3AIKCm7sDd+0YwLmkiM6NfR7XsEZH6hdOq51+EBkOvAI4hNOTiI5EMSpqnf4E6axOR+oWT+NPdfSpg7r7M3X8JnBTZsKQ5+nXMYk1RCcWlFXtfWUTiTjiJv9TMEoCFZna1mZ0OqCOYfVi/oOuGJSr1i0gdwkn81wEZwLXAcOA8QkMmyj6qdmdtIiK7a/DmbjBs4tnufiNQDFzUKlFJs/TMyyQxwdSyR0Tq1NAIXEnuXgmMbcV4pAWkJCXQKy+DxevVskdE9tRQif8TYBjwmZm9CDwF1GQSd382wrFJM/TTMIwiUo9weudMAzYCxxLqusGCv0r8+7B+BVm8vWA9FZVVJGkYRhGppaHE39HMrge+5JuEX01dP+7j+hVkUl7pLN+0g74FaoQlIt9oKPEnEmq2aXUsU+Lfx/WradmzXYlfRHbRUOJf4+63t1ok0qKqe+lcXFjM8XSKcjQisi9pqPK3rpJ+2Mysh5m9ZWZzzWyOmV0XzM8zszfMbGHwN7c5x5G65aQnU5CtYRhFZE8NJf5xzdx3BXCDuw8CRgFXmdkg4CZgqrsPAKYG0xIB/Qoy1bJHRPZQb+J3903N2bG7r3H3mcH7bcA8oBtwKjAhWG0CcFpzjiP1698xi8UahlFEdtMq7fzMrDcwFPgY6OTua4JFa6HuCmgzu8zMppvZ9MLCwtYIs83pV5DF1pIKNhSXRTsUEdmHRDzxm1kW8AzwE3ffWnuZh4qidRZH3f1Bdx/h7iMKCgoiHWabVHsYRhGRahFN/GaWTCjpP1rrSd91ZtYlWN4FWB/JGOLZoK7tSDB4+yt9xCLyjYglfjMz4CFgnrv/qdaiF/mmd88LgRciFUO8y89K5YRBnXny0xWUlFdGOxwR2UdEssR/OHA+cKyZzQpe3wF+CxxvZguB44JpiZALRvdi845yXpq9Zu8ri0hcCKevniZx9/eo/1mA5jYVlTCN7teB/h2zmPjhUs4c3j3a4YjIPkC9d7VxZsYFo3sxe2URs1ZsAWBHWQX3TFnIwnXbohydiESDEn8cOGNYd7JSk5j4wVKWb9zBGX/9gLunfMXfpy2JdmgiEgURq+qRfUdWahJnDOvG45+sYOr8UAufgZ2z+WjJxihHJiLRoBJ/nLhgdC8qqqrokpPG5KvHcu7InqzaspMVm3ZEOzQRaWUq8ceJ/h2zef2nR9E9N5205ERG9+sAwIeLN9IjLyPK0YlIa1KJP47075hFWnIiAAM6ZtEhM0XVPSJxSIk/TpkZo/p24MMlG9WJm0icUeKPY6P65rGmqITlqucXiStK/HGsdj2/iMQPJf441q8gi/ysVNXzi8QZteqJY6F6/ryaev4NxWXc9Mxs5q/95one04Z25WffGrjLdv96/2umL9vMfecMJSGhWSN0ikgUqMQf50b368C6raU8P2sVp9z3Hu8v3sBhffIY3a8D+Vkp/OOdr9lQXFqzfkl5JfdMXch/Z6/h0U+WRzFyEWkqJf44N6pvqJ7/p098TnKS8cyVY/jT2Yfwh+8fzB/POoSyyiqe+HRFzfqTP1/Nlh3l9MzL4PevzGfd1pKaZWuKdvLKF2tapJXQlLnrdvnCEZGWo8Qf5/rmZzK4azuO2b+AyVePZXDXnJpl/TtmcXj/Djz60TIqKqtwdyZ+uIwBHbOY8KORlFZWcdvkOQB8sGgDJ937Hlc+OpMfPzqT4tKKJsc0f+1WLpk4nZ8+MUtNTUUiQIk/zpkZL10zln9dNJL2GSl7LL9gdG9WF5Uwdf56Zq3YwherirhgdC/65Gdy3bgBvPzFWq5/chbnPfQxeZkpXHNsf16bs5bT73+fJYVNG/Jx4ofLAHh34QZemLW6WecnInvSzV0hNFha3cYN7EjXnDQmfriUjtlpZKUmcfqwUL/+lx7RlxdmreLZmas4cXBn/nDWwWSlJjGqbweu/s9Mjv3jtDr3mZ6cyC++M5DzRvXa49hFO8t5buYqvjesO4sLi7n9pbkctV8BuZkpLFi7jav/M5M++Znc9f2DyUlPBmDd1hKueewzPvl6U81+BnbO5v7xw2rGHd7d0zNWcvOzsymv3PMXRUZKIi9efTj9O2Y3+LmJxCqLhZ/SI0aM8OnTp0c7jLh1/1uLuOu1BSQlGOMP68ltpw6pWbZs43ZmrdjCdw/uuksSX7VlJ8/NXFlnYp25fDPvLtzA94d351enDanpRgLg4fe+5vaX5vLSNWNJTDBOue89Th/ajaP2L+DnT88mLTmRrTvL6ZGXwQPnD6doZzlXPjKTHWUVXDC6N6lJCVS58+jHyymvqOLusw/huEGddjn+2qISjvvTNPoVZHL0/h13WebAA9MWc+bw7vz69ANb6BMUiQ4zm+HuI/aYr8Qve7OhuJQxd75JWWUVU64/iv4d6y5Fh6uqyvnz1IXcO3UhB3XP4W/nDadb+3Sqqpzj/jSNnIxknvvx4QD89pX5/H3aYgCG98rlr+OHsXzTjtB9hJIKyiurar4E9uv0TQl91ZadXDFpBl+sKuLacQP4ybgBNU1Pr5g0g7cWrOf1nx5Jrw6Ze8T3s6c+579frOGjX4yjXVpys85VJJrqS/yq45e9ys9K5aKxvfnesO7NTvoACQnG9cfvxz8uGMHXhds55b73+GDxBt5btIElG7Zz4ejeNeteN24AI3vn8cMxvXns0lF0apfGob3zeOmasQzr1Z4TBnfi+asO3yXpA3Rrn85TV4zmzOHduXfqQi6ZOJ2ineW8Pmctr85Zy3XHDagz6UPovsaOskqembGy2ecqsi9SiV+ianFhMZdPmsHXG7bTrX0620sr+ODmY0lNStz7xmFwdx75aBm3TZ5L99x0SsqraJ+RzORrxpKcWH+55/S/vk/RjnKmXH9UzS+FHWUVZKTseVtsZ1klackJe9yvKCmvJDkxgUQ95CZRohK/7JP6FWTx/FWHc/wBnVi+aQfnjuzZYkkfQjeuzx/dm8cuG8X2skrWbSvhN2cc2GDSh9DANUs2bOf9xRtwd/757hIO/OXr3PDk55SUV9as99/Zaxh+x4UY0qUAABDDSURBVBuc/9AnbN5eVjP/06WbOOL3b3HSve+ybOP2FjsfkZagEr/sE9yd9xdt5NA+uS2a+Gsr3FbK8k07GN4rd6/rllZUMubONxncLYec9GQmf76aA7vl8MWqIg7slsP9PxjGo58s44FpSxjYOZslhdspyE7lgfOHM3P5Zm6fPJeu7dMp2lmOu3PvuUP3uJEsEmm6uSvSSHe9Np/731qMGdx4wv78+Oh+TJm3nuufmMWO8koqq5zxh/Xk1lMGM3fNVq58ZAbrt5VSWeUcO7Ajd599CEU7yrls0nQWrNvGDcfvx4+P7q/+jaTVKPGLNNL6bSXc/MwXXDCmN0ftV1Azf0nwfMF3hnThrEN71MzfUFzKrS/MYWDnbK465psEv7Oskpuenc0Ls1ZzwqBO/PGsg8lWayFpBa2e+M3sYeBkYL27Dwnm5QFPAL2BpcBZ7r55b/tS4pdY5+489N7X3PnKfHp3yOCB80fs0UKqrKKKTdvL6JyTFqUopa2Jxs3dfwMn7jbvJmCquw8ApgbTIm2emXHJEX2ZdPFItuwo59KJ06mq2rXQdftLcxj7uzeZ8MFS9VEkERWxxO/u7wCbdpt9KjAheD8BOC1SxxfZF43pl8+t3x3M1xu2M21hYc38oh3lPD1jJRkpidz64hxueGrX1kMiLam1++rp5O5rgvdrgU71rWhmlwGXAfTs2bMVQhNpHScO7kxBdioTP1jKMUFLn6dmrKCkvIqnrxjDlHnr+POUhXy0eCMdslL32D47LYm//GAYeZl7dqonEo6oteP30G/Zen/PuvuD7j7C3UcUFBTUt5pIzElJSuDckT15+6tClm3cTlWVM+mjZYzolcuQbjn85Lj9+NcPD2VQ1xwKslN3eXXISuGDxRt5TIPgSDO0dol/nZl1cfc1ZtYFWN/KxxfZJ4w/rCd/fWsRj3y0jDH981m2cQc3nLB/zfJjBnbkmIF1t/sf/8+PePSjZVx+ZF+SggfRqp+DqB68JjHBGHdAxzqfNBZp7X8VLwIXAr8N/r7QyscX2Sd0apfGtwZ35snpK5mzeiv5WamcOLhzWNteMLo3l0+awZR56zlxSGibp2as5OdPz95lvauP6c+N39q/rl1InItYVY+ZPQZ8COxvZivN7GJCCf94M1sIHBdMi8SlC0b3omhnOR8s3sgPDutJSlJ4/x2rx0iY9NFSADYWl/Kbl+cxolcub914NG/deDRH7VfA458up7RCN4hlT5Fs1XOuu3dx92R37+7uD7n7Rncf5+4D3P04d9+91Y9I3BjZJ4/9O2WTmGD8YGT4DRiSEhMYP6oX7y/ayKL127jjv/PYXlrBnWccSJ/8TPrkZ/KjsX3YUFzGq1+ujeAZSKxSJ20iUWJm3Pm9A/nD9w9q9ENb5xzag5TEBH729Gye+2wVVx7VjwG1uqY+on8+ffIzmfDB0haOWtoCJX6RKBrWM5fTh3Zv9HYdslI5+aAufLZ8C33zM/nxMf13WZ6QYJw3qhczl2/hy1VFe93fzOWbmb92a6PjkNikxC8So340tg856cn85owDdxm+stqZw7uTnpzIxA+XNrifFZt2MP4fHzP+Hx+zZUdZg+tK26DELxKjhnTL4fNbT2BU3w51Ls9JT+a0od14YdZq1m8toaS8co+ngd2dW174EoAtO8u58+X5EY9bok+NfEXasAtG9+KxT5Yz8jdTa+adOLgzd33/ILLTknlp9hreXlDILScPYv22Eh6YtoTTh3Wr98tE2gZ1yyzSxr34+WpWbd4JhJp+/uuDpfTqkMFdZx7M5ZOm07V9Os/9+HDKKqo44c/TSE5M4JXrjojYgDjSetQfv4gA8NGSjVz16Ew2bi8jMcF44arDGdItB4BpXxVy4cOfcPmRfbnp2wP3GEdYYovG3BURAEb17cBL147lyP0KuOGE/WqSPsBR+xVw9ogePPDOEm548nN2lukBsLZIdfwicahLTjoTfzSyzmV3nnEg3XLTuXvKV8xfu43fnHEgWamhVFGQnUpOev2jh7k7RTvLaZ+xZ8+ha4p2sr204S+SBINeHTJJ1PCUEaXELyK7SEgwrh03gCHd2nHd47M47f73a5ZlpiTyx7MOqekjqLYdZRXc9MwXTJ69mp+M249rjg0NP1lRWcVdry/ggWlLwjr+iF65/PW8YXTM1khkkaI6fhGp1+otO5m+LDQ6qrvz8PtL+XzFFq46ph/XH79/Tcl82cbtXD5pBgvWbePQXnl8snQTxx3Qif87eRC/eO4L3lu0gXMO7cGY/vkNHm/91hL++PpXZKcl8bfzhjO8V27Ez3Ff5u7Nus+im7si0mylFZXc+sIcHv90BQd0aUfH7NBAMZ8t34yZcd+5QzliQD4TPljKHf+dR0WVk5KUwB2nDtllYPqGzFuzlcsnzWBN0U5G9e1AQpD4jt6/gB+O6V2TCKu/iN75qrDO/fTIS+fnJw6kXa2B7T9cvJFnZ67kp8fvR9f26TXzv1q3jb9PW8wlY/syqGu7vcZYXFrBH15bwME9cnZ58rq8soq/vLmIzNRELhnbl4SEb2J95OPlTJm7rs79dW2fxv+cOHCXKrLpSzdx5yvz+ecFI8ht4qA7Svwi0mKe+HQ5j3+6guphgwuyUvi/kwfTs0NGzTofL9nIg+8s4dpxAzi4R/tG7b9oRzm3vTSHxYXbAdhRWsHC9cWcdGAXfn/mQThww5OzeG3OOvbrlEX67uMOuDNn9VZ65mXwwPnD6d8xq2aw+8oqp0NmCvePH8aovh14+Ys13PjU5+woqyQtOYHffe8gTj2kW72xLS4s5vJJM1i0vhiA80f14paTB7G1pJyrHp3Jx1+H+p487oBO/Onsg0lOSOCmZ2fzwqzV9O+YRWbqnjXs81ZvpVNOKg+cN4IDumTzyEfLuG3yXLrnpvPwDw+lb0FWoz6/akr8IhKz3J1/vLuE374yn/4ds6hy+HrDdm7+9kAuHtunzuqQT77exI8fncHOskpG9M5j2leFnDi4M1ce3Y+fPjmLZRt3MG5gR16fu45hPdtz+6lDuH3yXD5ZuomLx/bh5m8PrBnoptobc9dx/ROzSE5K4J5zDuG9hRt44J0lDO3ZnrVFJWzeUcadZxzI1p0V/OqlufTIyyA1KYEF67Zx4wn78+Oj+9UZ62fLN3PlIzPZsrOMMf3yeXP+eo4d2JG7zz6kwZvpe6PELyIx772FG7jmsZmYGX/5wVDG9Gv4nsHaohKueGQGn6/cskvi3VZSzg1Pfs7rc9cx/rCe3HrKYFKSEiivrOLX/53Hvz9Yyqi+efzlB8PIz0qlqsr585SvuPfNRRzYLYe/nz+cbkFV0eTPV/Pzp2fTISuFB84fzuCuoeaxoS+emZRXVnHPOYdw9P51j6hWrXBbKVc9OpNPlm7i2nED+Mm4ATVVRU2lxC8ibcLm7WWYUWeT0bqUV1axbmsJ3XMzdplfVeWs2LyDXh0y99jm2ZkrufnZL8jLTOGuMw/m4fe/5s356zlzeHfuOG3IHp3ird9WQmZK0h7VOEU7y6mscvLCrKOvqKxiTVEJPfIy9r5yGJT4RUQa4ctVRVw+aQartuwkKcG49buDOe+wnjH1NHN9iV/t+EVE6jCkWw6TrxnLfW8u5KQDuzCid160Q2oxSvwiIvXIy0zh1lMGRzuMFqe+ekRE4owSv4hInFHiFxGJM0r8IiJxRolfRCTORCXxm9mJZrbAzBaZ2U3RiEFEJF61euI3s0TgfuDbwCDgXDMb1NpxiIjEq2iU+EcCi9x9ibuXAY8Dp0YhDhGRuBSNB7i6AStqTa8EDtt9JTO7DLgsmCw2swVNPF4+sKGJ28ayeDzveDxniM/z1jmHp1ddM/fZJ3fd/UHgwebux8ym19VXRVsXj+cdj+cM8XneOufmiUZVzyqg9lA83YN5IiLSCqKR+D8FBphZHzNLAc4BXoxCHCIicanVq3rcvcLMrgZeAxKBh919TgQP2ezqohgVj+cdj+cM8XneOudmiIn++EVEpOXoyV0RkTijxC8iEmfadOKPh64hzKyHmb1lZnPNbI6ZXRfMzzOzN8xsYfA3N9qxtjQzSzSzz8zspWC6j5l9HFzvJ4LGA22KmbU3s6fNbL6ZzTOz0W39WpvZT4N/21+a2WNmltYWr7WZPWxm683sy1rz6ry2FnJvcP6zzWxYY47VZhN/HHUNUQHc4O6DgFHAVcF53gRMdfcBwNRguq25DphXa/p3wN3u3h/YDFwclagi6x7gVXcfCBxM6Pzb7LU2s27AtcAIdx9CqEHIObTNa/1v4MTd5tV3bb8NDAhelwF/a8yB2mziJ066hnD3Ne4+M3i/jVAi6EboXCcEq00ATotOhJFhZt2Bk4B/BtMGHAs8HazSFs85BzgSeAjA3cvcfQtt/FoTan2YbmZJQAawhjZ4rd39HWDTbrPru7anAhM95COgvZl1CfdYbTnx19U1RLcoxdIqzKw3MBT4GOjk7muCRWuBTlEKK1L+DPwcqAqmOwBb3L0imG6L17sPUAj8K6ji+qeZZdKGr7W7rwL+ACwnlPCLgBm0/Wtdrb5r26z81pYTf1wxsyzgGeAn7r619jIPtdltM+12zexkYL27z4h2LK0sCRgG/M3dhwLb2a1apw1e61xCpds+QFcgkz2rQ+JCS17btpz446ZrCDNLJpT0H3X3Z4PZ66p/+gV/10crvgg4HPiumS0lVIV3LKG67/ZBdQC0zeu9Eljp7h8H008T+iJoy9f6OOBrdy9093LgWULXv61f62r1Xdtm5be2nPjjomuIoG77IWCeu/+p1qIXgQuD9xcCL7R2bJHi7je7e3d3703our7p7uOBt4Azg9Xa1DkDuPtaYIWZ7R/MGgfMpQ1fa0JVPKPMLCP4t159zm36WtdS37V9EbggaN0zCiiqVSW0d+7eZl/Ad4CvgMXA/4t2PBE6x7GEfv7NBmYFr+8QqvOeCiwEpgB50Y41Qud/NPBS8L4v8AmwCHgKSI12fBE430OA6cH1fh7IbevXGrgNmA98CUwCUtvitQYeI3Qfo5zQr7uL67u2gBFqtbgY+IJQq6ewj6UuG0RE4kxbruoREZE6KPGLiMQZJX4RkTijxC8iEmeU+EVE4owSv8QtM6s0s1m1Xi3WuZmZ9a7dy6LIvqTVh14U2YfsdPdDoh2ESGtTiV9kN2a21Mx+b2ZfmNknZtY/mN/bzN4M+j+famY9g/mdzOw5M/s8eI0JdpVoZv8I+pJ/3czSg/WvDcZPmG1mj0fpNCWOKfFLPEvfrarn7FrLitz9QOAvhHoCBbgPmODuBwGPAvcG8+8Fprn7wYT6zpkTzB8A3O/ug4EtwPeC+TcBQ4P9XBGpkxOpj57clbhlZsXunlXH/KXAse6+JOgAb627dzCzDUAXdy8P5q9x93wzKwS6u3tprX30Bt7w0AAamNn/AMnufoeZvQoUE+py4Xl3L47wqYrsQiV+kbp5Pe8bo7TW+0q+uad2EqF+VoYBn9bqZVKkVSjxi9Tt7Fp/Pwzef0CoN1CA8cC7wfupwJVQMw5wTn07NbMEoIe7vwX8D5AD7PGrQySSVNKQeJZuZrNqTb/q7tVNOnPNbDahUvu5wbxrCI1+9TNCI2FdFMy/DnjQzC4mVLK/klAvi3VJBB4JvhwMuNdDwyeKtBrV8YvsJqjjH+HuG6Idi0gkqKpHRCTOqMQvIhJnVOIXEYkzSvwiInFGiV9EJM4o8YuIxBklfhGROPP/AdC4IAoAH9wNAAAAAElFTkSuQmCC\n","text/plain":["
"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"96MfVaKNk5sG"},"source":["Use the independent test sample to test the network performance.\n","Note that the Iris dataset is very small, and we only use it here because the calculations are very fast and the data are already available within the libraries we use.\n","Feel free to experiment with different samples."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"6vdK-2MNk6tm","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"2df10700-67be-47da-eb0f-1acc60c1f4f2"},"source":["test_samples = np.transpose(X_test)\n","test_targets = np.transpose(y_test)\n"," \n","y_hats = forward_pass(W1, W2, b1, b2, test_samples)[-1]\n","preds = np.argmax(y_hats, axis=0) \n","truth = np.argmax(test_targets, axis=0)\n","test_error = 1.*np.sum(preds!=truth)/preds.shape[0]\n","\n","print('Test error:', test_error, '%')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Test error: 0.0 %\n"],"name":"stdout"}]}]} \ No newline at end of file +{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"DirectFeedbackAlignment.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.2"}},"cells":[{"cell_type":"markdown","metadata":{"colab_type":"text","id":"4FI8TOmPj9Bf"},"source":["# Direct Feedback Alignment\n","\n","Although backpropagation remains the most popular choice to train a neural network, other methods exist. In backpropagation, the error is propagated backwards from the output layer through all hidden layers to the input layer, and the weights defining the layers are changed according to the propagated error.\n","\n","[\"Direct Feedback Alignment\"](https://arxiv.org/abs/1609.01596), building on work by [Lillicrap et al.](https://www.nature.com/articles/ncomms13276), takes a different approach. Instead of using the weight matrix from the layer downstream of the current layer that needs to be trained, a random (but fixed) matrix is used.\n","\n","This means for a small network given by:\n","(Input Layer: x) $\\rightarrow W_0 \\rightarrow$ (Hidden Layer: h) $\\rightarrow W \\rightarrow$ (Output Layer: y)\n","\n","in backpropagation:\n","* $\\Delta W \\propto eh^T$\n","* $\\Delta W_0 \\propto - W^T ex^T$\n","\n","where $e$ is the error in the output layer $e = y - \\hat{y}$, and $h=W_0x$ is the output of the hidden layer (prior to the activation function).\n","\n","Instead, in Direct Feedback Alignment, a random (but fixed) matrix $B$ is used instead of $W$ in the second step and each hidden layer is trained using its own fixed random matrix. This way, the output error is used to directly update the weights of the hidden layers instead of propagating the error backwards.\n","\n","*Note*: \n","The following network is a direct copy of the notebook \"neural network from scratch\" in this repository, where the minimal change required to replace backpropagation with Direct Feedback Aligment was done."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"kDKBsH37j0L6","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"6d2319d5-1dd4-4d9d-f2ef-fef57f36fe45"},"source":["import matplotlib.pyplot as plt\n","import numpy as np\n","\n","#for logistic sigmoid function (a.ka. expit)\n","from scipy.special import expit\n","\n","# import of sigmoid and crossentropy\n","from scipy.special import expit\n","from sklearn.metrics import log_loss\n","\n","\n","import matplotlib.pyplot as plt\n","import numpy as np\n","from keras.datasets import mnist\n","from keras.utils.np_utils import to_categorical\n","np.random.seed(1234)\n","%matplotlib inline\n","import sys"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Using TensorFlow backend.\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"hMKhoL9jkEme"},"source":["## Iris Dataset\n","This example will use the popular \"iris\" dataset.\n","\n","The [Iris dataset](http://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html) was [originally introduced](http://en.wikipedia.org/wiki/Iris_flower_data_set) by Sir Robert Fisher in 1936 as an example for discriminant analysis.\n","The data focus on how to discriminate between three different types of the [iris flower](http://en.wikipedia.org/wiki/Iris_(plant)) :\n","\n","* Setosa, \n","* Versicolour and\n","* Virginica\n","\n","Each row in the dataset contains the following features (measured in cm):\n","\n","* Sepal Length, \n","* Sepal Width, \n","* Petal Length and \n","* Petal Width.\n","\n","The labels (true values) are mapped as integers in $[0,1,2]$ for the three different flower types. We will transform this via one-hot encoding to make the computations regarding the loss function easier later.\n","\n","As this is a popular dataset, it is contained in various machine learning packages.\n","Here we use the data from the [SciKit-Learn](https://scikit-learn.org/stable/) machine learning suite.\n","\n","Note that the dataset is quite small with only 150 observations. However, it has the benefit that the subsequent processing and network training is very fast, and it serves us well to illustrate the principles."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"o3uI8HzokHq9","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"a04da136-fcdb-4872-f649-07496ed8bd58"},"source":["from sklearn.datasets import load_iris\n","iris = load_iris()\n","print('number of samples: {}'.format(len(iris.data)))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["number of samples: 150\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab_type":"code","id":"Iz6lZielkIxY","colab":{}},"source":["\n","# access the data and split into helper arrays.\n","x1 = iris.data\n","y1 = iris.target\n","\n","# The samples are ordered by target class # in the original dataset. \n","# As a first step, we shuffle the order before splitting the data into\n","# training and test data\n","perm = np.random.permutation(x1.shape[0])\n","y2 = y1[perm]\n","x2 = x1[perm,:]\n","\n","# now we take 90% of the data for trainig and 10% for testing\n","frac_train = 0.9\n","train_index = int(round(len(iris.data)*frac_train))\n","X_train = x2[0:train_index]\n","X_test = x2[train_index+1 :]\n","\n","y_train = y2[0:train_index]\n","y_test = y2[train_index+1 :]\n","\n","# convert the class number into one-hot encoding.\n","# e.g. target class 2 -> [0,0,1]\n","n_classes = 3\n","y_train = to_categorical(y_train, n_classes)\n","y_test = to_categorical(y_test, n_classes)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"tbxJDBzFkK4H"},"source":["## Forward and DFA Pass\n","\n","This is the core of the neural network training. During the forward\n","pass we use the current settings of the weights and calculate the network response. During the backward pass in backpropagation, we compare the output to the desired output and calculate the change of the weights.\n","\n","Note that we train the network such that during each weight update or learning step we present a batch of input signals to the network instead of just one.\n","For example, if we have four input variables, we do not just present $(x_1,x_2,x_3,x_4)$ to the network to learn from this pattern, but each of $x_1,x_2,x_3,$ and $x_4$ is a vector itself that containts $n$ training samples.\n","\n","### Forward Pass\n","We need to calculate the response of the network.\n","In our small network, we only need to compute the path from the input to the hidden layer and from the hidden to the output layer.\n","\n","* Input to hidden layer: \\\n","$a_1 = W_1 x + b_1$, $h_1 = f(a_1)$ where $f(.)$ is the activation function for the hidden layer. We will use the tanh(x) function here.\n","\n","* Hidden to output layer: \\\n","$a_2 = W_2 h_1 + b_2$, $\\hat{y} = f_y(a_2)$ where $f_y(.)$ is the activation function for the output layer. We will use the sigmoid function here because it is more convenient in the backward pass.\n","The sigmoid function is given by: $ \\sigma(x) = \\frac{1}{1+e^{-x}}$\n","\n","### DFA Pass\n","\n","The random matrix $B_1$ is used to update the weights in the hidden layer.\n","This is the only change to switch from backpropagation to Direct Feedback Alignment."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"SFnsSm9FkyuX","colab":{}},"source":["#\n","# Forward pass: calculate network response\n","# N.B. need to append as many components to the bias b1 and b2\n","# as we have elements in each training batch using np.tile \n","def forward_pass(W1, W2, b1, b2, x):\n"," #input to hidden layer\n"," a1 = np.matmul(W1,x) + np.tile(b1, x.shape[1]) \n"," h1 = np.tanh(a1)\n","\n"," #hidden to output layer\n"," a2 = np.matmul(W2, h1) + np.tile(b2, x.shape[1])\n","\n"," #output layer has sigmoid activation\n"," y_hat = expit(a2)\n","\n"," return a1, h1, a2, y_hat\n","\n","#\n","# DFA pass: calculate changes to weights\n","# using random matrix B1\n","#\n","def backward_pass_DFA(e, h1, B1, a1, x):\n"," dW2 = -np.matmul(e, np.transpose(h1)) \n"," da1 = np.matmul(B1, e)*(1-np.tanh(a1)**2) # <--- only change w.r.t.\n"," # backpropagation\n","\n"," dW1 = -np.matmul(da1, np.transpose(x)) \n","\n"," # change in bias b1 and b2:\n"," db2 = -np.sum(e, axis=1)\n"," db1 = -np.sum(da1, axis=1)\n","\n"," return dW1, dW2, db1[:, np.newaxis], db2[:,np.newaxis]\n","\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"8Oq-hHJFkmdX"},"source":["## Network training"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"rx2wmPeTkl30","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"9ca077af-69a4-47ce-bd7d-9ac55ad8846a"},"source":["#\n","# Network parameters\n","#\n","n_input = 4\n","n_hidden = 100\n","n_output = n_classes\n","\n","batch_size = 32\n","learn_rate = 1e-4\n","n_epochs = 100\n","\n","# we need to change the shape of the data from a long array \n","# with four variables per observation to an array with the four\n","# input variables, each with the observations (per variable)\n","# and accordingly for the output for the calculations of the weights.\n","x = np.transpose(X_train)\n","y = np.transpose(y_train)\n","\n","#\n","# initialize network weights\n","#\n","W1 = np.random.randn(n_hidden, n_input)\n","W2 = np.random.randn(n_output, n_hidden)\n","\n","b1 = np.random.randn(n_hidden, 1)\n","b2 = np.random.randn(n_output, 1)\n","\n","\n","# DFA matrix: random but fixed matrix\n","# the rest of this function is exactly the same as in the case of \n","# training with backpropagation\n","B1 = np.random.randn(n_hidden, n_output)\n","\n","dataset_size = x.shape[1]\n","n_batches = dataset_size//batch_size\n","print('Dataset size: {}, batch_size: {}, number of batches {}'.format(dataset_size, batch_size, n_batches))\n","\n","# save the total error for each epoch, summed over all batches\n","train_error_epoch = []\n","\n","\n","\n","for i in range(n_epochs):\n"," #shuffle training data\n"," perm = np.random.permutation(x.shape[1])\n"," x = x[:, perm]\n"," y = y[:, perm]\n","\n"," # value of loss function and training error\n"," loss = 0.0\n"," train_error = 0.0\n","\n"," #\n"," # loop over batches\n"," #\n"," for j in range(n_batches):\n"," train_data = x[:, j*batch_size:(j+1)*batch_size]\n"," targets = y[:, j*batch_size:(j+1)*batch_size]\n"," #\n"," # forward pass\n"," #\n"," a1, h1, a2, y_hat = forward_pass(W1=W1, W2=W2, b1=b1, b2=b2, x=train_data)\n","\n"," # metrics\n"," error = y_hat - targets\n","\n"," # due to one-hot encoding, each prediction and true label is an array\n"," # recover the class number via np.argmax\n"," preds = np.argmax(y_hat, axis=0) \n"," truth = np.argmax(targets, axis=0)\n","\n"," # error is the difference between prediction and true, summed over\n"," # all elements in the batch\n"," train_error += np.sum(preds!=truth)\n","\n"," # cross entropy loss\n"," loss_on_batch = log_loss(targets, y_hat)\n","\n"," #\n"," # calculate change of weights - backward pass\n"," #\n"," dW1, dW2, db1, db2 = backward_pass_DFA(e=error, h1=h1, B1=B1, a1=a1, x=train_data)\n","\n"," #weight update\n"," W1 += learn_rate * dW1\n"," W2 += learn_rate * dW2\n"," b1 += learn_rate * db1\n"," b2 += learn_rate * db2\n"," loss += loss_on_batch\n","\n"," #update errors per epoch\n"," training_error = 100.*train_error/x.shape[1]\n"," train_error_epoch.append(training_error)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Dataset size: 135, batch_size: 32, number of batches 4\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"xPWIdDdtk2Rn"},"source":["Visualize the training error per epoch"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"L1t2uPk3k3ho","colab":{"base_uri":"https://localhost:8080/","height":295},"outputId":"c82f802c-c4d9-473a-df21-0d4245b7e1b4"},"source":["x = range(0,n_epochs)\n","plt.plot(x, train_error_epoch, label='training error')\n","plt.title('Network training error')\n","plt.xlabel('Epochs')\n","plt.ylabel('Training error %')\n","#plt.legend(loc='best')\n","plt.show()"],"execution_count":null,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd5xU5dn/8c+1vbLsskvvRRGwUERArKgxUWOJsQRLjD22RE0eze/xMRoTk5jEqDGJJpoAGnvFWEHFXgARpUmR3pa2sMD26/fHnF0X2F1my+wwO9/36zWvndOvMweuuec+97lvc3dERCR+JEQ7ABERaV1K/CIicUaJX0Qkzijxi4jEGSV+EZE4o8QvIhJnlPilzTOzt83sklY83t/N7JaWXlekpSRFOwCJTWa2FMgA+rj79mDeJcB57n50GNv/G1jp7v8bwTAbLTivS9x9SlP34e5XRGJdkZaiEr80RyJwXbSDqI+FtOi/cTNrc4Wluj6nxp5nW/xc2jIlfmmOu4Abzax9XQvNbKCZvWFmm8xsgZmdFcy/DBgP/NzMis1sspldZGaTa2270MyeqjW9wswOCd6PMbNPzawo+Dum1npvm9mvzex9YAfQd7eYupjZbDP7WR3xTgJ6ApODuH5uZr3NzM3sYjNbDrwZrPuUma0NYnjHzAbX2s+/zeyO4P3RZrbSzG4ws/VmtsbMLmriuh2Cz2prcN53mNl79V0cMxtlZh+Y2RYz+9zMjm7ocwrO8yozWwgsDNa71MwWBdfwRTPrWmsfe6wvMcLd9dKr0S9gKXAc8CxwRzDvEuDt4H0msAK4iFCV4lBgAzAoWP7v6u2C6b7AFkKFka7AMkJVQdXLNgfL8oL35wf7PTeY7hCs+zawHBgcLE8O5l0C9AG+Ai7b23nVmu4NODAxOKf0YP6PgGwgFfgzMKvWNjXnBhwNVAC3B7F8h1CizW3Cuo8HrwxgUPD5vlfPeXQDNgb7SACOD6YLGvicHHgj+IzTgWODazYsOM/7gHdqHWOX9aP9b1Kv8F8q8Utz/R9wjZkV7Db/ZGCpu//L3Svc/TPgGeD7de3E3ZcA24BDgCOB14DVZjYQOAp4192rgJOAhe4+KdjvY8B84JRau/u3u88JlpcH8wYBbwG3uvuDTTjPX7r7dnffGcT7sLtvc/dS4JfAwWaWU8+25cDt7l7u7i8DxcD+jVnXzBKB7wXx73D3ucCEBuI9D3jZ3V929yp3fwOYTuiLoFpdn9Od7r4pOM/xwMPuPjM4z5uB0WbWu9Y+aq8vMUKJX5rF3b8EXgJu2m1RL+CwoJphi5ltIZRIOjewu2mESr1HBu/fJpT0jwqm4ZtfA7UtI1TCrbaijn2PB1YBTzd8RvWq2aeZJZrZb81ssZltJfQrASC/nm03untFrekdQFYj1y0gVDKvfW51nWe1XsD3d/v8xwJd9rJ97Xm7fNbuXkzoV8PePmvZxynxS0u4FbiUPRPCNHdvX+uV5e5XBsvr6ha2OvEfEbyfxp6JfzWhpFZbT0JJvVpd+/4loWqL/wSl5/rU111t7fk/AE4lVNWVQ6g6CMAa2G9zFRKqBupea16PBtZfAUza7fPPdPff1lqnrnOtPW+Xz9rMMoEO7P2zln2cEr80m7svAp4Arq01+yVgPzM738ySg9ehZnZAsHwdu914JZTcjyFUX7wSeBc4kVCy+SxY5+Vgvz8wsyQzO5tQNc5LewmznFA1UyYwsYHWPnXFtbtsoJRQ6TcD+M1e1m82d68kdD/ll2aWEVSBXdDAJo8Ap5jZt4JfKGnBzePuDWyzu8eAi8zsEDNLJXSeH7v70qaeh+wblPilpdxOKKkC4O7bgBOAcwiVHNcCvyN0kxDgIWBQUA3xfLDNV4TqtN8NprcCS4D3g8SHu28kdP/gBkKJ9+fAye6+YW8BunsZcAbQCXi4nuR/J/C/QVw31rOriYSqQFYBc4GP9nbsFnI1oV8Ya4FJhBJzaV0ruvsKQr9KfkHo18IK4Gc04v+8h55luIXQvZk1QD9C11NinLnrl5pILDKz3wGd3f3CaMcisUUlfpEYETwXcZCFjAQuBp6LdlwSe/S0nUjsyCZUvdOV0L2IPwIvRDUiiUmq6hERiTOq6hERiTMxUdWTn5/vvXv3jnYYIiIxZcaMGRvcffen6mMj8ffu3Zvp06dHOwwRkZhiZrs/5Q6oqkdEJO4o8YuIxBklfhGROKPELyISZ5T4RUTijBK/iEicUeIXEYkzMdGOvyXMWV3Ea1+urXNZ55x0zj60B4kJTR9Ho6KyiqdmrGRU3w70ya/pnRh35+Uv1jKgUxb7dcpu8v5FRFpKXCR+d+dnT81m7pqtWB253R1e+XIN954zlNzMlEbvf2NxKVf/5zM+XLKR7NQk7j77EI4b1ImdZZX84rkveO6zVXTPTef1nx5JRkpcfOQisg+Liyw0c/lm5q7Zyq9PH8L4w3YftQ+e+HQ5tzw/h1P+8h4PnD+cwV33HDO7orKKpMQ9a8Zmr9zCFZNmsGF7GbecPIjnP1vFJROnc/mRfXl34Qbmrd3K94d356kZK7lnykJu/s4BNduGRryHhL380nB3rK5vLBGRJoiLOv6JHy4jOzWJ0w7pVufysw/tyZNXjKayyvne3z7g+c++GVK0qsq5Z8pCBt36Gve/tYjavZk+NX0FZ/79Q8yMZ64Yw8Vj+/DUFaM5c3h3HnhnCSs37+DhCw/lru8fzNkjevDP975mzuoiAL7esJ1v3/Mu5z30MVVVdfeQurWknCsmzeD4u99BvaiKSEuJiW6ZR4wY4U3tq6dwWyljfjuV80b14tZTBu913av+M5NPvt7Ejw7vwzXH9udnT89myrx19CvIZHHhdr49pDN3nnEgf3z9KyZ9tIwx/Tpw37lD6ZCVWrMfd+fN+evZr1M2PfIyANiyo4zj/jSNru3TuebYAVz/5CzKK6soKa/iV6cN4fxRu/4SWbR+G5dNnMGSDdsBmHnL8eQ1oRpKROKXmc1w9xG7z2/zJf7HP1lOeaXvkVjrUpCdyqOXHMZFh/fm4fe/ZtSdU3l7wXp+ecogplx/FP970gG8Pncdh/1mKpM+WsZlR/Zl4o9G7pL0AcyMcQd0qkn6AO0zUrjl5EHMXlnEpROn0zMvgzd+ehSH9+/A71+Zz7qtJTXrvvrlGk79y/tsLSnn0iP6ALB6y84W+kREJN616cRfUVnFox8v54gB+fQtyAprm+TEBG49ZTB3n30wB3Rpx6OXHMYPD++DmXHJEX2ZdPFIBnZpx73nDuUX3zmgznr/+nz34K6cO7IH4w/ryTNXjqFHXga/Pu1AyiqruG3yHCqrnN+/Op8rHplJ/07ZTL5mLKcc3BVQ4heRltOmb+6+MXcda7eW8KvThjR629OHduf0od33mD+mXz4vXJXfpHjMjDvPOGiXeb3zM7l23ADuem0Byza+x5zVWznn0B7cdupgUpMSSUoIfbGsKSqpa5ciIo0W0RK/mbU3s6fNbL6ZzTOz0WaWZ2ZvmNnC4G9upI4/8cNldGufzrEDO0bqEC3i0iP6sn+nbL5at43fnH4gv/3eQaQmJQLQITOFlKQElfhFpMVEusR/D/Cqu59pZilABvALYKq7/9bMbgJuAv4nEgf/1WlDWFtU0qwHs1pDSlICj156GMUlFfSu9fAXhJp6dslJY7VK/CLSQiKW+M0sBzgS+CGAu5cBZWZ2KnB0sNoE4G0ilPj7d8yif8fw6vajLT8rlfzdbhJX65KTxhqV+EWkhUSyqqcPUAj8y8w+M7N/mlkm0Mnd1wTrrAU61bWxmV1mZtPNbHphYWEEw9z3dW2frqoeEWkxkUz8ScAw4G/uPhTYTqhap4aHHiKo80ECd3/Q3Ue4+4iCgj3GCo4rXXPSWbetlMp6HvQSEWmMSCb+lcBKd/84mH6a0BfBOjPrAhD8XR/BGNqELu3TqKxy1m9TPb+INF/EEr+7rwVWmNn+waxxwFzgReDCYN6FwAuRiqGt6No+HVBbfhFpGZFu1XMN8GjQomcJcBGhL5snzexiYBlwVoRjiHldc6oTfwnD9/4AsohIgyKa+N19FrBHPxGESv8Spi7t0wBYU6QSv4g0X5vusqGtaJeWTHZqEqu3qI5fRJpPiT9GdGmfpjp+EWkRSvwxoktOuvrrEZEWocQfI/QQl4i0FCX+GNE1J42N28soKa+MdigiEuOU+GNEdVv+taruEZFmUuKPEdVNOlXdIyLNpcQfI2oe4lKJX0SaSYk/RnTOCR7iUolfRJpJiT9GpCUnkp+Vwmo9vSsizaTEH0O65KTr6V0RaTYl/hjStX2a+usRkWZT4o8hKvGLSEtQ4o8hXdunUVxawbaS8miHIiIxTIk/huRmpACwebsSv4g0nRJ/DKlJ/DvKohyJiMQyJf4YkpuZDCjxi0jzKPHHkPZBiX/LDlX1iEjTKfHHEFX1iEhLUOKPITnpyZjBZpX4RaQZlPhjSGKC0S4tmS0q8YtIMyjxx5jcjGSV+EWkWZT4Y0z7jBSV+EWkWZT4Y0xuRrJa9YhIs0Q08ZvZUjP7wsxmmdn0YF6emb1hZguDv7mRjKGtyc1IUaseEWmW1ijxH+Puh7j7iGD6JmCquw8ApgbTEqZQVY9K/CLSdNGo6jkVmBC8nwCcFoUYYlZuRjLFpRWUVVRFOxQRiVGRTvwOvG5mM8zssmBeJ3dfE7xfC3Sqa0Mzu8zMppvZ9MLCwgiHGTvaZ4S6bdiyU9U9ItI0kU78Y919GPBt4CozO7L2Qnd3Ql8Oe3D3B919hLuPKCgoiHCYsUPdNohIc0U08bv7quDveuA5YCSwzsy6AAR/10cyhrbmm66ZVeIXkaaJWOI3s0wzy65+D5wAfAm8CFwYrHYh8EKkYmiLqqt69BCXiDRVUgT33Ql4zsyqj/Mfd3/VzD4FnjSzi4FlwFkRjKHNyc2srupRiV9EmiZiid/dlwAH1zF/IzAuUsdt63JV4heRZtKTuzEmPTmRlKQElfhFpMmU+GOMmQUdtSnxi0jTKPHHoFC3DarqEZGmUeKPQe0z1Ce/iDSdEn8MUolfRJpDiT8GqU9+EWkOJf4YVN0nf6jHCxGRxlHij0G5GSlUVDnbSiuiHYqIxCAl/hhU00PndtXzi0jjKfHHoJqO2lTPLyJNoMQfg3Izq7ttUOIXkcZT4o9B6pNfRJpDiT8GqapHRJoj7MRvZqPM7FUze9vMNE5uFLVLC3Wqqoe4RKQp6u2W2cw6u/vaWrOuB04HDPgYeD7CsUk9khITaJeWpIe4RKRJGuqP/+9mNhP4vbuXAFuAM4EqYGtrBCf1y81Utw0i0jT1VvW4+2nAZ8BLZnYB8BMgFegAqKonytRtg4g0VYN1/O4+GfgWkENosPSv3P1edy9sjeCkfuqTX0Saqt7Eb2bfNbO3gFcJDZJ+NnCqmT1uZv1aK0CpW25GippzikiTNFTHfwcwEkgHXnP3kcANZjYA+DVwTivEJ/VoH3TUJiLSWA0l/iLgDCADWF89090XoqQfdbkZKRSXVlBWUUVKkh7HEJHwNZQxTid0IzcJ+EHrhCPhyq3uqG2n6vlFpHHqLfG7+wbgvlaMRRohLzMVgI3FZXTMTotyNCISS1RHEKPys0LdNmwoLo1yJCISayKe+M0s0cw+M7OXguk+ZvaxmS0ysyfMLCXSMbRF+dmhEr8Sv4g0VoOJP0jabzXzGNcB82pN/w642937A5uBi5u5/7iUnxUk/m2q4xeRxtnbA1yVQJWZ5TRl52bWHTgJ+GcwbcCxwNPBKhPQU8BN0i4tiZTEBJX4RaTRGmrOWa0Y+MLM3gC2V89092vD2PbPwM+B7GC6A7DF3asHi10JdKtrQzO7DLgMoGfPnmEcKr6YGflZKRQq8YtII4WT+J8NXo1iZicD6919hpkd3djt3f1B4EGAESNGeGO3jwf52alsKFZVj4g0zl4Tv7tPCG7A7hfMWuDu4TwyejjwXTP7DpAGtAPuAdqbWVJQ6u8OrGpa6JKflcraopJohyEiMWavrXqC0vpC4H7gr8BXZnbk3rZz95vdvbu79yb0pO+b7j4eeItQ984AFwIvNC10yc9KUR2/iDRaOM05/wic4O5HufuRhHrrvLsZx/wf4HozW0Sozv+hZuwrruVnpbJxexlVVaoJE5HwhVPHn+zuC6on3P0rM0tuzEHc/W3g7eD9EkKdv0kz5WelUlnlbNlZTl6mHocQkfCEk/hnmNk/gUeC6fHA9MiFJOGq/RCXEr+IhCucqp4rgLnAtcFrLnBlJIOS8NR027BN9fwiEr4GS/xmlgh87u4DgT+1TkgSroLg6V215ReRxgjnyd0FZqYnqPZBNd02qC2/iDRCOHX8ucAcM/uEXZ/c/W7EopKw5KQnk5RgatIpIo0STuK/JeJRSJMkJBgdslJUxy8ijRJOHf8DQR2/7IPys1JV4heRRlEdf4wrUH89ItJIquOPcflZqSxYuy3aYYhIDFEdf4zLz0plY3EZ7k5ouAMRkYaF0zvnNDPrBQxw9ylmlgEkRj40CUd+VgpllVVs3VlBTkajetIQkTgVTu+clxIaMeuBYFY34PlIBiXhK8jWQ1wi0jjhdNlwFaG+9bcCuPtCoGMkg5LwffMQlxK/iIQnnMRf6u41zUbMLAlQP8D7CCV+EWmscBL/NDP7BZBuZscDTwGTIxuWhEsdtYlIY4WT+G8CCoEvgMuBl4H/jWRQEr7cjBQSE0xt+UUkbOG06qkC/hG8ZB+TkGDkZWoIRhEJXzglftnHqdsGEWkMJf42ID8rhUJV9YhImJT424CCrFTd3BWRsO21jt/MJrNn880iQuPuPuDuJZEITMKXnx2q6lG3DSISjnBK/EuAYr65wbsV2Absh2747hPys1IoraiiuLQi2qGISAwIp5O2Me5+aK3pyWb2qbsfamZzIhWYhK/2EIzZaeqvR0QaFk6JP6t2f/zB+6xgUncU9wGdc9IAWLV5Z5QjEZFYEE7ivwF4z8zeMrO3gXeBG80sE5hQ30ZmlmZmn5jZ52Y2x8xuC+b3MbOPzWyRmT1hZiktcSLxrH9B6Ht4cWFxlCMRkVgQzgNcL5vZAKB6+MUFtW7o/rmBTUuBY9292MySCX15vAJcD9zt7o+b2d+Bi4G/Nf0UpCA7lezUJCV+EQlLuM05hwODgYOBs8zsgr1t4CHVmSg5eDlwLKFuniH0i+G0RkUsezAz+nbMUuIXkbCE05xzEtAPmAVUBrMdmBjGtonADKA/cD+wGNji7tXNT1YS6t+/rm0vAy4D6NlTQ/7uTf+CLN5bVBjtMEQkBoTTqmcEMMjdG90VczBY+yFm1h54jm+qi8LZ9kHgQYARI0aoG+i96Ncxk2dmrmRbSbla9ohIg8Kp6vkS6Nycg7j7FuAtYDTQPujTH6A7sKo5+5aQfsEN3iWF26MciYjs68JJ/PnAXDN7zcxerH7tbSMzKwhK+phZOnA8MI/QF8CZwWoXAi80LXSprX/HUOJftF71/CLSsHCqen7ZxH13ASYE9fwJwJPu/pKZzQUeN7M7gM+Ah5q4f6mlZ14GSQmmG7wislfhNOec1pQdu/tsYGgd85cAI5uyT6lfcmICvTpkqMQvIntVb1WPmb0X/N1mZltrvbaZ2dbWC1HC1V9NOkUkDPWW+N19bPA3u/XCkeboV5DF1HnrKa+sIjlRPW6LSN3Cyg5mlmhmXc2sZ/Ur0oFJ4/UryKKiylm2cUe0QxGRfVg4D3BdA9wKrAOqgtkOHBTBuKQJqlv2LC4srnkvIrK7cFr1XAfs7+4bIx2MNE/fgkxAnbWJSMPCqepZQWjELdnHZacl06ldqlr2iEiDwinxLwHeNrP/EupxEwB3/1PEopIm61eQxWI9vSsiDQinxL8ceANIAbJrvWQf1L9jFkvWF9OErpVEJE6E8wDXba0RiLSMfgVZbCutYP22Ujq1S4t2OCKyD6o38ZvZn939J2Y2mVArnl24+3cjGpk0SXVnbYvWFyvxi0idGirxTwr+/qE1ApGWUd2Mc0lhMYf3z49yNCKyL2royd0Zwd8m9dUj0dGpXSqZKYlq2SMi9QrnAa4BwJ3AIKCm7sDd+0YwLmkiM6NfR7XsEZH6hdOq51+EBkOvAI4hNOTiI5EMSpqnf4E6axOR+oWT+NPdfSpg7r7M3X8JnBTZsKQ5+nXMYk1RCcWlFXtfWUTiTjiJv9TMEoCFZna1mZ0OqCOYfVi/oOuGJSr1i0gdwkn81wEZwLXAcOA8QkMmyj6qdmdtIiK7a/DmbjBs4tnufiNQDFzUKlFJs/TMyyQxwdSyR0Tq1NAIXEnuXgmMbcV4pAWkJCXQKy+DxevVskdE9tRQif8TYBjwmZm9CDwF1GQSd382wrFJM/TTMIwiUo9weudMAzYCxxLqusGCv0r8+7B+BVm8vWA9FZVVJGkYRhGppaHE39HMrge+5JuEX01dP+7j+hVkUl7pLN+0g74FaoQlIt9oKPEnEmq2aXUsU+Lfx/WradmzXYlfRHbRUOJf4+63t1ok0qKqe+lcXFjM8XSKcjQisi9pqPK3rpJ+2Mysh5m9ZWZzzWyOmV0XzM8zszfMbGHwN7c5x5G65aQnU5CtYRhFZE8NJf5xzdx3BXCDuw8CRgFXmdkg4CZgqrsPAKYG0xIB/Qoy1bJHRPZQb+J3903N2bG7r3H3mcH7bcA8oBtwKjAhWG0CcFpzjiP1698xi8UahlFEdtMq7fzMrDcwFPgY6OTua4JFa6HuCmgzu8zMppvZ9MLCwtYIs83pV5DF1pIKNhSXRTsUEdmHRDzxm1kW8AzwE3ffWnuZh4qidRZH3f1Bdx/h7iMKCgoiHWabVHsYRhGRahFN/GaWTCjpP1rrSd91ZtYlWN4FWB/JGOLZoK7tSDB4+yt9xCLyjYglfjMz4CFgnrv/qdaiF/mmd88LgRciFUO8y89K5YRBnXny0xWUlFdGOxwR2UdEssR/OHA+cKyZzQpe3wF+CxxvZguB44JpiZALRvdi845yXpq9Zu8ri0hcCKevniZx9/eo/1mA5jYVlTCN7teB/h2zmPjhUs4c3j3a4YjIPkC9d7VxZsYFo3sxe2URs1ZsAWBHWQX3TFnIwnXbohydiESDEn8cOGNYd7JSk5j4wVKWb9zBGX/9gLunfMXfpy2JdmgiEgURq+qRfUdWahJnDOvG45+sYOr8UAufgZ2z+WjJxihHJiLRoBJ/nLhgdC8qqqrokpPG5KvHcu7InqzaspMVm3ZEOzQRaWUq8ceJ/h2zef2nR9E9N5205ERG9+sAwIeLN9IjLyPK0YlIa1KJP47075hFWnIiAAM6ZtEhM0XVPSJxSIk/TpkZo/p24MMlG9WJm0icUeKPY6P65rGmqITlqucXiStK/HGsdj2/iMQPJf441q8gi/ysVNXzi8QZteqJY6F6/ryaev4NxWXc9Mxs5q/95one04Z25WffGrjLdv96/2umL9vMfecMJSGhWSN0ikgUqMQf50b368C6raU8P2sVp9z3Hu8v3sBhffIY3a8D+Vkp/OOdr9lQXFqzfkl5JfdMXch/Z6/h0U+WRzFyEWkqJf44N6pvqJ7/p098TnKS8cyVY/jT2Yfwh+8fzB/POoSyyiqe+HRFzfqTP1/Nlh3l9MzL4PevzGfd1pKaZWuKdvLKF2tapJXQlLnrdvnCEZGWo8Qf5/rmZzK4azuO2b+AyVePZXDXnJpl/TtmcXj/Djz60TIqKqtwdyZ+uIwBHbOY8KORlFZWcdvkOQB8sGgDJ937Hlc+OpMfPzqT4tKKJsc0f+1WLpk4nZ8+MUtNTUUiQIk/zpkZL10zln9dNJL2GSl7LL9gdG9WF5Uwdf56Zq3YwherirhgdC/65Gdy3bgBvPzFWq5/chbnPfQxeZkpXHNsf16bs5bT73+fJYVNG/Jx4ofLAHh34QZemLW6WecnInvSzV0hNFha3cYN7EjXnDQmfriUjtlpZKUmcfqwUL/+lx7RlxdmreLZmas4cXBn/nDWwWSlJjGqbweu/s9Mjv3jtDr3mZ6cyC++M5DzRvXa49hFO8t5buYqvjesO4sLi7n9pbkctV8BuZkpLFi7jav/M5M++Znc9f2DyUlPBmDd1hKueewzPvl6U81+BnbO5v7xw2rGHd7d0zNWcvOzsymv3PMXRUZKIi9efTj9O2Y3+LmJxCqLhZ/SI0aM8OnTp0c7jLh1/1uLuOu1BSQlGOMP68ltpw6pWbZs43ZmrdjCdw/uuksSX7VlJ8/NXFlnYp25fDPvLtzA94d351enDanpRgLg4fe+5vaX5vLSNWNJTDBOue89Th/ajaP2L+DnT88mLTmRrTvL6ZGXwQPnD6doZzlXPjKTHWUVXDC6N6lJCVS58+jHyymvqOLusw/huEGddjn+2qISjvvTNPoVZHL0/h13WebAA9MWc+bw7vz69ANb6BMUiQ4zm+HuI/aYr8Qve7OhuJQxd75JWWUVU64/iv4d6y5Fh6uqyvnz1IXcO3UhB3XP4W/nDadb+3Sqqpzj/jSNnIxknvvx4QD89pX5/H3aYgCG98rlr+OHsXzTjtB9hJIKyiurar4E9uv0TQl91ZadXDFpBl+sKuLacQP4ybgBNU1Pr5g0g7cWrOf1nx5Jrw6Ze8T3s6c+579frOGjX4yjXVpys85VJJrqS/yq45e9ys9K5aKxvfnesO7NTvoACQnG9cfvxz8uGMHXhds55b73+GDxBt5btIElG7Zz4ejeNeteN24AI3vn8cMxvXns0lF0apfGob3zeOmasQzr1Z4TBnfi+asO3yXpA3Rrn85TV4zmzOHduXfqQi6ZOJ2ineW8Pmctr85Zy3XHDagz6UPovsaOskqembGy2ecqsi9SiV+ianFhMZdPmsHXG7bTrX0620sr+ODmY0lNStz7xmFwdx75aBm3TZ5L99x0SsqraJ+RzORrxpKcWH+55/S/vk/RjnKmXH9UzS+FHWUVZKTseVtsZ1klackJe9yvKCmvJDkxgUQ95CZRohK/7JP6FWTx/FWHc/wBnVi+aQfnjuzZYkkfQjeuzx/dm8cuG8X2skrWbSvhN2cc2GDSh9DANUs2bOf9xRtwd/757hIO/OXr3PDk55SUV9as99/Zaxh+x4UY0qUAABDDSURBVBuc/9AnbN5eVjP/06WbOOL3b3HSve+ybOP2FjsfkZagEr/sE9yd9xdt5NA+uS2a+Gsr3FbK8k07GN4rd6/rllZUMubONxncLYec9GQmf76aA7vl8MWqIg7slsP9PxjGo58s44FpSxjYOZslhdspyE7lgfOHM3P5Zm6fPJeu7dMp2lmOu3PvuUP3uJEsEmm6uSvSSHe9Np/731qMGdx4wv78+Oh+TJm3nuufmMWO8koqq5zxh/Xk1lMGM3fNVq58ZAbrt5VSWeUcO7Ajd599CEU7yrls0nQWrNvGDcfvx4+P7q/+jaTVKPGLNNL6bSXc/MwXXDCmN0ftV1Azf0nwfMF3hnThrEN71MzfUFzKrS/MYWDnbK465psEv7Oskpuenc0Ls1ZzwqBO/PGsg8lWayFpBa2e+M3sYeBkYL27Dwnm5QFPAL2BpcBZ7r55b/tS4pdY5+489N7X3PnKfHp3yOCB80fs0UKqrKKKTdvL6JyTFqUopa2Jxs3dfwMn7jbvJmCquw8ApgbTIm2emXHJEX2ZdPFItuwo59KJ06mq2rXQdftLcxj7uzeZ8MFS9VEkERWxxO/u7wCbdpt9KjAheD8BOC1SxxfZF43pl8+t3x3M1xu2M21hYc38oh3lPD1jJRkpidz64hxueGrX1kMiLam1++rp5O5rgvdrgU71rWhmlwGXAfTs2bMVQhNpHScO7kxBdioTP1jKMUFLn6dmrKCkvIqnrxjDlHnr+POUhXy0eCMdslL32D47LYm//GAYeZl7dqonEo6oteP30G/Zen/PuvuD7j7C3UcUFBTUt5pIzElJSuDckT15+6tClm3cTlWVM+mjZYzolcuQbjn85Lj9+NcPD2VQ1xwKslN3eXXISuGDxRt5TIPgSDO0dol/nZl1cfc1ZtYFWN/KxxfZJ4w/rCd/fWsRj3y0jDH981m2cQc3nLB/zfJjBnbkmIF1t/sf/8+PePSjZVx+ZF+SggfRqp+DqB68JjHBGHdAxzqfNBZp7X8VLwIXAr8N/r7QyscX2Sd0apfGtwZ35snpK5mzeiv5WamcOLhzWNteMLo3l0+awZR56zlxSGibp2as5OdPz95lvauP6c+N39q/rl1InItYVY+ZPQZ8COxvZivN7GJCCf94M1sIHBdMi8SlC0b3omhnOR8s3sgPDutJSlJ4/x2rx0iY9NFSADYWl/Kbl+cxolcub914NG/deDRH7VfA458up7RCN4hlT5Fs1XOuu3dx92R37+7uD7n7Rncf5+4D3P04d9+91Y9I3BjZJ4/9O2WTmGD8YGT4DRiSEhMYP6oX7y/ayKL127jjv/PYXlrBnWccSJ/8TPrkZ/KjsX3YUFzGq1+ujeAZSKxSJ20iUWJm3Pm9A/nD9w9q9ENb5xzag5TEBH729Gye+2wVVx7VjwG1uqY+on8+ffIzmfDB0haOWtoCJX6RKBrWM5fTh3Zv9HYdslI5+aAufLZ8C33zM/nxMf13WZ6QYJw3qhczl2/hy1VFe93fzOWbmb92a6PjkNikxC8So340tg856cn85owDdxm+stqZw7uTnpzIxA+XNrifFZt2MP4fHzP+Hx+zZUdZg+tK26DELxKjhnTL4fNbT2BU3w51Ls9JT+a0od14YdZq1m8toaS8co+ngd2dW174EoAtO8u58+X5EY9bok+NfEXasAtG9+KxT5Yz8jdTa+adOLgzd33/ILLTknlp9hreXlDILScPYv22Eh6YtoTTh3Wr98tE2gZ1yyzSxr34+WpWbd4JhJp+/uuDpfTqkMFdZx7M5ZOm07V9Os/9+HDKKqo44c/TSE5M4JXrjojYgDjSetQfv4gA8NGSjVz16Ew2bi8jMcF44arDGdItB4BpXxVy4cOfcPmRfbnp2wP3GEdYYovG3BURAEb17cBL147lyP0KuOGE/WqSPsBR+xVw9ogePPDOEm548nN2lukBsLZIdfwicahLTjoTfzSyzmV3nnEg3XLTuXvKV8xfu43fnHEgWamhVFGQnUpOev2jh7k7RTvLaZ+xZ8+ha4p2sr204S+SBINeHTJJ1PCUEaXELyK7SEgwrh03gCHd2nHd47M47f73a5ZlpiTyx7MOqekjqLYdZRXc9MwXTJ69mp+M249rjg0NP1lRWcVdry/ggWlLwjr+iF65/PW8YXTM1khkkaI6fhGp1+otO5m+LDQ6qrvz8PtL+XzFFq46ph/XH79/Tcl82cbtXD5pBgvWbePQXnl8snQTxx3Qif87eRC/eO4L3lu0gXMO7cGY/vkNHm/91hL++PpXZKcl8bfzhjO8V27Ez3Ff5u7Nus+im7si0mylFZXc+sIcHv90BQd0aUfH7NBAMZ8t34yZcd+5QzliQD4TPljKHf+dR0WVk5KUwB2nDtllYPqGzFuzlcsnzWBN0U5G9e1AQpD4jt6/gB+O6V2TCKu/iN75qrDO/fTIS+fnJw6kXa2B7T9cvJFnZ67kp8fvR9f26TXzv1q3jb9PW8wlY/syqGu7vcZYXFrBH15bwME9cnZ58rq8soq/vLmIzNRELhnbl4SEb2J95OPlTJm7rs79dW2fxv+cOHCXKrLpSzdx5yvz+ecFI8ht4qA7Svwi0mKe+HQ5j3+6guphgwuyUvi/kwfTs0NGzTofL9nIg+8s4dpxAzi4R/tG7b9oRzm3vTSHxYXbAdhRWsHC9cWcdGAXfn/mQThww5OzeG3OOvbrlEX67uMOuDNn9VZ65mXwwPnD6d8xq2aw+8oqp0NmCvePH8aovh14+Ys13PjU5+woqyQtOYHffe8gTj2kW72xLS4s5vJJM1i0vhiA80f14paTB7G1pJyrHp3Jx1+H+p487oBO/Onsg0lOSOCmZ2fzwqzV9O+YRWbqnjXs81ZvpVNOKg+cN4IDumTzyEfLuG3yXLrnpvPwDw+lb0FWoz6/akr8IhKz3J1/vLuE374yn/4ds6hy+HrDdm7+9kAuHtunzuqQT77exI8fncHOskpG9M5j2leFnDi4M1ce3Y+fPjmLZRt3MG5gR16fu45hPdtz+6lDuH3yXD5ZuomLx/bh5m8PrBnoptobc9dx/ROzSE5K4J5zDuG9hRt44J0lDO3ZnrVFJWzeUcadZxzI1p0V/OqlufTIyyA1KYEF67Zx4wn78+Oj+9UZ62fLN3PlIzPZsrOMMf3yeXP+eo4d2JG7zz6kwZvpe6PELyIx772FG7jmsZmYGX/5wVDG9Gv4nsHaohKueGQGn6/cskvi3VZSzg1Pfs7rc9cx/rCe3HrKYFKSEiivrOLX/53Hvz9Yyqi+efzlB8PIz0qlqsr585SvuPfNRRzYLYe/nz+cbkFV0eTPV/Pzp2fTISuFB84fzuCuoeaxoS+emZRXVnHPOYdw9P51j6hWrXBbKVc9OpNPlm7i2nED+Mm4ATVVRU2lxC8ibcLm7WWYUWeT0bqUV1axbmsJ3XMzdplfVeWs2LyDXh0y99jm2ZkrufnZL8jLTOGuMw/m4fe/5s356zlzeHfuOG3IHp3ird9WQmZK0h7VOEU7y6mscvLCrKOvqKxiTVEJPfIy9r5yGJT4RUQa4ctVRVw+aQartuwkKcG49buDOe+wnjH1NHN9iV/t+EVE6jCkWw6TrxnLfW8u5KQDuzCid160Q2oxSvwiIvXIy0zh1lMGRzuMFqe+ekRE4owSv4hInFHiFxGJM0r8IiJxRolfRCTORCXxm9mJZrbAzBaZ2U3RiEFEJF61euI3s0TgfuDbwCDgXDMb1NpxiIjEq2iU+EcCi9x9ibuXAY8Dp0YhDhGRuBSNB7i6AStqTa8EDtt9JTO7DLgsmCw2swVNPF4+sKGJ28ayeDzveDxniM/z1jmHp1ddM/fZJ3fd/UHgwebux8ym19VXRVsXj+cdj+cM8XneOufmiUZVzyqg9lA83YN5IiLSCqKR+D8FBphZHzNLAc4BXoxCHCIicanVq3rcvcLMrgZeAxKBh919TgQP2ezqohgVj+cdj+cM8XneOudmiIn++EVEpOXoyV0RkTijxC8iEmfadOKPh64hzKyHmb1lZnPNbI6ZXRfMzzOzN8xsYfA3N9qxtjQzSzSzz8zspWC6j5l9HFzvJ4LGA22KmbU3s6fNbL6ZzTOz0W39WpvZT4N/21+a2WNmltYWr7WZPWxm683sy1rz6ry2FnJvcP6zzWxYY47VZhN/HHUNUQHc4O6DgFHAVcF53gRMdfcBwNRguq25DphXa/p3wN3u3h/YDFwclagi6x7gVXcfCBxM6Pzb7LU2s27AtcAIdx9CqEHIObTNa/1v4MTd5tV3bb8NDAhelwF/a8yB2mziJ066hnD3Ne4+M3i/jVAi6EboXCcEq00ATotOhJFhZt2Bk4B/BtMGHAs8HazSFs85BzgSeAjA3cvcfQtt/FoTan2YbmZJQAawhjZ4rd39HWDTbrPru7anAhM95COgvZl1CfdYbTnx19U1RLcoxdIqzKw3MBT4GOjk7muCRWuBTlEKK1L+DPwcqAqmOwBb3L0imG6L17sPUAj8K6ji+qeZZdKGr7W7rwL+ACwnlPCLgBm0/Wtdrb5r26z81pYTf1wxsyzgGeAn7r619jIPtdltM+12zexkYL27z4h2LK0sCRgG/M3dhwLb2a1apw1e61xCpds+QFcgkz2rQ+JCS17btpz446ZrCDNLJpT0H3X3Z4PZ66p/+gV/10crvgg4HPiumS0lVIV3LKG67/ZBdQC0zeu9Eljp7h8H008T+iJoy9f6OOBrdy9093LgWULXv61f62r1Xdtm5be2nPjjomuIoG77IWCeu/+p1qIXgQuD9xcCL7R2bJHi7je7e3d3703our7p7uOBt4Azg9Xa1DkDuPtaYIWZ7R/MGgfMpQ1fa0JVPKPMLCP4t159zm36WtdS37V9EbggaN0zCiiqVSW0d+7eZl/Ad4CvgMXA/4t2PBE6x7GEfv7NBmYFr+8QqvOeCiwEpgB50Y41Qud/NPBS8L4v8AmwCHgKSI12fBE430OA6cH1fh7IbevXGrgNmA98CUwCUtvitQYeI3Qfo5zQr7uL67u2gBFqtbgY+IJQq6ewj6UuG0RE4kxbruoREZE6KPGLiMQZJX4RkTijxC8iEmeU+EVE4owSv8QtM6s0s1m1Xi3WuZmZ9a7dy6LIvqTVh14U2YfsdPdDoh2ESGtTiV9kN2a21Mx+b2ZfmNknZtY/mN/bzN4M+j+famY9g/mdzOw5M/s8eI0JdpVoZv8I+pJ/3czSg/WvDcZPmG1mj0fpNCWOKfFLPEvfrarn7FrLitz9QOAvhHoCBbgPmODuBwGPAvcG8+8Fprn7wYT6zpkTzB8A3O/ug4EtwPeC+TcBQ4P9XBGpkxOpj57clbhlZsXunlXH/KXAse6+JOgAb627dzCzDUAXdy8P5q9x93wzKwS6u3tprX30Bt7w0AAamNn/AMnufoeZvQoUE+py4Xl3L47wqYrsQiV+kbp5Pe8bo7TW+0q+uad2EqF+VoYBn9bqZVKkVSjxi9Tt7Fp/Pwzef0CoN1CA8cC7wfupwJVQMw5wTn07NbMEoIe7vwX8D5AD7PGrQySSVNKQeJZuZrNqTb/q7tVNOnPNbDahUvu5wbxrCI1+9TNCI2FdFMy/DnjQzC4mVLK/klAvi3VJBB4JvhwMuNdDwyeKtBrV8YvsJqjjH+HuG6Idi0gkqKpHRCTOqMQvIhJnVOIXEYkzSvwiInFGiV9EJM4o8YuIxBklfhGROPP/AdC4IAoAH9wNAAAAAElFTkSuQmCC\n","text/plain":["
"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"colab_type":"text","id":"96MfVaKNk5sG"},"source":["Use the independent test sample to test the network performance.\n","Note that the Iris dataset is very small, and we only use it here because the calculations are very fast and the data are already available within the libraries we use.\n","Feel free to experiment with different samples."]},{"cell_type":"code","metadata":{"colab_type":"code","id":"6vdK-2MNk6tm","colab":{"base_uri":"https://localhost:8080/","height":34},"outputId":"2df10700-67be-47da-eb0f-1acc60c1f4f2"},"source":["test_samples = np.transpose(X_test)\n","test_targets = np.transpose(y_test)\n"," \n","y_hats = forward_pass(W1, W2, b1, b2, test_samples)[-1]\n","preds = np.argmax(y_hats, axis=0) \n","truth = np.argmax(test_targets, axis=0)\n","test_error = 1.*np.sum(preds!=truth)/preds.shape[0]\n","\n","print('Test error:', test_error, '%')"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Test error: 0.0 %\n"],"name":"stdout"}]}]}