{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "description: \"Logging... logging everywhere! \\U0001F52E\"\n",
    "---\n",
    "\n",
    "# Loggers\n",
    "\n",
    "Welcome to the _\"Logging\"_ tutorial of the _\"From Zero to Hero\"_ series. In this part we will present the functionalities offered by the _Avalanche_ `logging` module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/ContinualAI/avalanche.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 📑 The Logging Module\n",
    "\n",
    "In the previous tutorial we have learned how to evaluate a continual learning algorithm in _Avalanche_, through different metrics that can be used _off-the-shelf_ via the _Evaluation Plugin_ or stand-alone. However, computing metrics and collecting results, may not be enough at times.\n",
    "\n",
    "While running complex experiments with long _waiting times_, **logging** results over-time is fundamental to \"_babysit_\" your experiments in real-time, or even understand what went wrong in the aftermath.\n",
    "\n",
    "This is why in Avalanche we decided to put a strong emphasis on logging and **provide a number of loggers** that can be used with any set of metrics!\n",
    "\n",
    "### Loggers\n",
    "\n",
    "_Avalanche_ at the moment supports three main Loggers:\n",
    "\n",
    "* **InteractiveLogger**: This logger provides a nice progress bar and displays real-time metrics results in an interactive way \\(meant for `stdout`\\).\n",
    "* **TextLogger**: This logger, mostly intended for file logging, is the plain text version of the `InteractiveLogger`. Keep in mind that it may be very verbose.\n",
    "* **TensorboardLogger**: It logs all the metrics on [Tensorboard](https://www.tensorflow.org/tensorboard) in real-time. Perfect for real-time plotting.\n",
    "\n",
    "In order to keep track of when each metric value has been logged, we leverage a `global counter`. You can see the `global counter` reported in the x axis of the logged plots.\n",
    "\n",
    "The `global counter` is an ever-increasing value which starts from 0 and it is increased by one each time a training or evaluation iteration is performed (i.e. after each training or evaluation minibatch).\n",
    "The `global counter` is updated automatically by the strategy. It should be reset by creating a new instance of the strategy.\n",
    "\n",
    "#### How to use Them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting experiment...\n",
      "Start of experience:  0\n",
      "Current Classes:  [0, 9]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 0 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:14<00:00,  1.61it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720915.6689\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720915.3457\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.0546\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.2314\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0334\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 14.4948\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.7566\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9839\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.26it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 369.0000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720916.5859\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 0.1841\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.9693\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.39it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 364.3000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720917.2822\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 4.4039\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 16.62it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 372.0000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720917.9785\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 4.9915\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.13it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 358.5000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720918.6748\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.7982\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 17.01it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 369.6000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720919.3711\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 4.1429\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[ 950,    0,    0,    0,    0,    0,    0,    0,    0,   30],\n",
      "        [   6,    0,    0,    0,    0,    0,    0,    0,    0, 1129],\n",
      "        [ 510,    0,    0,    0,    0,    0,    0,    0,    0,  522],\n",
      "        [ 397,    0,    0,    0,    0,    0,    0,    0,    0,  613],\n",
      "        [  38,    0,    0,    0,    0,    0,    0,    0,    0,  944],\n",
      "        [ 416,    0,    0,    0,    0,    0,    0,    0,    0,  476],\n",
      "        [ 448,    0,    0,    0,    0,    0,    0,    0,    0,  510],\n",
      "        [  35,    0,    0,    0,    0,    0,    0,    0,    0,  993],\n",
      "        [ 154,    0,    0,    0,    0,    0,    0,    0,    0,  820],\n",
      "        [  31,    0,    0,    0,    0,    0,    0,    0,    0,  978]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream = 720920.0059\n",
      "\tLoss_Stream/eval_phase/test_stream = 3.6963\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.1928\n",
      "Start of experience:  1\n",
      "Current Classes:  [2, 7]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 1 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:15<00:00,  1.66it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720948.1162\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720947.7930\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.8814\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.3365\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0324\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 14.7142\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.4811\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9596\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.26it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 356.7000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720951.2979\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6347\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0850\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.54it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 357.1000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720952.0020\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2836\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9578\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 16.20it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 360.1000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720952.7061\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 4.8760\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.78it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 377.6000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720953.4102\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.9460\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 17.54it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 374.7000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720954.1143\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 4.0850\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[169,   0, 697,   0,   0,   0,   0, 114,   0,   0],\n",
      "        [  0,   0, 549,   0,   0,   0,   0, 586,   0,   0],\n",
      "        [  0,   0, 991,   0,   0,   0,   0,  41,   0,   0],\n",
      "        [  0,   0, 762,   0,   0,   0,   0, 248,   0,   0],\n",
      "        [  0,   0, 147,   0,   0,   0,   0, 835,   0,   0],\n",
      "        [  0,   0, 537,   0,   0,   0,   0, 355,   0,   0],\n",
      "        [  0,   0, 928,   0,   0,   0,   0,  30,   0,   0],\n",
      "        [  0,   0,  46,   0,   0,   0,   0, 982,   0,   0],\n",
      "        [  0,   0, 730,   0,   0,   0,   0, 244,   0,   0],\n",
      "        [  0,   0,  51,   0,   0,   0,   0, 958,   0,   0]])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tDiskUsage_Stream/eval_phase/test_stream = 720954.7568\n",
      "\tLoss_Stream/eval_phase/test_stream = 3.3306\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2142\n",
      "Start of experience:  2\n",
      "Current Classes:  [5, 6]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 2 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 23/23 [00:14<00:00,  1.58it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720963.3389\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720963.0156\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.4399\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.7766\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0335\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 14.1599\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.3270\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.8555\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.69it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 382.1000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720964.2637\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6700\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0327\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 17.14it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 363.2000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720964.9678\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 1.4572\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.4282\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 15.69it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 352.9000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720965.6719\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 0.6560\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.9032\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.79it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 363.2000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720966.3760\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 3.7727\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 16.65it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 344.9000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720967.0801\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.3084\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[  65,    0,    0,    0,    0,  296,  617,    2,    0,    0],\n",
      "        [   0,    0,    7,    0,    0, 1028,  100,    0,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,  126,  902,    4,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,  855,  142,   13,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,  154,  683,  145,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,  720,  166,    6,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    6,  951,    1,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,   73,   73,  882,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,  598,  368,    8,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,  335,  305,  369,    0,    0]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream = 720967.7227\n",
      "\tLoss_Stream/eval_phase/test_stream = 2.4015\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2618\n",
      "Start of experience:  3\n",
      "Current Classes:  [8, 3]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 3 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:14<00:00,  1.69it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720976.7256\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720976.4023\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2646\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.9822\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0326\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 13.7838\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2737\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.7925\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.09it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 365.2000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720977.6504\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6221\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0417\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 15.61it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 328.5000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720978.3545\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.1221\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2772\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 15.27it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 347.9000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720979.0586\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 1.5153\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.4162\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.56it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 354.7000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720979.7627\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 0.8728\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.8261\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 16.50it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 343.2000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720980.4668\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.2265\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[ 83,   0,   0, 611,   0,   5, 162,   0, 119,   0],\n",
      "        [  0,   0,   0, 989,   0,   0,   3,   0, 143,   0],\n",
      "        [  0,   0,   0, 488,   0,   0, 175,   0, 369,   0],\n",
      "        [  0,   0,   0, 977,   0,   0,   1,   2,  30,   0],\n",
      "        [  0,   0,   0, 418,   0,   0, 199,   2, 363,   0],\n",
      "        [  0,   0,   0, 651,   0,   6,   6,   1, 228,   0],\n",
      "        [  0,   0,   0, 131,   0,   0, 764,   0,  63,   0],\n",
      "        [  0,   0,   0, 236,   0,   6,   4, 571, 211,   0],\n",
      "        [  0,   0,   0, 310,   0,   0,   2,   0, 662,   0],\n",
      "        [  0,   0,   0, 558,   0,   1,  57,  39, 354,   0]])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tDiskUsage_Stream/eval_phase/test_stream = 720981.1094\n",
      "\tLoss_Stream/eval_phase/test_stream = 2.0952\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.3063\n",
      "Start of experience:  4\n",
      "Current Classes:  [1, 4]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 4 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 26/26 [00:15<00:00,  1.68it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720990.6611\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720990.3379\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2713\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 1.1053\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0316\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 15.0845\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2687\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9048\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.42it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 372.1000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720991.5859\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6000\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0392\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 17.46it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 382.3000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720992.2900\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.3497\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2112\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 16.65it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 381.0000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720992.9941\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 2.1057\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.2589\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.85it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 383.6000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720993.6982\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 1.2233\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.6532\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 17.54it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 381.4000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720994.4023\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9423\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9622\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[  78,    7,    0,  556,   55,    1,   65,    0,  218,    0],\n",
      "        [   0, 1133,    0,    2,    0,    0,    0,    0,    0,    0],\n",
      "        [   0,  330,    0,  232,  122,    0,   33,    0,  315,    0],\n",
      "        [   0,  117,    0,  848,   17,    0,    0,    1,   27,    0],\n",
      "        [   0,   68,    0,    2,  904,    0,    6,    0,    2,    0],\n",
      "        [   0,  260,    0,  448,   73,    0,    0,    0,  111,    0],\n",
      "        [   0,  146,    0,   57,  190,    0,  479,    0,   86,    0],\n",
      "        [   0,  201,    0,   26,  325,    0,    0,  435,   41,    0],\n",
      "        [   0,  353,    0,  117,   56,    0,    0,    0,  448,    0],\n",
      "        [   0,  124,    0,   20,  846,    0,    4,    0,   15,    0]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream = 720995.0449\n",
      "\tLoss_Stream/eval_phase/test_stream = 1.8329\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.4325\n"
     ]
    }
   ],
   "source": [
    "from torch.optim import SGD\n",
    "from torch.nn import CrossEntropyLoss\n",
    "from avalanche.benchmarks.classic import SplitMNIST\n",
    "from avalanche.evaluation.metrics import forgetting_metrics, \\\n",
    "accuracy_metrics, loss_metrics, timing_metrics, cpu_usage_metrics, \\\n",
    "confusion_matrix_metrics, disk_usage_metrics\n",
    "from avalanche.models import SimpleMLP\n",
    "from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger\n",
    "from avalanche.training.plugins import EvaluationPlugin\n",
    "from avalanche.training.strategies import Naive\n",
    "\n",
    "scenario = SplitMNIST(n_experiences=5)\n",
    "\n",
    "# MODEL CREATION\n",
    "model = SimpleMLP(num_classes=scenario.n_classes)\n",
    "\n",
    "# DEFINE THE EVALUATION PLUGIN and LOGGERS\n",
    "# The evaluation plugin manages the metrics computation.\n",
    "# It takes as argument a list of metrics, collectes their results and returns\n",
    "# them to the strategy it is attached to.\n",
    "\n",
    "# log to Tensorboard\n",
    "tb_logger = TensorboardLogger()\n",
    "\n",
    "# log to text file\n",
    "text_logger = TextLogger(open('log.txt', 'a'))\n",
    "\n",
    "# print to stdout\n",
    "interactive_logger = InteractiveLogger()\n",
    "\n",
    "eval_plugin = EvaluationPlugin(\n",
    "    accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    timing_metrics(epoch=True, epoch_running=True),\n",
    "    cpu_usage_metrics(experience=True),\n",
    "    forgetting_metrics(experience=True, stream=True),\n",
    "    confusion_matrix_metrics(num_classes=scenario.n_classes, save_image=False,\n",
    "                             stream=True),\n",
    "    disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loggers=[interactive_logger, text_logger, tb_logger]\n",
    ")\n",
    "\n",
    "# CREATE THE STRATEGY INSTANCE (NAIVE)\n",
    "cl_strategy = Naive(\n",
    "    model, SGD(model.parameters(), lr=0.001, momentum=0.9),\n",
    "    CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,\n",
    "    evaluator=eval_plugin)\n",
    "\n",
    "# TRAINING LOOP\n",
    "print('Starting experiment...')\n",
    "results = []\n",
    "for experience in scenario.train_stream:\n",
    "    print(\"Start of experience: \", experience.current_experience)\n",
    "    print(\"Current Classes: \", experience.classes_in_this_experience)\n",
    "\n",
    "    # train returns a dictionary which contains all the metric values\n",
    "    res = cl_strategy.train(experience)\n",
    "    print('Training completed')\n",
    "\n",
    "    print('Computing accuracy on the whole test set')\n",
    "    # test also returns a dictionary which contains all the metric values\n",
    "    results.append(cl_strategy.eval(scenario.test_stream))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create your Logger\n",
    "\n",
    "If the available loggers are not sufficient to suit your needs, you can always implement a custom logger by specializing the behaviors of the `StrategyLogger` base class.\n",
    "\n",
    "This completes the \"_Logging_\" tutorial for the \"_From Zero to Hero_\" series. We hope you enjoyed it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🤝 Run it on Google Colab\n",
    "\n",
    "You can run _this chapter_ and play with it on Google Colaboratory: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ContinualAI/colab/blob/master/notebooks/avalanche/loggers.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}