{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "description: Design Your Continual Learning Experiments\n",
    "---\n",
    "# Putting All Together\n",
    "\n",
    "Welcome to the \"_Putting All Together_\" tutorial of the \"_From Zero to Hero_\" series. In this part we will summarize the major Avalanche features and how you can put them together for your continual learning experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/ContinualAI/avalanche.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 🛴 A Comprehensive Example\n",
    "\n",
    "Here we report a complete example of the _Avalanche_ usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting experiment...\n",
      "Start of experience:  0\n",
      "Current Classes:  [6, 7]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 0 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:15<00:00,  1.57it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720922.1084\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720921.7852\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.0135\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.1866\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0329\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 15.5306\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.8093\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 1.0000\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.03it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 350.1000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720923.0254\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 0.1702\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.9844\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 14.77it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 370.0000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720923.7217\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 4.5884\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 13.02it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 353.2000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720925.0605\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 5.4115\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.13it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 363.8000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720925.7568\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.7702\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.54it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 374.4000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720926.4531\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 5.0441\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[   0,    0,    0,    0,    0,    0,  689,  291,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,  622,  513,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,  840,  192,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,  439,  571,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,  443,  539,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,  459,  433,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,  949,    9,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,   22, 1006,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,  539,  435,    0,    0],\n",
      "        [   0,    0,    0,    0,    0,    0,   99,  910,    0,    0]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream = 720927.0879\n",
      "\tLoss_Stream/eval_phase/test_stream = 3.9981\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.1955\n",
      "Start of experience:  1\n",
      "Current Classes:  [9, 1]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 1 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 26/26 [00:16<00:00,  1.62it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720937.0283\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720936.7051\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.7894\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.2300\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0320\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 15.6686\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.5327\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9843\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.00it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 351.7000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720937.9531\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6838\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0076\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 18.49it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 374.5000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720938.6572\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.1732\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9846\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 17.67it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 368.8000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720939.3613\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 5.3703\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 17.83it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 384.5000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720940.0654\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.8450\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.08it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 314.3000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720940.7695\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 5.1888\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[   0,   24,    0,    0,    0,    0,    3,    0,    0,  953],\n",
      "        [   0, 1129,    0,    0,    0,    0,    0,    0,    0,    6],\n",
      "        [   0,  453,    0,    0,    0,    0,    5,    0,    0,  574],\n",
      "        [   0,  535,    0,    0,    0,    0,    0,    0,    0,  475],\n",
      "        [   0,   34,    0,    0,    0,    0,    0,    0,    0,  948],\n",
      "        [   0,  197,    0,    0,    0,    0,    0,    0,    0,  695],\n",
      "        [   0,  112,    0,    0,    0,    0,   15,    0,    0,  831],\n",
      "        [   0,   82,    0,    0,    0,    0,    0,    0,    0,  946],\n",
      "        [   0,  273,    0,    0,    0,    0,    0,    0,    0,  701],\n",
      "        [   0,   27,    0,    0,    0,    0,    0,    0,    0,  982]])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tDiskUsage_Stream/eval_phase/test_stream = 720941.4121\n",
      "\tLoss_Stream/eval_phase/test_stream = 3.5860\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2126\n",
      "Start of experience:  2\n",
      "Current Classes:  [0, 8]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 2 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:15<00:00,  1.59it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720950.4150\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720950.0918\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2516\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 0.4531\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0322\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 14.7301\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.4297\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9562\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.72it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 367.5000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720951.3398\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.5768\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 14.26it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 347.9000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720952.0439\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 1.4551\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.3759\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 13.51it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 337.5000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720952.7480\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 0.3746\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.9591\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.05it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 351.9000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720953.4521\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 3.9869\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 14.13it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 329.9000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720954.1562\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.9686\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[962,   0,   0,   0,   0,   0,   0,   0,  18,   0],\n",
      "        [  0, 794,   0,   0,   0,   0,   0,   0, 341,   0],\n",
      "        [250,  13,   0,   0,   0,   0,   0,   0, 769,   0],\n",
      "        [279,   2,   0,   0,   0,   0,   0,   0, 729,   0],\n",
      "        [179,   1,   0,   0,   0,   0,   0,   0, 794,   8],\n",
      "        [315,  10,   0,   0,   0,   0,   0,   0, 567,   0],\n",
      "        [597,   2,   0,   0,   0,   0,   0,   0, 359,   0],\n",
      "        [133,  12,   0,   0,   0,   0,   0,   0, 832,  51],\n",
      "        [ 62,   0,   0,   0,   0,   0,   0,   0, 912,   0],\n",
      "        [131,   0,   0,   0,   0,   0,   0,   0, 866,  12]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream = 720954.7988\n",
      "\tLoss_Stream/eval_phase/test_stream = 2.4545\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2680\n",
      "Start of experience:  3\n",
      "Current Classes:  [3, 5]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 3 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:16<00:00,  1.46it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720963.7041\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720963.3809\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.3772\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 1.1030\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0369\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 15.9744\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2455\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.7500\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.35it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 334.7000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720964.6289\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.3725\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0076\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 16.39it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 365.6000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720965.3330\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 1.8747\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.3918\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.21it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 371.1000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720966.0371\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 1.7633\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.1796\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.97it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 359.2000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720966.7412\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 0.8960\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.7739\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 14.10it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 315.6000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720967.4453\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.2840\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[348,   0,   0, 232,   0, 400,   0,   0,   0,   0],\n",
      "        [  0, 822,   0, 267,   0,   6,   0,   0,  40,   0],\n",
      "        [ 21,  10,   0, 891,   0,  84,   0,   0,  25,   1],\n",
      "        [  0,   1,   0, 987,   0,  17,   0,   0,   4,   1],\n",
      "        [ 55,   2,   0, 260,   0, 619,   0,   0,  32,  14],\n",
      "        [  0,   5,   0, 395,   0, 485,   0,   0,   7,   0],\n",
      "        [194,   2,   0, 498,   0, 237,  13,   0,  14,   0],\n",
      "        [ 15,  22,   0, 333,   0, 519,   0,   2,  74,  63],\n",
      "        [  0,   0,   0, 768,   0, 203,   0,   0,   3,   0],\n",
      "        [ 28,   0,   0, 327,   0, 621,   0,   0,  15,  18]])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tDiskUsage_Stream/eval_phase/test_stream = 720968.0879\n",
      "\tLoss_Stream/eval_phase/test_stream = 2.0495\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2678\n",
      "Start of experience:  4\n",
      "Current Classes:  [2, 4]\n",
      "-- >> Start of training phase << --\n",
      "-- Starting training on experience 4 (Task 0) from train stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:15<00:00,  1.50it/s]\n",
      "Epoch 0 ended.\n",
      "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720976.9932\n",
      "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720976.6699\n",
      "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2955\n",
      "\tLoss_MB/train_phase/train_stream/Task000 = 1.1687\n",
      "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0367\n",
      "\tTime_Epoch/train_phase/train_stream/Task000 = 15.4365\n",
      "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2381\n",
      "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9200\n",
      "-- >> End of training phase << --\n",
      "Training completed\n",
      "Computing accuracy on the whole test set\n",
      "-- >> Start of eval phase << --\n",
      "-- Starting eval on experience 0 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.34it/s]\n",
      "> Eval on experience 0 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 353.1000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720977.9180\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.4657\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n",
      "-- Starting eval on experience 1 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 15.10it/s]\n",
      "> Eval on experience 1 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 341.7000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720978.6221\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.0167\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.3540\n",
      "-- Starting eval on experience 2 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.03it/s]\n",
      "> Eval on experience 2 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 335.3000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720979.3262\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 2.4946\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0328\n",
      "-- Starting eval on experience 3 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.09it/s]\n",
      "> Eval on experience 3 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 360.2000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720980.0303\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 1.3292\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.5484\n",
      "-- Starting eval on experience 4 (Task 0) from test stream --\n",
      "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 15.63it/s]\n",
      "> Eval on experience 4 (Task 0) from test stream ended.\n",
      "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 318.6000\n",
      "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720980.7344\n",
      "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 1.0089\n",
      "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9613\n",
      "-- >> End of eval phase << --\n",
      "\tConfusionMatrix_Stream/eval_phase/test_stream = \n",
      "tensor([[ 64,   0, 416,  12,   6, 482,   0,   0,   0,   0],\n",
      "        [  0, 759, 285,  83,   1,   7,   0,   0,   0,   0],\n",
      "        [  0,   2, 990,   4,  31,   5,   0,   0,   0,   0],\n",
      "        [  0,   1, 464, 511,  17,  17,   0,   0,   0,   0],\n",
      "        [  1,   2,  29,   1, 946,   3,   0,   0,   0,   0],\n",
      "        [  0,   2, 180, 142,  36, 532,   0,   0,   0,   0],\n",
      "        [  6,   2, 798,   2, 108,  42,   0,   0,   0,   0],\n",
      "        [  0,  18, 211,  13, 751,  30,   0,   0,   0,   5],\n",
      "        [  0,   0, 808,  24,  69,  73,   0,   0,   0,   0],\n",
      "        [  0,   0,  58,  17, 901,  33,   0,   0,   0,   0]])\n",
      "\tDiskUsage_Stream/eval_phase/test_stream = 720981.3770\n",
      "\tLoss_Stream/eval_phase/test_stream = 1.8655\n",
      "\tTop1_Acc_Stream/eval_phase/test_stream = 0.3802\n"
     ]
    }
   ],
   "source": [
    "from torch.optim import SGD\n",
    "from torch.nn import CrossEntropyLoss\n",
    "from avalanche.benchmarks.classic import SplitMNIST\n",
    "from avalanche.evaluation.metrics import forgetting_metrics, accuracy_metrics, \\\n",
    "    loss_metrics, timing_metrics, cpu_usage_metrics, confusion_matrix_metrics, disk_usage_metrics\n",
    "from avalanche.models import SimpleMLP\n",
    "from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger\n",
    "from avalanche.training.plugins import EvaluationPlugin\n",
    "from avalanche.training.strategies import Naive\n",
    "\n",
    "scenario = SplitMNIST(n_experiences=5)\n",
    "\n",
    "# MODEL CREATION\n",
    "model = SimpleMLP(num_classes=scenario.n_classes)\n",
    "\n",
    "# DEFINE THE EVALUATION PLUGIN and LOGGERS\n",
    "# The evaluation plugin manages the metrics computation.\n",
    "# It takes as argument a list of metrics, collectes their results and returns\n",
    "# them to the strategy it is attached to.\n",
    "\n",
    "# log to Tensorboard\n",
    "tb_logger = TensorboardLogger()\n",
    "\n",
    "# log to text file\n",
    "text_logger = TextLogger(open('log.txt', 'a'))\n",
    "\n",
    "# print to stdout\n",
    "interactive_logger = InteractiveLogger()\n",
    "\n",
    "eval_plugin = EvaluationPlugin(\n",
    "    accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    timing_metrics(epoch=True, epoch_running=True),\n",
    "    forgetting_metrics(experience=True, stream=True),\n",
    "    cpu_usage_metrics(experience=True),\n",
    "    confusion_matrix_metrics(num_classes=scenario.n_classes, save_image=False,\n",
    "                             stream=True),\n",
    "    disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n",
    "    loggers=[interactive_logger, text_logger, tb_logger]\n",
    ")\n",
    "\n",
    "# CREATE THE STRATEGY INSTANCE (NAIVE)\n",
    "cl_strategy = Naive(\n",
    "    model, SGD(model.parameters(), lr=0.001, momentum=0.9),\n",
    "    CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,\n",
    "    evaluator=eval_plugin)\n",
    "\n",
    "# TRAINING LOOP\n",
    "print('Starting experiment...')\n",
    "results = []\n",
    "for experience in scenario.train_stream:\n",
    "    print(\"Start of experience: \", experience.current_experience)\n",
    "    print(\"Current Classes: \", experience.classes_in_this_experience)\n",
    "\n",
    "    # train returns a dictionary which contains all the metric values\n",
    "    res = cl_strategy.train(experience)\n",
    "    print('Training completed')\n",
    "\n",
    "    print('Computing accuracy on the whole test set')\n",
    "    # test also returns a dictionary which contains all the metric values\n",
    "    results.append(cl_strategy.eval(scenario.test_stream))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 🤝 Run it on Google Colab\n",
    "\n",
    "You can run _this chapter_ and play with it on Google Colaboratory: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ContinualAI/colab/blob/master/notebooks/avalanche/6.-putting-all-together.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}