{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "description: \"Logging... logging everywhere! \\U0001F52E\"\n", "---\n", "\n", "# Loggers\n", "\n", "Welcome to the _\"Logging\"_ tutorial of the _\"From Zero to Hero\"_ series. In this part we will present the functionalities offered by the _Avalanche_ `logging` module." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "!pip install git+https://github.com/ContinualAI/avalanche.git" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 📑 The Logging Module\n", "\n", "In the previous tutorial we have learned how to evaluate a continual learning algorithm in _Avalanche_, through different metrics that can be used _off-the-shelf_ via the _Evaluation Plugin_ or stand-alone. However, computing metrics and collecting results, may not be enough at times.\n", "\n", "While running complex experiments with long _waiting times_, **logging** results over-time is fundamental to \"_babysit_\" your experiments in real-time, or even understand what went wrong in the aftermath.\n", "\n", "This is why in Avalanche we decided to put a strong emphasis on logging and **provide a number of loggers** that can be used with any set of metrics!\n", "\n", "### Loggers\n", "\n", "_Avalanche_ at the moment supports three main Loggers:\n", "\n", "* **InteractiveLogger**: This logger provides a nice progress bar and displays real-time metrics results in an interactive way \\(meant for `stdout`\\).\n", "* **TextLogger**: This logger, mostly intended for file logging, is the plain text version of the `InteractiveLogger`. Keep in mind that it may be very verbose.\n", "* **TensorboardLogger**: It logs all the metrics on [Tensorboard](https://www.tensorflow.org/tensorboard) in real-time. Perfect for real-time plotting.\n", "\n", "In order to keep track of when each metric value has been logged, we leverage a `global counter`. You can see the `global counter` reported in the x axis of the logged plots.\n", "\n", "The `global counter` is an ever-increasing value which starts from 0 and it is increased by one each time a training or evaluation iteration is performed (i.e. after each training or evaluation minibatch).\n", "The `global counter` is updated automatically by the strategy. It should be reset by creating a new instance of the strategy.\n", "\n", "#### How to use Them" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting experiment...\n", "Start of experience: 0\n", "Current Classes: [0, 9]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 0 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:14<00:00, 1.61it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720915.6689\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720915.3457\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.0546\n", "\tLoss_MB/train_phase/train_stream/Task000 = 0.2314\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0334\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 14.4948\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.7566\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9839\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.26it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 369.0000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720916.5859\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 0.1841\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.9693\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.39it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 364.3000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720917.2822\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 4.4039\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 16.62it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 372.0000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720917.9785\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 4.9915\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.13it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 358.5000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720918.6748\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.7982\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 17.01it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 369.6000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720919.3711\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 4.1429\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[ 950, 0, 0, 0, 0, 0, 0, 0, 0, 30],\n", " [ 6, 0, 0, 0, 0, 0, 0, 0, 0, 1129],\n", " [ 510, 0, 0, 0, 0, 0, 0, 0, 0, 522],\n", " [ 397, 0, 0, 0, 0, 0, 0, 0, 0, 613],\n", " [ 38, 0, 0, 0, 0, 0, 0, 0, 0, 944],\n", " [ 416, 0, 0, 0, 0, 0, 0, 0, 0, 476],\n", " [ 448, 0, 0, 0, 0, 0, 0, 0, 0, 510],\n", " [ 35, 0, 0, 0, 0, 0, 0, 0, 0, 993],\n", " [ 154, 0, 0, 0, 0, 0, 0, 0, 0, 820],\n", " [ 31, 0, 0, 0, 0, 0, 0, 0, 0, 978]])\n", "\tDiskUsage_Stream/eval_phase/test_stream = 720920.0059\n", "\tLoss_Stream/eval_phase/test_stream = 3.6963\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.1928\n", "Start of experience: 1\n", "Current Classes: [2, 7]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 1 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:15<00:00, 1.66it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720948.1162\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720947.7930\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.8814\n", "\tLoss_MB/train_phase/train_stream/Task000 = 0.3365\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0324\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 14.7142\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.4811\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9596\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.26it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 356.7000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720951.2979\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6347\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0850\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.54it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 357.1000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720952.0020\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2836\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9578\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 16.20it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 360.1000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720952.7061\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 4.8760\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.78it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 377.6000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720953.4102\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.9460\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 17.54it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 374.7000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720954.1143\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 4.0850\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[169, 0, 697, 0, 0, 0, 0, 114, 0, 0],\n", " [ 0, 0, 549, 0, 0, 0, 0, 586, 0, 0],\n", " [ 0, 0, 991, 0, 0, 0, 0, 41, 0, 0],\n", " [ 0, 0, 762, 0, 0, 0, 0, 248, 0, 0],\n", " [ 0, 0, 147, 0, 0, 0, 0, 835, 0, 0],\n", " [ 0, 0, 537, 0, 0, 0, 0, 355, 0, 0],\n", " [ 0, 0, 928, 0, 0, 0, 0, 30, 0, 0],\n", " [ 0, 0, 46, 0, 0, 0, 0, 982, 0, 0],\n", " [ 0, 0, 730, 0, 0, 0, 0, 244, 0, 0],\n", " [ 0, 0, 51, 0, 0, 0, 0, 958, 0, 0]])\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\tDiskUsage_Stream/eval_phase/test_stream = 720954.7568\n", "\tLoss_Stream/eval_phase/test_stream = 3.3306\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2142\n", "Start of experience: 2\n", "Current Classes: [5, 6]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 2 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 23/23 [00:14<00:00, 1.58it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720963.3389\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720963.0156\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.4399\n", "\tLoss_MB/train_phase/train_stream/Task000 = 0.7766\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0335\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 14.1599\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.3270\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.8555\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.69it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 382.1000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720964.2637\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6700\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0327\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 17.14it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 363.2000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720964.9678\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 1.4572\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.4282\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 15.69it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 352.9000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720965.6719\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 0.6560\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.9032\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.79it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 363.2000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720966.3760\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 3.7727\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 16.65it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 344.9000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720967.0801\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.3084\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[ 65, 0, 0, 0, 0, 296, 617, 2, 0, 0],\n", " [ 0, 0, 7, 0, 0, 1028, 100, 0, 0, 0],\n", " [ 0, 0, 0, 0, 0, 126, 902, 4, 0, 0],\n", " [ 0, 0, 0, 0, 0, 855, 142, 13, 0, 0],\n", " [ 0, 0, 0, 0, 0, 154, 683, 145, 0, 0],\n", " [ 0, 0, 0, 0, 0, 720, 166, 6, 0, 0],\n", " [ 0, 0, 0, 0, 0, 6, 951, 1, 0, 0],\n", " [ 0, 0, 0, 0, 0, 73, 73, 882, 0, 0],\n", " [ 0, 0, 0, 0, 0, 598, 368, 8, 0, 0],\n", " [ 0, 0, 0, 0, 0, 335, 305, 369, 0, 0]])\n", "\tDiskUsage_Stream/eval_phase/test_stream = 720967.7227\n", "\tLoss_Stream/eval_phase/test_stream = 2.4015\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2618\n", "Start of experience: 3\n", "Current Classes: [8, 3]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 3 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:14<00:00, 1.69it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720976.7256\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720976.4023\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2646\n", "\tLoss_MB/train_phase/train_stream/Task000 = 0.9822\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0326\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 13.7838\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2737\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.7925\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.09it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 365.2000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720977.6504\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6221\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0417\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 15.61it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 328.5000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720978.3545\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.1221\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2772\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 15.27it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 347.9000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720979.0586\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 1.5153\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.4162\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.56it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 354.7000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720979.7627\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 0.8728\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.8261\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 16.50it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 343.2000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720980.4668\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.2265\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[ 83, 0, 0, 611, 0, 5, 162, 0, 119, 0],\n", " [ 0, 0, 0, 989, 0, 0, 3, 0, 143, 0],\n", " [ 0, 0, 0, 488, 0, 0, 175, 0, 369, 0],\n", " [ 0, 0, 0, 977, 0, 0, 1, 2, 30, 0],\n", " [ 0, 0, 0, 418, 0, 0, 199, 2, 363, 0],\n", " [ 0, 0, 0, 651, 0, 6, 6, 1, 228, 0],\n", " [ 0, 0, 0, 131, 0, 0, 764, 0, 63, 0],\n", " [ 0, 0, 0, 236, 0, 6, 4, 571, 211, 0],\n", " [ 0, 0, 0, 310, 0, 0, 2, 0, 662, 0],\n", " [ 0, 0, 0, 558, 0, 1, 57, 39, 354, 0]])\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\tDiskUsage_Stream/eval_phase/test_stream = 720981.1094\n", "\tLoss_Stream/eval_phase/test_stream = 2.0952\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.3063\n", "Start of experience: 4\n", "Current Classes: [1, 4]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 4 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 26/26 [00:15<00:00, 1.68it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720990.6611\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720990.3379\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2713\n", "\tLoss_MB/train_phase/train_stream/Task000 = 1.1053\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0316\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 15.0845\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2687\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9048\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.42it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 372.1000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720991.5859\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6000\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0392\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 17.46it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 382.3000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720992.2900\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.3497\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.2112\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 19/19 [00:01<00:00, 16.65it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 381.0000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720992.9941\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 2.1057\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.2589\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.85it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 383.6000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720993.6982\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 1.2233\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.6532\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 17.54it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 381.4000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720994.4023\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9423\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9622\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[ 78, 7, 0, 556, 55, 1, 65, 0, 218, 0],\n", " [ 0, 1133, 0, 2, 0, 0, 0, 0, 0, 0],\n", " [ 0, 330, 0, 232, 122, 0, 33, 0, 315, 0],\n", " [ 0, 117, 0, 848, 17, 0, 0, 1, 27, 0],\n", " [ 0, 68, 0, 2, 904, 0, 6, 0, 2, 0],\n", " [ 0, 260, 0, 448, 73, 0, 0, 0, 111, 0],\n", " [ 0, 146, 0, 57, 190, 0, 479, 0, 86, 0],\n", " [ 0, 201, 0, 26, 325, 0, 0, 435, 41, 0],\n", " [ 0, 353, 0, 117, 56, 0, 0, 0, 448, 0],\n", " [ 0, 124, 0, 20, 846, 0, 4, 0, 15, 0]])\n", "\tDiskUsage_Stream/eval_phase/test_stream = 720995.0449\n", "\tLoss_Stream/eval_phase/test_stream = 1.8329\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.4325\n" ] } ], "source": [ "from torch.optim import SGD\n", "from torch.nn import CrossEntropyLoss\n", "from avalanche.benchmarks.classic import SplitMNIST\n", "from avalanche.evaluation.metrics import forgetting_metrics, \\\n", "accuracy_metrics, loss_metrics, timing_metrics, cpu_usage_metrics, \\\n", "confusion_matrix_metrics, disk_usage_metrics\n", "from avalanche.models import SimpleMLP\n", "from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger\n", "from avalanche.training.plugins import EvaluationPlugin\n", "from avalanche.training.strategies import Naive\n", "\n", "scenario = SplitMNIST(n_experiences=5)\n", "\n", "# MODEL CREATION\n", "model = SimpleMLP(num_classes=scenario.n_classes)\n", "\n", "# DEFINE THE EVALUATION PLUGIN and LOGGERS\n", "# The evaluation plugin manages the metrics computation.\n", "# It takes as argument a list of metrics, collectes their results and returns\n", "# them to the strategy it is attached to.\n", "\n", "# log to Tensorboard\n", "tb_logger = TensorboardLogger()\n", "\n", "# log to text file\n", "text_logger = TextLogger(open('log.txt', 'a'))\n", "\n", "# print to stdout\n", "interactive_logger = InteractiveLogger()\n", "\n", "eval_plugin = EvaluationPlugin(\n", " accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n", " loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n", " timing_metrics(epoch=True, epoch_running=True),\n", " cpu_usage_metrics(experience=True),\n", " forgetting_metrics(experience=True, stream=True),\n", " confusion_matrix_metrics(num_classes=scenario.n_classes, save_image=False,\n", " stream=True),\n", " disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n", " loggers=[interactive_logger, text_logger, tb_logger]\n", ")\n", "\n", "# CREATE THE STRATEGY INSTANCE (NAIVE)\n", "cl_strategy = Naive(\n", " model, SGD(model.parameters(), lr=0.001, momentum=0.9),\n", " CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,\n", " evaluator=eval_plugin)\n", "\n", "# TRAINING LOOP\n", "print('Starting experiment...')\n", "results = []\n", "for experience in scenario.train_stream:\n", " print(\"Start of experience: \", experience.current_experience)\n", " print(\"Current Classes: \", experience.classes_in_this_experience)\n", "\n", " # train returns a dictionary which contains all the metric values\n", " res = cl_strategy.train(experience)\n", " print('Training completed')\n", "\n", " print('Computing accuracy on the whole test set')\n", " # test also returns a dictionary which contains all the metric values\n", " results.append(cl_strategy.eval(scenario.test_stream))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Create your Logger\n", "\n", "If the available loggers are not sufficient to suit your needs, you can always implement a custom logger by specializing the behaviors of the `StrategyLogger` base class.\n", "\n", "This completes the \"_Logging_\" tutorial for the \"_From Zero to Hero_\" series. We hope you enjoyed it!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🤝 Run it on Google Colab\n", "\n", "You can run _this chapter_ and play with it on Google Colaboratory: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ContinualAI/colab/blob/master/notebooks/avalanche/loggers.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 2 }