{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "description: Design Your Continual Learning Experiments\n", "---\n", "# Putting All Together\n", "\n", "Welcome to the \"_Putting All Together_\" tutorial of the \"_From Zero to Hero_\" series. In this part we will summarize the major Avalanche features and how you can put them together for your continual learning experiments." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "!pip install git+https://github.com/ContinualAI/avalanche.git" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 🛴 A Comprehensive Example\n", "\n", "Here we report a complete example of the _Avalanche_ usage:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Starting experiment...\n", "Start of experience: 0\n", "Current Classes: [6, 7]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 0 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 25/25 [00:15<00:00, 1.57it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720922.1084\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720921.7852\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.0135\n", "\tLoss_MB/train_phase/train_stream/Task000 = 0.1866\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0329\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 15.5306\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.8093\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 1.0000\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.03it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 350.1000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720923.0254\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 0.1702\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.9844\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 14.77it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 370.0000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720923.7217\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 4.5884\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 13.02it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 353.2000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720925.0605\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 5.4115\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.13it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 363.8000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720925.7568\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.7702\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.54it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 374.4000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720926.4531\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 5.0441\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[ 0, 0, 0, 0, 0, 0, 689, 291, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 622, 513, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 840, 192, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 439, 571, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 443, 539, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 459, 433, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 949, 9, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 22, 1006, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 539, 435, 0, 0],\n", " [ 0, 0, 0, 0, 0, 0, 99, 910, 0, 0]])\n", "\tDiskUsage_Stream/eval_phase/test_stream = 720927.0879\n", "\tLoss_Stream/eval_phase/test_stream = 3.9981\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.1955\n", "Start of experience: 1\n", "Current Classes: [9, 1]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 1 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 26/26 [00:16<00:00, 1.62it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720937.0283\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720936.7051\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 1.7894\n", "\tLoss_MB/train_phase/train_stream/Task000 = 0.2300\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0320\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 15.6686\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.5327\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9843\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 16.00it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 351.7000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720937.9531\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.6838\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0076\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 18.49it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 374.5000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720938.6572\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 0.1732\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.9846\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 17.67it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 368.8000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720939.3613\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 5.3703\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 17.83it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 384.5000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720940.0654\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 4.8450\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 16.08it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 314.3000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720940.7695\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 5.1888\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[ 0, 24, 0, 0, 0, 0, 3, 0, 0, 953],\n", " [ 0, 1129, 0, 0, 0, 0, 0, 0, 0, 6],\n", " [ 0, 453, 0, 0, 0, 0, 5, 0, 0, 574],\n", " [ 0, 535, 0, 0, 0, 0, 0, 0, 0, 475],\n", " [ 0, 34, 0, 0, 0, 0, 0, 0, 0, 948],\n", " [ 0, 197, 0, 0, 0, 0, 0, 0, 0, 695],\n", " [ 0, 112, 0, 0, 0, 0, 15, 0, 0, 831],\n", " [ 0, 82, 0, 0, 0, 0, 0, 0, 0, 946],\n", " [ 0, 273, 0, 0, 0, 0, 0, 0, 0, 701],\n", " [ 0, 27, 0, 0, 0, 0, 0, 0, 0, 982]])\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\tDiskUsage_Stream/eval_phase/test_stream = 720941.4121\n", "\tLoss_Stream/eval_phase/test_stream = 3.5860\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2126\n", "Start of experience: 2\n", "Current Classes: [0, 8]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 2 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:15<00:00, 1.59it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720950.4150\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720950.0918\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2516\n", "\tLoss_MB/train_phase/train_stream/Task000 = 0.4531\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0322\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 14.7301\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.4297\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9562\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.72it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 367.5000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720951.3398\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.5768\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 14.26it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 347.9000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720952.0439\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 1.4551\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.3759\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 13.51it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 337.5000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720952.7480\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 0.3746\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.9591\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.05it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 351.9000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720953.4521\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 3.9869\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.0000\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 14.13it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 329.9000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720954.1562\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.9686\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[962, 0, 0, 0, 0, 0, 0, 0, 18, 0],\n", " [ 0, 794, 0, 0, 0, 0, 0, 0, 341, 0],\n", " [250, 13, 0, 0, 0, 0, 0, 0, 769, 0],\n", " [279, 2, 0, 0, 0, 0, 0, 0, 729, 0],\n", " [179, 1, 0, 0, 0, 0, 0, 0, 794, 8],\n", " [315, 10, 0, 0, 0, 0, 0, 0, 567, 0],\n", " [597, 2, 0, 0, 0, 0, 0, 0, 359, 0],\n", " [133, 12, 0, 0, 0, 0, 0, 0, 832, 51],\n", " [ 62, 0, 0, 0, 0, 0, 0, 0, 912, 0],\n", " [131, 0, 0, 0, 0, 0, 0, 0, 866, 12]])\n", "\tDiskUsage_Stream/eval_phase/test_stream = 720954.7988\n", "\tLoss_Stream/eval_phase/test_stream = 2.4545\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2680\n", "Start of experience: 3\n", "Current Classes: [3, 5]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 3 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:16<00:00, 1.46it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720963.7041\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720963.3809\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.3772\n", "\tLoss_MB/train_phase/train_stream/Task000 = 1.1030\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0369\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 15.9744\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2455\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.7500\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.35it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 334.7000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720964.6289\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.3725\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0076\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 16.39it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 365.6000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720965.3330\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 1.8747\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.3918\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.21it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 371.1000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720966.0371\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 1.7633\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.1796\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.97it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 359.2000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720966.7412\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 0.8960\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.7739\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 14.10it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 315.6000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720967.4453\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 3.2840\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.0000\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[348, 0, 0, 232, 0, 400, 0, 0, 0, 0],\n", " [ 0, 822, 0, 267, 0, 6, 0, 0, 40, 0],\n", " [ 21, 10, 0, 891, 0, 84, 0, 0, 25, 1],\n", " [ 0, 1, 0, 987, 0, 17, 0, 0, 4, 1],\n", " [ 55, 2, 0, 260, 0, 619, 0, 0, 32, 14],\n", " [ 0, 5, 0, 395, 0, 485, 0, 0, 7, 0],\n", " [194, 2, 0, 498, 0, 237, 13, 0, 14, 0],\n", " [ 15, 22, 0, 333, 0, 519, 0, 2, 74, 63],\n", " [ 0, 0, 0, 768, 0, 203, 0, 0, 3, 0],\n", " [ 28, 0, 0, 327, 0, 621, 0, 0, 15, 18]])\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\tDiskUsage_Stream/eval_phase/test_stream = 720968.0879\n", "\tLoss_Stream/eval_phase/test_stream = 2.0495\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.2678\n", "Start of experience: 4\n", "Current Classes: [2, 4]\n", "-- >> Start of training phase << --\n", "-- Starting training on experience 4 (Task 0) from train stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 24/24 [00:15<00:00, 1.50it/s]\n", "Epoch 0 ended.\n", "\tDiskUsage_Epoch/train_phase/train_stream/Task000 = 720976.9932\n", "\tDiskUsage_MB/train_phase/train_stream/Task000 = 720976.6699\n", "\tLoss_Epoch/train_phase/train_stream/Task000 = 2.2955\n", "\tLoss_MB/train_phase/train_stream/Task000 = 1.1687\n", "\tRunningTime_Epoch/train_phase/train_stream/Task000 = 0.0367\n", "\tTime_Epoch/train_phase/train_stream/Task000 = 15.4365\n", "\tTop1_Acc_Epoch/train_phase/train_stream/Task000 = 0.2381\n", "\tTop1_Acc_MB/train_phase/train_stream/Task000 = 0.9200\n", "-- >> End of training phase << --\n", "Training completed\n", "Computing accuracy on the whole test set\n", "-- >> Start of eval phase << --\n", "-- Starting eval on experience 0 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.34it/s]\n", "> Eval on experience 0 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 353.1000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp000 = 720977.9180\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.4657\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000\n", "-- Starting eval on experience 1 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 22/22 [00:01<00:00, 15.10it/s]\n", "> Eval on experience 1 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 341.7000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp001 = 720978.6221\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.0167\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.3540\n", "-- Starting eval on experience 2 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.03it/s]\n", "> Eval on experience 2 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 335.3000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp002 = 720979.3262\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp002 = 2.4946\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0328\n", "-- Starting eval on experience 3 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.09it/s]\n", "> Eval on experience 3 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 360.2000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp003 = 720980.0303\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp003 = 1.3292\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.5484\n", "-- Starting eval on experience 4 (Task 0) from test stream --\n", "100%|██████████████████████████████████████████████████████████████████████████████████| 21/21 [00:01<00:00, 15.63it/s]\n", "> Eval on experience 4 (Task 0) from test stream ended.\n", "\tCPUUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 318.6000\n", "\tDiskUsage_Exp/eval_phase/test_stream/Task000/Exp004 = 720980.7344\n", "\tLoss_Exp/eval_phase/test_stream/Task000/Exp004 = 1.0089\n", "\tTop1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9613\n", "-- >> End of eval phase << --\n", "\tConfusionMatrix_Stream/eval_phase/test_stream = \n", "tensor([[ 64, 0, 416, 12, 6, 482, 0, 0, 0, 0],\n", " [ 0, 759, 285, 83, 1, 7, 0, 0, 0, 0],\n", " [ 0, 2, 990, 4, 31, 5, 0, 0, 0, 0],\n", " [ 0, 1, 464, 511, 17, 17, 0, 0, 0, 0],\n", " [ 1, 2, 29, 1, 946, 3, 0, 0, 0, 0],\n", " [ 0, 2, 180, 142, 36, 532, 0, 0, 0, 0],\n", " [ 6, 2, 798, 2, 108, 42, 0, 0, 0, 0],\n", " [ 0, 18, 211, 13, 751, 30, 0, 0, 0, 5],\n", " [ 0, 0, 808, 24, 69, 73, 0, 0, 0, 0],\n", " [ 0, 0, 58, 17, 901, 33, 0, 0, 0, 0]])\n", "\tDiskUsage_Stream/eval_phase/test_stream = 720981.3770\n", "\tLoss_Stream/eval_phase/test_stream = 1.8655\n", "\tTop1_Acc_Stream/eval_phase/test_stream = 0.3802\n" ] } ], "source": [ "from torch.optim import SGD\n", "from torch.nn import CrossEntropyLoss\n", "from avalanche.benchmarks.classic import SplitMNIST\n", "from avalanche.evaluation.metrics import forgetting_metrics, accuracy_metrics, \\\n", " loss_metrics, timing_metrics, cpu_usage_metrics, confusion_matrix_metrics, disk_usage_metrics\n", "from avalanche.models import SimpleMLP\n", "from avalanche.logging import InteractiveLogger, TextLogger, TensorboardLogger\n", "from avalanche.training.plugins import EvaluationPlugin\n", "from avalanche.training.strategies import Naive\n", "\n", "scenario = SplitMNIST(n_experiences=5)\n", "\n", "# MODEL CREATION\n", "model = SimpleMLP(num_classes=scenario.n_classes)\n", "\n", "# DEFINE THE EVALUATION PLUGIN and LOGGERS\n", "# The evaluation plugin manages the metrics computation.\n", "# It takes as argument a list of metrics, collectes their results and returns\n", "# them to the strategy it is attached to.\n", "\n", "# log to Tensorboard\n", "tb_logger = TensorboardLogger()\n", "\n", "# log to text file\n", "text_logger = TextLogger(open('log.txt', 'a'))\n", "\n", "# print to stdout\n", "interactive_logger = InteractiveLogger()\n", "\n", "eval_plugin = EvaluationPlugin(\n", " accuracy_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n", " loss_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n", " timing_metrics(epoch=True, epoch_running=True),\n", " forgetting_metrics(experience=True, stream=True),\n", " cpu_usage_metrics(experience=True),\n", " confusion_matrix_metrics(num_classes=scenario.n_classes, save_image=False,\n", " stream=True),\n", " disk_usage_metrics(minibatch=True, epoch=True, experience=True, stream=True),\n", " loggers=[interactive_logger, text_logger, tb_logger]\n", ")\n", "\n", "# CREATE THE STRATEGY INSTANCE (NAIVE)\n", "cl_strategy = Naive(\n", " model, SGD(model.parameters(), lr=0.001, momentum=0.9),\n", " CrossEntropyLoss(), train_mb_size=500, train_epochs=1, eval_mb_size=100,\n", " evaluator=eval_plugin)\n", "\n", "# TRAINING LOOP\n", "print('Starting experiment...')\n", "results = []\n", "for experience in scenario.train_stream:\n", " print(\"Start of experience: \", experience.current_experience)\n", " print(\"Current Classes: \", experience.classes_in_this_experience)\n", "\n", " # train returns a dictionary which contains all the metric values\n", " res = cl_strategy.train(experience)\n", " print('Training completed')\n", "\n", " print('Computing accuracy on the whole test set')\n", " # test also returns a dictionary which contains all the metric values\n", " results.append(cl_strategy.eval(scenario.test_stream))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 🤝 Run it on Google Colab\n", "\n", "You can run _this chapter_ and play with it on Google Colaboratory: [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ContinualAI/colab/blob/master/notebooks/avalanche/6.-putting-all-together.ipynb)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.9" } }, "nbformat": 4, "nbformat_minor": 2 }