{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze dataset\n",
    "In this notebook, we analyze the dataset structure and statistics.\n",
    "\n",
    "## Verify dataset and create statistics\n",
    "We verify that every folder (= session) contains exactly three subfolders:\n",
    "* Lapse (time lapse images taken every hour)\n",
    "* Motion (images taken on motion)\n",
    "* Full (pre-selected motion images with actual moving entities, should be a subset of Motion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.view()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 32 sessions\n"
     ]
    }
   ],
   "source": [
    "DIR = '/home/AMMOD_data/camera_traps/BayerWald/Vielkadaver-Projekt/'\n",
    "\n",
    "ds = Dataset(DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Beaver_01', 'Marten_01', 'Raccoon_01', 'Reddeer_01', 'Roedeer_01', 'Wildboar_01', 'Badger_02', 'Beaver_02', 'Fox_02', 'Marten_02', 'Raccoon_02', 'Rat_02', 'Reddeer_02', 'Roedeer_02', 'Badger_03', 'Fox_03', 'Raccoon_03', 'Reddeer_03', 'Wildboar_03', 'Badger_04', 'Rat_04', 'Reddeer_04', 'Wildboar_04', 'Badger_05', 'Beaver_05', 'Ermine_05', 'Fox_05', 'Marten_05', 'Raccoon_05', 'Reddeer_05', 'Roedeer_05', 'Wildboar_05']\n"
     ]
    }
   ],
   "source": [
    "print(ds.get_sessions())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 32 sessions with unique names. The name consists of the animal type and a session number. We will not count the images in all subfolders to create more advanced statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 32/32 [00:34<00:00,  1.09s/it]\n"
     ]
    }
   ],
   "source": [
    "# Only run this when statistics have not been created yet.\n",
    "# Otherwise, use the code in the cell below to load the stats from file.\n",
    "stats = ds.create_statistics()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It takes a while to create the statistics, so it makes sense to save them for later usage. The statistics can be restored using the load() method or via the constructor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded from dataset_stats.npy.\n"
     ]
    }
   ],
   "source": [
    "stats.add_total_row().save()\n",
    "# stats.load()\n",
    "# stats = DatasetStatistics(load_from_file=\"dataset_stats.npy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the stats."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Lapse</th>\n",
       "      <th>Motion</th>\n",
       "      <th>Full</th>\n",
       "      <th>Total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Badger_02</th>\n",
       "      <td>1728</td>\n",
       "      <td>4715</td>\n",
       "      <td>202</td>\n",
       "      <td>6645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Badger_03</th>\n",
       "      <td>46</td>\n",
       "      <td>4245</td>\n",
       "      <td>67</td>\n",
       "      <td>4358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Badger_04</th>\n",
       "      <td>56</td>\n",
       "      <td>480</td>\n",
       "      <td>192</td>\n",
       "      <td>728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Badger_05</th>\n",
       "      <td>1174</td>\n",
       "      <td>3860</td>\n",
       "      <td>108</td>\n",
       "      <td>5142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beaver_01</th>\n",
       "      <td>1734</td>\n",
       "      <td>695</td>\n",
       "      <td>200</td>\n",
       "      <td>2629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beaver_02</th>\n",
       "      <td>1727</td>\n",
       "      <td>2890</td>\n",
       "      <td>270</td>\n",
       "      <td>4887</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beaver_05</th>\n",
       "      <td>1321</td>\n",
       "      <td>2415</td>\n",
       "      <td>32</td>\n",
       "      <td>3768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ermine_05</th>\n",
       "      <td>867</td>\n",
       "      <td>2380</td>\n",
       "      <td>135</td>\n",
       "      <td>3382</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fox_02</th>\n",
       "      <td>957</td>\n",
       "      <td>1110</td>\n",
       "      <td>200</td>\n",
       "      <td>2267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fox_03</th>\n",
       "      <td>38</td>\n",
       "      <td>5495</td>\n",
       "      <td>206</td>\n",
       "      <td>5739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Fox_05</th>\n",
       "      <td>1083</td>\n",
       "      <td>753</td>\n",
       "      <td>65</td>\n",
       "      <td>1901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Marten_01</th>\n",
       "      <td>2462</td>\n",
       "      <td>3105</td>\n",
       "      <td>200</td>\n",
       "      <td>5767</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Marten_02</th>\n",
       "      <td>1726</td>\n",
       "      <td>883</td>\n",
       "      <td>200</td>\n",
       "      <td>2809</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Marten_05</th>\n",
       "      <td>890</td>\n",
       "      <td>16170</td>\n",
       "      <td>70</td>\n",
       "      <td>17130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Raccoon_01</th>\n",
       "      <td>850</td>\n",
       "      <td>375</td>\n",
       "      <td>120</td>\n",
       "      <td>1345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Raccoon_02</th>\n",
       "      <td>1639</td>\n",
       "      <td>332</td>\n",
       "      <td>162</td>\n",
       "      <td>2133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Raccoon_03</th>\n",
       "      <td>35</td>\n",
       "      <td>475</td>\n",
       "      <td>100</td>\n",
       "      <td>610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Raccoon_05</th>\n",
       "      <td>1089</td>\n",
       "      <td>1600</td>\n",
       "      <td>116</td>\n",
       "      <td>2805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rat_02</th>\n",
       "      <td>626</td>\n",
       "      <td>845</td>\n",
       "      <td>118</td>\n",
       "      <td>1589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Rat_04</th>\n",
       "      <td>39</td>\n",
       "      <td>640</td>\n",
       "      <td>96</td>\n",
       "      <td>775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Reddeer_01</th>\n",
       "      <td>1628</td>\n",
       "      <td>9380</td>\n",
       "      <td>200</td>\n",
       "      <td>11208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Reddeer_02</th>\n",
       "      <td>1635</td>\n",
       "      <td>675</td>\n",
       "      <td>200</td>\n",
       "      <td>2510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Reddeer_03</th>\n",
       "      <td>1027</td>\n",
       "      <td>1770</td>\n",
       "      <td>141</td>\n",
       "      <td>2938</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Reddeer_04</th>\n",
       "      <td>46</td>\n",
       "      <td>1690</td>\n",
       "      <td>89</td>\n",
       "      <td>1825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Reddeer_05</th>\n",
       "      <td>1078</td>\n",
       "      <td>18251</td>\n",
       "      <td>166</td>\n",
       "      <td>19495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Roedeer_01</th>\n",
       "      <td>1380</td>\n",
       "      <td>38820</td>\n",
       "      <td>18</td>\n",
       "      <td>40218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Roedeer_02</th>\n",
       "      <td>888</td>\n",
       "      <td>2770</td>\n",
       "      <td>206</td>\n",
       "      <td>3864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Roedeer_05</th>\n",
       "      <td>1090</td>\n",
       "      <td>13176</td>\n",
       "      <td>79</td>\n",
       "      <td>14345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wildboar_01</th>\n",
       "      <td>1732</td>\n",
       "      <td>2895</td>\n",
       "      <td>137</td>\n",
       "      <td>4764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wildboar_03</th>\n",
       "      <td>46</td>\n",
       "      <td>515</td>\n",
       "      <td>102</td>\n",
       "      <td>663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wildboar_04</th>\n",
       "      <td>39</td>\n",
       "      <td>2250</td>\n",
       "      <td>107</td>\n",
       "      <td>2396</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wildboar_05</th>\n",
       "      <td>903</td>\n",
       "      <td>22263</td>\n",
       "      <td>86</td>\n",
       "      <td>23252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Z_Total</th>\n",
       "      <td>31579</td>\n",
       "      <td>167918</td>\n",
       "      <td>4390</td>\n",
       "      <td>203887</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Lapse  Motion  Full   Total\n",
       "Badger_02     1728    4715   202    6645\n",
       "Badger_03       46    4245    67    4358\n",
       "Badger_04       56     480   192     728\n",
       "Badger_05     1174    3860   108    5142\n",
       "Beaver_01     1734     695   200    2629\n",
       "Beaver_02     1727    2890   270    4887\n",
       "Beaver_05     1321    2415    32    3768\n",
       "Ermine_05      867    2380   135    3382\n",
       "Fox_02         957    1110   200    2267\n",
       "Fox_03          38    5495   206    5739\n",
       "Fox_05        1083     753    65    1901\n",
       "Marten_01     2462    3105   200    5767\n",
       "Marten_02     1726     883   200    2809\n",
       "Marten_05      890   16170    70   17130\n",
       "Raccoon_01     850     375   120    1345\n",
       "Raccoon_02    1639     332   162    2133\n",
       "Raccoon_03      35     475   100     610\n",
       "Raccoon_05    1089    1600   116    2805\n",
       "Rat_02         626     845   118    1589\n",
       "Rat_04          39     640    96     775\n",
       "Reddeer_01    1628    9380   200   11208\n",
       "Reddeer_02    1635     675   200    2510\n",
       "Reddeer_03    1027    1770   141    2938\n",
       "Reddeer_04      46    1690    89    1825\n",
       "Reddeer_05    1078   18251   166   19495\n",
       "Roedeer_01    1380   38820    18   40218\n",
       "Roedeer_02     888    2770   206    3864\n",
       "Roedeer_05    1090   13176    79   14345\n",
       "Wildboar_01   1732    2895   137    4764\n",
       "Wildboar_03     46     515   102     663\n",
       "Wildboar_04     39    2250   107    2396\n",
       "Wildboar_05    903   22263    86   23252\n",
       "Z_Total      31579  167918  4390  203887"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.view()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Total     1.000000\n",
       "Full      0.021532\n",
       "Lapse     0.154885\n",
       "Motion    0.823584\n",
       "Name: Z_Total, dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats.df.iloc[-1] / 203887"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a total of 203,887 images. Overall, there are significantly more motion images than lapse images. We can take a better look by plotting a bar chart:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 28072 characters]",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "stats.plot_sessions(exclude_last_row=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Verify that 'Full' folder is a subset of 'Motion'\n",
    "We expect the 'Full' folder to be a subset of 'Motion'. The following code checks that by iterating over all files in 'Full' for every session and looking for them in 'Motion' of the same session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cac1c192a0149ab8c5cc0a2d52247a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/32 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VIELAAS_Spring_Session01-VIELAAS_Beaver_01: 200 of 200 found (0 not found)\n",
      "VIELAAS_Spring_Session01-VIELAAS_Marten_01: 200 of 200 found (0 not found)\n",
      "VIELAAS_Spring_Session01-VIELAAS_Raccoon_01: 120 of 120 found (0 not found)\n",
      "VIELAAS_Spring_Session01-VIELAAS_Reddeer_01: 200 of 200 found (0 not found)\n",
      "VIELAAS_Spring_Session01-VIELAAS_Roedeer_01: 18 of 18 found (0 not found)\n",
      "VIELAAS_Spring_Session01-VIELAAS_Wildboar_01: 137 of 137 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Badger_02: 202 of 202 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Beaver_02: 270 of 270 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Fox_02: 200 of 200 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Marten_02: 200 of 200 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Raccoon_02: 162 of 162 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Rat_02: 118 of 118 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Reddeer_02: 200 of 200 found (0 not found)\n",
      "VIELAAS_Spring_Session02-VIELAAS_Roedeer_02: 206 of 206 found (0 not found)\n",
      "VIELAAS_Spring_Session03-VIELAAS_Badger_03: 67 of 67 found (0 not found)\n",
      "VIELAAS_Spring_Session03-VIELAAS_Fox_03: 206 of 206 found (0 not found)\n",
      "VIELAAS_Spring_Session03-VIELAAS_Raccoon_03: 100 of 100 found (0 not found)\n",
      "VIELAAS_Spring_Session03-VIELAAS_Reddeer_03: 141 of 141 found (0 not found)\n",
      "VIELAAS_Spring_Session03-VIELAAS_Wildboar_03: 102 of 102 found (0 not found)\n",
      "VIELAAS_Spring_Session04-VIELAAS_Badger_04: 192 of 192 found (0 not found)\n",
      "VIELAAS_Spring_Session04-VIELAAS_Rat_04: 96 of 96 found (0 not found)\n",
      "VIELAAS_Spring_Session04-VIELAAS_Reddeer_04: 89 of 89 found (0 not found)\n",
      "VIELAAS_Spring_Session04-VIELAAS_Wildboar_04: 107 of 107 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Badger_05: 108 of 108 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Beaver_05: 32 of 32 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Ermine_05: 135 of 135 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Fox_05: 65 of 65 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Marten_05: 70 of 70 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Raccoon_05: 116 of 116 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Reddeer_05: 166 of 166 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Roedeer_05: 79 of 79 found (0 not found)\n",
      "VIELAAS_Spring_Session05-VIELAAS_Wildboar_05: 86 of 86 found (0 not found)\n"
     ]
    }
   ],
   "source": [
    "ok = True\n",
    "for folder in tqdm(ds.raw_sessions):\n",
    "    fullFolder = os.path.join(DIR, folder, \"Full\")\n",
    "    motionFolder = os.path.join(DIR, folder, \"Motion\")\n",
    "    fullFiles = [os.path.basename(path) for path in list_jpegs_recursive(fullFolder)]\n",
    "    motionFiles = [os.path.basename(path) for path in list_jpegs_recursive(motionFolder)]\n",
    "    fullFilesFound = 0\n",
    "    for fullFile in fullFiles:\n",
    "        if fullFile in motionFiles:\n",
    "            fullFilesFound += 1\n",
    "    print(f\"{folder}: {fullFilesFound} of {len(fullFiles)} found ({len(fullFiles) - fullFilesFound} not found)\")\n",
    "    if len(fullFiles) > fullFilesFound:\n",
    "        ok = False\n",
    "if not ok:\n",
    "    print(\"There were files missing!\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
  },
  "kernelspec": {
   "display_name": "Python 3.6.9 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
