{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Beaver_01\n",
    "1734 Lapse images, 695 Motion images.\n",
    "\n",
    "| Approach | Configuration | Best AUC | TNR @TPR $\\geq$ 0.9 | TNR @TPR $\\geq$ 0.95 | TNR @TPR $\\geq$ 0.99 | approx train time | approx eval time |\n",
    "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |\n",
    "| 1a - Basic Frame Differencing | abs var | 0.7414 | 0.4865 | 0.4189 | 0.2432 | 0 | 1:00 min |\n",
    "| | $\\sigma=2$, sq var | 0.8986 | 0.7162 | 0.6081 | 0.5270 | 0 | 1:30 min |\n",
    "| | $\\sigma=4$, sq var | 0.9156 | 0.7973 | 0.6486 | 0.5676 | 0 | 1:30 min |\n",
    "| 1b - Histogram Comparison | p-mean | 0.6707 | | | | | |\n",
    "| 2 - Background Estimation | sq var | 0.7897 | 0.6622 | 0.5946 | 0.2703 | 0 | < 1 min |\n",
    "| | $\\sigma=2$, sq var | 0.8735 | 0.7973 | 0.7162 | 0.4865 | 0 | 1:00 min |\n",
    "| | $\\sigma=4$, sq var | 0.8776 | 0.7838 | 0.7027 | 0.4459 | 0 | 1:00 min |\n",
    "| 3 - BOW | $k=1024, kp=30$ | 0.7698 | 0.3929 | 0.3800 | 0.0757 | | |\n",
    "| | $k=2048, kp=30$ | 0.7741 | 0.4976 | 0.3382 | 0.0564 | | |\n",
    "| | $k=4096, kp=30$ | 0.7837 | 0.5797 | 0.2866 | 0.0451 | 4:00 h | 2:10 min |\n",
    "| | $k=2048, kp=40$ | 0.7611 | 0.3317 | 0.1610 | 0.1320 | 1:10 h | 1:30 min |\n",
    "| 3 - BOW +motion | $k=1024, kp=30$, +motion | 0.7056 | 0.2432 | 0.2222 | 0.0821 | | |\n",
    "| | $k=2048, kp=30$, +motion | 0.7390 | 0.3172 | 0.3092 | 0.0612 | | |\n",
    "| | $k=4096, kp=30$, +motion | 0.7542 | 0.3768 | 0.2963 | 0.0515 | 5:30 h | 2:10 min |\n",
    "| | $k=2048, kp=40$, +motion | 0.7388 | 0.1852 | 0.1820 | 0.0467 | 2:40 h | 2:20 min |\n",
    "| 3 - BOW random | $k=2048, kp=30$, random | 0.8002 | 0.6296 | 0.5588 | 0.0258 | 8 min | 1:30 min |\n",
    "| | $k=4096, kp=30$, random | 0.8022 | 0.6602 | 0.2738 | 0.1353 | 8 min | 2:00 min |\n",
    "| | $k=8192, kp=30$, random | 0.7973 | 0.6151 | 0.3913 | 0.2061 | 8 min | 3:30 min |\n",
    "| | $k=2048, kp=20$, random | 0.7943 | 0.5862 | 0.5539 | 0.2399 | 15 min | 3:00 min |\n",
    "| | $k=4096, kp=20$, random | 0.8088 | 0.6329 | 0.5459 | 0.2432 | 15 min | 4:00 min |\n",
    "| 4 - Autoencoder | Deep2 | 0.8678 | 0.5946 | 0.5000 | 0.0405 | 7:00 min | < 0:30 min |\n",
    "| | Deep2 +Noise(.015) | 0.8930 | 0.7432 | 0.4459 | 0.0000 | 7:00 min | < 0:30 min |\n",
    "| | Deep2 +Noise(.015) +Sparse(1e-4) | 0.8445 | 0.2703 | 0.1081 | 0.0541 | 7:00 min |  1:00 min |\n",
    "| | Deep3 | 0.8663 | 0.7703 | 0.5946 | 0.1081 | 7:00 min | < 0:30 min |\n",
    "| | Deep3 +Noise(.015) | 0.8542 | 0.8486 | 0.4324 | 0.0946 | 7:00 min | < 0:30 min |\n",
    "| | Deep3 +Noise(.015) +Sparse(1e-4) | 0.7608 | 0.0811 | 0.0676 | 0.0405 | 7:00 min |  1:00 min |\n",
    "| | Deep +Noise +Sparse Loss (lr=1e-4, 200 epochs, reg=0.1) | 0.7479 | 0.2086 | 0.1138 | 0.0008 | 8:30 min | 1:30 min |\n",
    "| | Deep +Noise +Sparse KDE | 0.9209 | 0.8514 | 0.6892 | 0.1216 | 6 min | < 0:30 min |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Marten_01\n",
    "2462 Lapse images (with many doubles), 3105 Motion images.\n",
    "\n",
    "| Approach | Configuration | Best AUC | TNR @TPR $\\geq$ 0.9 | TNR @TPR $\\geq$ 0.95 | TNR @TPR $\\geq$ 0.99 | approx train time | approx eval time |\n",
    "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |\n",
    "| 1a - Basic Frame Differencing | sq var | 0.6363 | 0.0244 | 0.0215 | 0.0160 | 0 | 5 min |\n",
    "| | $\\sigma=2$, sq var | 0.8004 | 0.3236 | 0.1606 | 0.0434 | 0 | 6 min |\n",
    "| | $\\sigma=4$, sq var | 0.8030 | 0.3536 | 0.2031 | 0.0801 | 0 | 6 min |\n",
    "| 2 - Background Estimation | sqmean | 0.5056 | 0.0295 | 0.0219 | 0.0169 | 0 | 2:30 min |\n",
    "| | $\\sigma=4$, sqvar | 0.7403 | 0.2090 | 0.1150 | 0.0253 | 0 | 4:00 min |\n",
    "| 3 - BOW | $k = 4096, kp = 30$, random | 0.6619 | 0.4973 | 0.2186 | 0.1298 | 12 min | 13 min |\n",
    "| 4 - Autoencoder | Deep3 | 0.6912 | 0.1471 | 0.0721 | 0.0013 | 10:00 min |  1:00 min |\n",
    "| | Deep3 +Noise(.015) | 0.7582 | 0.3767 | 0.0206 | 0.0206 | 10:00 min |  1:00 min |\n",
    "| | Deep3 +Noise(.015) +Sparse(1e-4) | 0.6120 | 0.1753 | 0.1037 | 0.0013 | 10:00 min |  1:00 min |\n",
    "| | Deep2 | 0.7207 | 0.1745 | 0.0464 | 0.0198 | 20:00 min | < 0:30 min |\n",
    "| | Deep2 +Noise(.015) | 0.7200 | 0.1884 | 0.0881 | 0.0088 | 20:00 min | < 0:30 min |\n",
    "| | Deep2 +Noise(.015) +Sparse(1e-4) | 0.6553 | 0.0194 | 0.0114 | 0.0038 | 20:00 min | < 0:30 min |\n",
    "| | Deep +Noise +Sparse Loss (lr=1e-4, 200 epochs, reg=0.1) | 0.7479 | 0.2086 | 0.1138 | 0.0008 | 8:30 min | 1:30 min |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GFox_03\n",
    "**Generated set:** The Lapse set was randomly selected from the labeled Motion set. This was necessary due to a lack of Lapse images (only one per day instead of per hour).\n",
    "\n",
    "**Lapse generation procedure:** Take a random set of consecutively taken Motion images. If all images are annotated as images, add the whole set to Lapse and remove it from Motion. Repeat until at least 1200 Lapse images were selected.\n",
    "\n",
    "| Approach | Configuration | Best AUC | TNR @TPR $\\geq$ 0.9 | TNR @TPR $\\geq$ 0.95 | TNR @TPR $\\geq$ 0.99 | approx train time | approx eval time |\n",
    "| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |\n",
    "| 1a - Basic Frame Differencing | n.a. | | | | | | |\n",
    "| 2 - Background Estimation | sqmean | 0.4745 | 0.0903 | 0.0493 | 0.0189 | 0 | 4:00 min |\n",
    "| | $\\sigma=4$, sqmean | 0.4793 | 0.0650 | 0.0434 | 0.0096 | 0 | 4:30 min |\n",
    "| 3 - BOW | $k = 4096, kp = 30$, random | 0.9743 | 0.9715 | 0.8333 | 0.4837 | 3:00 min | 8:00 min |\n",
    "| 4 - Autoencoder | Deep2 | 0.9713 | 0.9510 | 0.9049 | 0.6701 | 6:00 min | < 0:30 min |\n",
    "| | Deep2 (Loss) | 0.9861 | 0.9892 | 0.9470 | 0.6481 | 6:00 min | < 0:30 min |\n",
    "| | Deep2 +Noise(.015) | 0.9684 | 0.9274 | 0.8347 | 0.6838 | 6:00 min | < 0:30 min |\n",
    "| | Deep2 +Noise(.015) (Loss) | 0.9858 | 0.9852 | 0.9551 | 0.6344 | 6:00 min | < 0:30 min |\n",
    "| | Deep2 +Noise(.015) +Sparse(1e-4) | 0.9749 | 0.9631 | 0.8668 | 0.6320 | 6:00 min |  1:00 min |\n",
    "| | Deep2 +Noise(.015) +Sparse(1e-4) (Loss) | 0.9847 | 0.9916 | 0.9474 | 0.6140 | 6:00 min |  1:00 min |\n",
    "| | Deep3 | 0.9012 | 0.8455 | 0.7472 | 0.6746 | 6:00 min | < 0:30 min |\n",
    "| | Deep3 (Loss) | 0.9887 | 0.9932 | 0.9835 | 0.7913 | 6:00 min | < 0:30 min |\n",
    "| | Deep3 +Noise(.015) | 0.9042 | 0.8387 | 0.7476 | 0.6465 | 6:00 min | < 0:30 min |\n",
    "| | Deep3 +Noise(.015) (Loss) | 0.9889 | 0.9920 | 0.9715 | 0.8363 | 6:00 min | < 0:30 min |\n",
    "| | Deep3 +Noise(.015) +Sparse(1e-4) | 0.9356 | 0.8732 | 0.8351 | 0.1597 | 6:00 min |  1:00 min |\n",
    "| | Deep3 +Noise(.015) +Sparse(1e-4) (Loss) | 0.9829 | 0.9872 | 0.9117 | 0.6726 | 6:00 min |  1:00 min |\n",
    "| | Deep +Noise +Sparse KDE (lr=1e-4) | 0.9684 | 0.9579 | 0.9041 | 0.4964 | 5:00 min | < 0:30 min |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "gaia5: python train_bow.py ResizedSessions_NoBackup beaver_01 --clusters 4096 --include_motion && python eval_bow.py ResizedSessions_NoBackup beaver_01 --clusters 4096 --include_motion\n",
    "\n",
    "gaia4: python train_bow.py ResizedSessions_NoBackup beaver_01 --clusters 4096 && python eval_bow.py ResizedSessions_NoBackup beaver_01 --clusters 4096\n",
    "\n",
    "gaia3: python train_bow.py ResizedSessions_NoBackup beaver_01 --clusters 2048 --step_size 40 && python eval_bow.py ResizedSessions_NoBackup beaver_01 --clusters 2048 --step_size 40\n",
    "\n",
    "herkules: python train_bow.py ResizedSessions_NoBackup beaver_01 --clusters 2048 --step_size 40 --include_motion && python eval_bow.py ResizedSessions_NoBackup beaver_01 --clusters 2048 --step_size 40 --include_motion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.687806e-310, 0.000000e+000],\n",
       "       [0.000000e+000, 0.000000e+000]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "a = np.empty((3, 2))\n",
    "a[np.random.choice(len(a), size=2, replace=False)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6.9 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}