{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Approach 3 - Local Features\n",
    "For this approach, we extract local features from the Lapse images using SIFT and cluster them into a vocabulary (Bag of Words approach). Using this vocabulary, we can map every Lapse and Motion image to a BoW feature vector. The Lapse feature vectors make up the training data for a one-class SVM. The decision boundary of the SVM can then be customized to achieve the right compromise between correct keeps and correct eliminations.\n",
    "As we are using SIFT descriptors, we work on grayscale images.\n",
    "\n",
    "## Advantages\n",
    "- scale, rotation and position invariance = invariance to camera movements\n",
    "- not sensitive to lighting changes\n",
    "\n",
    "\n",
    "## Disadvantages\n",
    "- no color information\n",
    "- Instead of detecting the anomalies (animals), we detect the absence of a sufficiently large portion of the background. Smaller objects are therefore especially hard to detect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 2 sessions\n",
      "Session 'Beaver_01' at folder: /home/kleinsteuber/vscode/ResizedSessions_NoBackup/VIELAAS_Spring_Session01-VIELAAS_Beaver_01\n",
      "Loaded scans.\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "from tqdm.notebook import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import cv2 as cv\n",
    "\n",
    "from py.PlotUtils import plot_roc_curve, get_percentiles\n",
    "from py.Dataset import Dataset\n",
    "from py.ImageUtils import display_images\n",
    "\n",
    "DIR = '/home/kleinsteuber/vscode/ResizedSessions_NoBackup' # dataset directory\n",
    "\n",
    "ds = Dataset(DIR)\n",
    "session = ds.create_session(\"beaver_01\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute SIFT descriptors\n",
    "Since there are often thousands of SIFT keypoints, we limit the number of keypoints to 400. To select larger keypoints (i.e. in higher octaves), we choose a larger sigma of 2.4 (default: 1.6)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 653188 characters]",
      "text/plain": [
       "<Figure size 1152x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get motion image and closest lapse image for comparison\n",
    "motion_img = session.get_random_motion_image(day_only=True)\n",
    "before_img, after_img, rel = motion_img.get_closest_lapse_images()\n",
    "closest_img = before_img if rel <= 0.5 else after_img\n",
    "\n",
    "# read and display images\n",
    "img = motion_img.read_opencv(gray=True)\n",
    "comp_img = closest_img.read_opencv(gray=True)\n",
    "display_images([img, comp_img], [f\"Input motion img {img.shape}\", f\"Closest lapse img {comp_img.shape}\"], cmap=\"gray\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 995024 characters]",
      "text/plain": [
       "<Figure size 1152x360 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def dense_keypoints(img, step=30, size=40):\n",
    "    # calculate offset to center keypoints\n",
    "    off = ((img.shape[0] % step) // 2, (img.shape[1] % step) // 2)\n",
    "    border_dist = (step + 1) // 2\n",
    "    return [cv.KeyPoint(x, y, size) for y in range(border_dist + off[0], img.shape[0] - border_dist, step) \n",
    "                                    for x in range(border_dist + off[1], img.shape[1] - border_dist, step)]\n",
    "\n",
    "sift = cv.SIFT_create()\n",
    "\n",
    "img_kp = dense_keypoints(img)\n",
    "img_kp, img_des = sift.compute(img, img_kp)\n",
    "\n",
    "comp_img_kp = dense_keypoints(comp_img)\n",
    "comp_img_kp, comp_img_des = sift.compute(comp_img, comp_img_kp)\n",
    "\n",
    "kp_img = cv.drawKeypoints(img, img_kp, img, flags=cv.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "comp_kp_img = cv.drawKeypoints(comp_img, comp_img_kp, comp_img, flags=cv.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)\n",
    "\n",
    "display_images([kp_img, comp_kp_img], [f\"Input motion img ({len(img_kp)} keypoints)\", f\"Closest lapse img ({len(comp_img_kp)} keypoints)\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matching keypoints\n",
    "We match SIFT descriptors against each other using a BF (brute-force) matcher. In the following, we match the keypoints of the lapse and motion image for testing purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 828124 characters]",
      "text/plain": [
       "<Figure size 2304x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bf = cv.BFMatcher()\n",
    "# Match descriptors, calculate best two matches\n",
    "matches = bf.knnMatch(img_des, comp_img_des, k=2)\n",
    "\n",
    "# Apply ratio test\n",
    "good = []\n",
    "for m,n in matches:\n",
    "    if m.distance < 0.75*n.distance:\n",
    "        good.append([m])\n",
    "\n",
    "# cv.drawMatchesKnn expects list of lists as matches.\n",
    "matches_img = cv.drawMatchesKnn(img, img_kp, comp_img, comp_img_kp, good, None, flags=cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)\n",
    "display_images([matches_img], [\"Matches\"], size=(16,5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test data must be labeled (normal/anomalous)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test labels\n",
    "from py.Labels import LABELS\n",
    "\n",
    "clusters = 1024 # BOW vocabulary size\n",
    "step = 30 # DSIFT keypoint step size\n",
    "size = 40 # DSIFT keypoint size\n",
    "test_labels = []\n",
    "test_df = []\n",
    "with open(f\"./bow_train_NoBackup/{session.name}/bow_eval_{step}_{size}_{clusters}.csv\", \"r\") as f:\n",
    "    for line in f:\n",
    "        filename, df = line.split(\",\")\n",
    "        img_number = int(filename[10:-4])\n",
    "        if img_number > LABELS[session.name][\"max\"]:\n",
    "            continue\n",
    "        is_normal = (img_number in LABELS[session.name][\"normal\"])\n",
    "        test_labels.append(1 if is_normal else -1)\n",
    "        test_df.append(float(df))\n",
    "test_labels = np.array(test_labels)\n",
    "test_df = np.array(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.106 labeled normal, 0.894 anomalous\n",
      "Accuracy:                               0.816\n",
      "True anomalous rate (must be high!):    0.311 \t - How many animals are detected as animals? (recall)\n",
      "True normal rate (higher is better):    0.876 \t - How many empties are detected as empties?\n",
      "False anomalous rate (lower is better): 0.124 \t - How many empties are detected as animals?\n",
      "False normal rate (must be low!):       0.689 \t - How many animals are detected as empties?\n",
      "Precision (higher is better):           0.230 \t - How many detected animals are actually animals?\n"
     ]
    }
   ],
   "source": [
    "def eval_test(df_threshold = 0):\n",
    "    test_results = np.where(test_df >= df_threshold, np.ones(test_df.shape[0]), -np.ones(test_df.shape[0]))\n",
    "\n",
    "    accuracy = np.mean(test_results == test_labels)\n",
    "    tar = np.sum((test_results == 1) & (test_labels == 1)) / np.sum(test_labels == 1)\n",
    "    tnr = np.sum((test_results == -1) & (test_labels == -1)) / np.sum(test_labels == -1)\n",
    "    far = np.sum((test_results == 1) & (test_labels == -1)) / np.sum(test_labels == -1)\n",
    "    fnr = np.sum((test_results == -1) & (test_labels == 1)) / np.sum(test_labels == 1)\n",
    "    precision = (np.sum((test_results == 1) & (test_labels == 1)) / np.sum(test_results == 1)) if np.any(test_results == 1) else 0\n",
    "\n",
    "    return accuracy, tar, tnr, far, fnr, precision\n",
    "\n",
    "accuracy, tar, tnr, far, fnr, precision = eval_test()\n",
    "\n",
    "labeled_normal = np.mean(test_labels == 1)\n",
    "print(f\"{labeled_normal:.3f} labeled normal, {(1 - labeled_normal):.3f} anomalous\")\n",
    "\n",
    "print(f\"Accuracy:                               {accuracy:.3f}\")\n",
    "print(f\"True anomalous rate (must be high!):    {tar:.3f} \\t - How many animals are detected as animals? (recall)\")\n",
    "print(f\"True normal rate (higher is better):    {tnr:.3f} \\t - How many empties are detected as empties?\")\n",
    "print(f\"False anomalous rate (lower is better): {far:.3f} \\t - How many empties are detected as animals?\")\n",
    "print(f\"False normal rate (must be low!):       {fnr:.3f} \\t - How many animals are detected as empties?\")\n",
    "print(f\"Precision (higher is better):           {precision:.3f} \\t - How many detected animals are actually animals?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we want to find a compromise between a high number of correctly kept images (anomalies classified as anomalies, recall) and a high number of correctly eliminated images (normals classified as normals), we plot the two values against each other for various thresholds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "921b89a665be4465a54b3b074eee88c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_threshold_range = (-400, 100, 5)\n",
    "eval_results = np.empty((len(range(*df_threshold_range)), 7))\n",
    "\n",
    "for i, df_threshold in tqdm(enumerate(range(*df_threshold_range))):\n",
    "    metrics = eval_test(df_threshold)\n",
    "    eval_results[i] = [df_threshold, *metrics]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33164 characters]",
      "text/plain": [
       "<Figure size 1080x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "metrics_names = [\"Accuracy\", \"Recall/TAR - Correct keeps\", \"TNR - Correct eliminations\", \"FAR\", \"FNR\", \"Precision\"]\n",
    "enable_metrics = [1, 2]\n",
    "plt.figure(figsize=(15, 10))\n",
    "for i, metrics_name in enumerate(metrics_names):\n",
    "    if i in enable_metrics:\n",
    "        plt.plot(eval_results[:,0], eval_results[:,(i+1)], label=metrics_name, linewidth=2)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23596 characters]",
      "text/plain": [
       "<Figure size 576x576 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9 percentile : TPR = 0.9054, FPR = 0.6844 <-> TNR = 0.3156 @ thresh -99.50643053843308\n",
      "0.95 percentile : TPR = 0.9595, FPR = 0.7005 <-> TNR = 0.2995 @ thresh -101.83471300522734\n",
      "0.98 percentile : TPR = 0.9865, FPR = 0.8647 <-> TNR = 0.1353 @ thresh -125.85226202797605\n",
      "0.99 percentile : TPR = 1.0000, FPR = 0.8680 <-> TNR = 0.1320 @ thresh -126.19060529485833\n"
     ]
    }
   ],
   "source": [
    "fpr, tpr, thresholds, auc_score = plot_roc_curve(test_labels, test_df, f\"BOW k={clusters}, step={step}, size={size}\", savefile=f\"plots/approach3/roc_curves/{session.name}_{step}_{size}_{clusters}\")\n",
    "get_percentiles(fpr, tpr, thresholds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AUC Scores\n",
    "\n",
    "### Beaver_01\n",
    "\n",
    "| $k$ (clusters) | $kp$ (keypoint step size) | AUC |\n",
    "| --- | --- | ---: |\n",
    "| 512 | 30 | 0.7538 |\n",
    "| 1024 | 30 | 0.7698 |\n",
    "| 2048 | 30 | 0.7741 |\n",
    "| 1024 | 20 | 0.7406 |\n",
    "| 1024 | 30, size 60 | 0.7383 |\n",
    "\n",
    "### Marten_01\n",
    "\n",
    "| $k$ (clusters) | $kp$ (keypoint step size) | AUC |\n",
    "| --- | --- | ---: |\n",
    "| 512 | 30 | 0.7079 |\n",
    "| 1024 | 30 | 0.7099 |\n",
    "| 2048 | 30 | 0.6557 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6.9 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
