{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.x"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 5 - Clustering and Anomaly Detection 🧩🚨\n",
    "### CIS 350: AI for Cybersecurity - Smart Defense for the Digital Business\n",
    "\n",
    "**Worth:** 50 points  ·  **Estimated time:** ~60 minutes  ·  **Submit:** this `.ipynb` in Canvas\n",
    "**Required software:** Google Colab (or Jupyter) with `pandas`, `scikit-learn`, `matplotlib` (preinstalled in Colab).\n",
    "**Prerequisites:** Weeks 5-6 - unsupervised learning, clustering, k-means, scaling, and anomaly detection.\n",
    "\n",
    "## Background\n",
    "You are a security analyst with a month of **network sessions** and **no labels** - nobody has told you which sessions are normal or malicious. This is **unsupervised learning**. First you will **cluster** the traffic into behavior groups (after **scaling** and choosing **k** with the **elbow method**), then you will hunt **anomalies** with an **Isolation Forest** - exactly the workflow from this week's lectures.\n",
    "\n",
    "## Learning objectives\n",
    "By the end you can: (1) scale features and explain why; (2) choose k with the elbow method; (3) run and interpret k-means clusters on security data; (4) detect anomalies with an Isolation Forest and identify suspicious sessions.\n",
    "\n",
    "Run cells top to bottom with **Shift + Enter** and fill every **`TODO`** blank (`____`).\n",
    "\n",
    "> **Academic integrity:** individual or pair work. **AI tools are not permitted** on labs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 · Meet the data (no labels!)\n",
    "Each **row** is one network session described by 4 **features**. There is **no label** column - we must find structure ourselves."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "rng = np.random.default_rng(6)\n",
    "n = 80   # 80 sessions in each of three normal behavior groups\n",
    "\n",
    "def group(duration, volume, packets, failed, spread):\n",
    "    \"\"\"One behavior group: 80 network sessions around typical values.\"\"\"\n",
    "    return np.column_stack([\n",
    "        np.abs(rng.normal(duration, spread[0], n)),   # session length (minutes)\n",
    "        np.abs(rng.normal(volume,   spread[1], n)),   # data volume (MB)\n",
    "        np.abs(rng.normal(packets,  spread[2], n)),   # packets (x100)\n",
    "        np.abs(rng.normal(failed,   spread[3], n)),   # failed connections\n",
    "    ])\n",
    "\n",
    "A = group(2, 2, 0.3, 0.2, (0.45, 0.45, 0.06, 0.3))   # quick, low-volume (web)\n",
    "B = group(8, 2, 0.5, 0.2, (0.50, 0.50, 0.08, 0.3))   # long, low-volume (chatty)\n",
    "C = group(5, 8, 1.2, 0.3, (0.50, 0.50, 0.15, 0.3))   # bulk transfer (high volume)\n",
    "\n",
    "traffic = pd.DataFrame(np.vstack([A, B, C]),\n",
    "    columns=[\"duration\", \"data_volume\", \"packets\", \"failed_conns\"]).round(2)\n",
    "# NOTE: no \"label\" column - this is UNSUPERVISED. We do not know the groups in advance.\n",
    "\n",
    "print('sessions:', len(traffic))\n",
    "traffic.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 · Scale the features (do this FIRST)\n",
    "k-means uses distance, so a big-range feature would dominate. Standardizing puts features on a comparable scale."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "X = traffic[[\"duration\",\"data_volume\",\"packets\",\"failed_conns\"]]\n",
    "# TODO 1: scale the features so no single one dominates the distance.\n",
    "Xs = StandardScaler().fit_transform(____)\n",
    "print(\"scaled shape:\", Xs.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 · Choose k with the elbow method\n",
    "Try several k and look for the 'elbow' where extra clusters stop helping much."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "import matplotlib.pyplot as plt\n",
    "inertia = []\n",
    "for k in range(1, 8):\n",
    "    km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(Xs)\n",
    "    inertia.append(km.inertia_)\n",
    "plt.plot(range(1, 8), inertia, \"-o\")\n",
    "plt.xlabel(\"k\"); plt.ylabel(\"inertia\"); plt.title(\"Elbow method\"); plt.show()\n",
    "# TODO 2: from the plot, what k is the elbow? Set it here.\n",
    "best_k = ____"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4 · Cluster with k-means\n",
    "Fit k-means on the **scaled** data using your chosen k."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# TODO 3: cluster the SCALED data (Xs) into best_k groups.\n",
    "km = KMeans(n_clusters=best_k, n_init=10, random_state=0)\n",
    "traffic[\"cluster\"] = km.fit_predict(____)\n",
    "print(traffic[\"cluster\"].value_counts().sort_index())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5 · Interpret the clusters\n",
    "Look at each cluster's average features and give it a plain-language name."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# TODO 4: average each feature within each cluster.\n",
    "summary = traffic.groupby(\"cluster\")[[\"duration\",\"data_volume\",\"packets\",\"failed_conns\"]].____()\n",
    "print(summary.round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6 · New sessions arrive - hunt anomalies\n",
    "Eight new sessions just came in. We add them and look for ones that don't fit normal behavior."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "suspicious = pd.DataFrame([\n",
    "    [1.0, 1.0, 3.2, 14.0], [9.5, 9.0, 2.2, 0.0], [0.5, 0.5, 0.3, 22.0], [12.0, 1.0, 0.1, 0.0],\n",
    "    [5.0, 16.0, 3.0, 0.0], [0.7, 0.6, 3.5, 18.0], [10.5, 10.5, 2.5, 1.0], [2.0, 14.0, 0.2, 0.0]],\n",
    "    columns=[\"duration\",\"data_volume\",\"packets\",\"failed_conns\"])\n",
    "\n",
    "monitored = pd.concat([traffic[[\"duration\",\"data_volume\",\"packets\",\"failed_conns\"]],\n",
    "                       suspicious], ignore_index=True)\n",
    "# TODO 5: scale the monitored sessions before anomaly detection.\n",
    "XsM = StandardScaler().fit_transform(____)\n",
    "print(\"monitored sessions:\", len(monitored))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 7 · Isolation Forest (flag the outliers)\n",
    "`contamination=0.03` tells the model to expect about 3% anomalies. It returns -1 for anomalies."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.ensemble import IsolationForest\n",
    "iso = IsolationForest(contamination=0.03, random_state=0).fit(XsM)\n",
    "# TODO 6: predict which sessions are anomalies (-1) vs normal (1).\n",
    "monitored[\"anomaly\"] = iso.____(XsM)\n",
    "print(\"flagged anomalies:\", (monitored[\"anomaly\"] == -1).sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 8 · Inspect the flagged sessions\n",
    "Which sessions did the model flag? Do they look suspicious?"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# TODO 7: show only the rows the model flagged as anomalies (-1).\n",
    "flagged = monitored[monitored[\"anomaly\"] == ____]\n",
    "print(flagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 9 · Reflection questions\n",
    "Double-click and replace each *your answer here*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your answers**\n",
    "\n",
    "1. **Why did we SCALE the features before clustering? What could go wrong if we didn't?** *your answer here*\n",
    "2. **What k did the elbow suggest, and how many sessions landed in each cluster?** *your answer here*\n",
    "3. **Describe each cluster in plain words from its average features (e.g., 'long, low-volume sessions'). What behavior might each be?** *your answer here*\n",
    "4. **How many sessions did the Isolation Forest flag? Pick one flagged session and explain why it looks suspicious.** *your answer here*\n",
    "5. **This data had NO labels. Why is unsupervised anomaly detection useful for catching NEW attacks a labeled model was never trained on?** *your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ✅ Before you submit\n",
    "- [ ] **Runtime -> Run all** with no errors.  - [ ] All `TODO` blanks filled.\n",
    "- [ ] All 5 reflection questions answered.  - [ ] **File -> Download -> Download .ipynb**, upload to Canvas.\n",
    "\n",
    "## Troubleshooting\n",
    "- `NameError` -> you skipped a cell; run from the top with **Run all**.\n",
    "- Clusters look lopsided or weird -> did you fit on the **scaled** data `Xs` (not the raw `X`)?\n",
    "- `IsolationForest` flags too many/few -> check `contamination` (we use 0.03 = about 3%).\n",
    "- Different cluster **numbers** than a neighbor -> that's fine; cluster labels (0,1,2) are arbitrary names.\n",
    "\n",
    "## Extension (optional, ungraded)\n",
    "Run a `LocalOutlierFactor(n_neighbors=20, contamination=0.03)` on the same data and compare which sessions it flags versus the Isolation Forest. Where do they agree?"
   ]
  }
 ]
}