{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.x"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 4 - Judging a Phishing URL Detector ⚖️\n",
    "### CIS 350: AI for Cybersecurity - Smart Defense for the Digital Business\n",
    "\n",
    "**Worth:** 50 points  ·  **Estimated time:** ~60 minutes  ·  **Submit:** this `.ipynb` in Canvas\n",
    "**Required software:** Google Colab (or Jupyter) with `pandas`, `scikit-learn`, `matplotlib` (preinstalled in Colab).\n",
    "**Prerequisites:** Weeks 2-4 - datasets, features/labels, train/test, and model evaluation.\n",
    "\n",
    "## Background\n",
    "Last week you *built* a classifier. This week you become the **quality inspector**. A phishing-URL detector already exists - your job is to judge whether it is any good and to tune it for a real business. You will compute **accuracy**, read a **confusion matrix**, measure **precision and recall**, and slide the **alert threshold** to trade off false alarms against missed attacks - exactly the ideas from Wednesday's lecture.\n",
    "\n",
    "## Learning objectives\n",
    "By the end you can: (1) evaluate a classifier with accuracy and a confusion matrix; (2) compute and interpret precision and recall; (3) adjust a decision threshold and explain the false-positive / false-negative trade-off; (4) recommend a threshold for a business context.\n",
    "\n",
    "Run cells top to bottom with **Shift + Enter** and fill every **`TODO`** blank (`____`).\n",
    "\n",
    "> **Academic integrity:** individual or pair work. **AI tools are not permitted** on labs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 · Meet the data\n",
    "Each **row** is one URL described by simple **features**; `label` is phishing or legit."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "rng = np.random.default_rng(23)\n",
    "n = 300\n",
    "df = pd.DataFrame({\n",
    "    \"url_length\":  rng.integers(15, 90, n),   # characters in the URL\n",
    "    \"num_dots\":    rng.integers(1, 7, n),     # dots (many subdomains = suspicious)\n",
    "    \"num_hyphens\": rng.integers(0, 6, n),     # hyphens (look-alike domains)\n",
    "    \"has_at\":      rng.integers(0, 2, n),     # an @ symbol in the URL (suspicious)\n",
    "    \"has_https\":   rng.integers(0, 2, n),     # uses https (slightly reassuring)\n",
    "    \"is_ip\":       rng.integers(0, 2, n),     # raw IP address instead of a name\n",
    "})\n",
    "score = (0.04 * df.url_length + 0.5 * df.num_dots + 0.5 * df.num_hyphens\n",
    "         + 1.6 * df.has_at + 1.3 * df.is_ip - 1.2 * df.has_https + rng.normal(0, 1.2, n))\n",
    "df[\"label\"] = np.where(score > 7.0, \"phishing\", \"legit\")\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# TODO 1: how many URLs are phishing vs legit?\n",
    "# Hint: df[\"label\"].value_counts()\n",
    "print(____)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 · Features, label, and a train/test split"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X = df[[\"url_length\",\"num_dots\",\"num_hyphens\",\"has_at\",\"has_https\",\"is_ip\"]]\n",
    "y = df[\"label\"]\n",
    "# TODO 2: hold back 30% to test the detector honestly.\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "        X, y, test_size=____, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 · The detector (train it, then judge it)\n",
    "We use logistic regression - it outputs a probability we can threshold later."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "detector = LogisticRegression(max_iter=1000)\n",
    "# TODO 3: train the detector on the TRAINING data.\n",
    "detector.____(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4 · Accuracy (a first look)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "preds = detector.predict(X_test)\n",
    "# TODO 4: accuracy = compare true labels to predictions.\n",
    "acc = accuracy_score(____, ____)\n",
    "print(\"accuracy:\", round(acc, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5 · Confusion matrix (what KIND of mistakes?)\n",
    "Layout: rows = actual, columns = predicted. Off-diagonal = errors."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "# TODO 5: build the confusion matrix (true labels, predictions).\n",
    "print(confusion_matrix(____, ____, labels=[\"legit\",\"phishing\"]))\n",
    "# rows/cols order = [legit, phishing]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6 · Precision and recall (focus on phishing)"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "# TODO 6: set pos_label so the metrics focus on the \"phishing\" class.\n",
    "p = precision_score(y_test, preds, pos_label=____)\n",
    "r = recall_score(y_test, preds, pos_label=____)\n",
    "print(\"precision:\", round(p, 3), \" recall:\", round(r, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 7 · Slide the alert threshold\n",
    "The detector outputs a probability of *phishing*. We choose the cutoff."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "prob = detector.predict_proba(X_test)[:, list(detector.classes_).index(\"phishing\")]\n",
    "\n",
    "def errors_at(t):\n",
    "    flagged = np.where(prob >= t, \"phishing\", \"legit\")\n",
    "    fp = int(((flagged == \"phishing\") & (y_test.values == \"legit\")).sum())\n",
    "    fn = int(((flagged == \"legit\") & (y_test.values == \"phishing\")).sum())\n",
    "    return fp, fn\n",
    "\n",
    "# TODO 7: compare a stricter and a looser threshold to the default 0.5.\n",
    "for t in [0.3, 0.5, ____]:\n",
    "    fp, fn = errors_at(t)\n",
    "    print(f\"threshold={t}: false_alarms={fp}  missed_phish={fn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 8 · Reflection questions\n",
    "Double-click and replace each *your answer here*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Your answers**\n",
    "\n",
    "1. **How many URLs total, and how many phishing vs legit?** *your answer here*\n",
    "2. **What was the accuracy, and given the class balance, do you fully trust it? Why?** *your answer here*\n",
    "3. **From the confusion matrix, how many false alarms and how many missed phishing URLs at the default threshold?** *your answer here*\n",
    "4. **As you RAISED the threshold, what happened to false alarms vs missed phishing? Explain the trade-off.** *your answer here*\n",
    "5. **You run this detector for a bank. Would you pick a lower or higher threshold, and why? Which error is worse here?** *your answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ✅ Before you submit\n",
    "- [ ] **Runtime -> Run all** with no errors.  - [ ] All `TODO` blanks filled.\n",
    "- [ ] All 5 reflection questions answered.  - [ ] **File -> Download -> Download .ipynb**, upload to Canvas.\n",
    "\n",
    "## Troubleshooting\n",
    "- `NameError` -> you skipped a cell; run from the top with **Run all**.\n",
    "- `ValueError: pos_label` -> make sure `pos_label='phishing'` matches the label text exactly.\n",
    "- Predictions look reversed -> check you compared `y_test` to `preds` (not the other way) and used the test set.\n",
    "\n",
    "## Extension (optional, ungraded)\n",
    "Add a feature of your own (e.g., `num_digits`), retrain, and see whether precision/recall improve."
   ]
  }
 ]
}