{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.x"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 15 CTF - The Aging Malware Detector\n",
    "### CIS 350: AI for Cybersecurity - Robustness / Transfer Learning\n",
    "\n",
    "**Category:** Robustness / Transfer Learning  |  **Points:** 20  |  **Difficulty:** Hard\n",
    "**Ties to:** Week 15 activity - Domain Adaptation with Limited Labels.\n",
    "\n",
    "## Ethics & sandbox (read first)\n",
    "This is an AUTHORIZED, sandboxed EDUCATIONAL exercise: the model and all data are synthetic toys inside this notebook - never a real system, network, credential, or service.\n",
    "\n",
    "## Scenario\n",
    "Your company shipped a malware detector in **2023**. It was trained on 2023 behavioral telemetry and, back then, it was almost perfect. It is now **2025**. Attacker tooling has evolved: benign cloud apps are far chattier on the network, modern malware hides its network footprint and leans on registry/CPU tricks, and the old 'lots of encryption == malware' rule has flipped. The 2023 detector is quietly failing in production - this is **domain shift**.\n",
    "\n",
    "You are handed the deployed **2023 baseline model**, evaluation helpers, and - crucially - only **500 freshly labeled 2025 samples**. Your job is to **adapt** the detector so it works in 2025 again.\n",
    "\n",
    "## Objective\n",
    "Produce an `adapted_model` and call `check(adapted_model)`.\n",
    "\n",
    "## Success condition (stated plainly)\n",
    "`check()` reveals the flag only when BOTH hold on the held-out test sets:\n",
    "1. **`eval_2025(adapted_model) > 0.80`** - you fixed the 2025 accuracy, AND\n",
    "2. **`eval_2023(adapted_model) > 0.70`** - you did NOT wreck 2023 in the process.\n",
    "\n",
    "(Training on the 500 new samples *alone* will ace 2025 but collapse 2023 - that fails condition 2 on purpose, to discourage catastrophic forgetting.)\n",
    "\n",
    "## What you are given\n",
    "- `baseline_2023` - the deployed 2023 detector (already trained).\n",
    "- `X2023_train, y2023_train` - the original 2023 training data.\n",
    "- `X2025_labeled, y2025_labeled` - **500** labeled 2025 samples (your only new labels).\n",
    "- `eval_2023(model)`, `eval_2025(model)` - accuracy on large held-out test sets.\n",
    "- `check(model)` - the grader.\n",
    "\n",
    "## How to run (Google Colab)\n",
    "Upload this notebook (or open it in Colab), then **Runtime -> Run all**. Run cells top to bottom with **Shift + Enter**. Only `numpy` and `scikit-learn` are needed (preinstalled in Colab).\n",
    "\n",
    "## What to submit\n",
    "This completed `.ipynb` (with the SUCCESS line and flag visible in the output) **and** the flag string pasted into the Week 15 CTF box in Canvas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup - build the sandbox (run, do not edit)\n",
    "This cell creates the synthetic 2023/2025 worlds, trains the deployed 2023 detector, and defines the evaluation helpers. Everything is seeded so results are reproducible."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")   # keep output clean\n",
    "import numpy as np\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# ---------------------------------------------------------------------------\n",
    "# SANDBOX: 100% synthetic malware telemetry - fake/toy numbers, no real system.\n",
    "# Five numeric behavioral features per sample; label 0 = benign, 1 = malware.\n",
    "# ---------------------------------------------------------------------------\n",
    "FEATURES = [\"network_calls\", \"file_writes\", \"registry_edits\",\n",
    "            \"cpu_spikes\", \"encrypted_ratio\"]\n",
    "\n",
    "def make_year(year, n, seed):\n",
    "    \"\"\"Deterministically generate n synthetic samples for a given 'year'.\"\"\"\n",
    "    rng = np.random.RandomState(seed)\n",
    "    ne = n // 2\n",
    "    if year == 2023:\n",
    "        benign  = rng.normal([2, 2, 3, 3, 0.2], [1, 1, 1, 1, 0.08], (ne, 5))\n",
    "        malware = rng.normal([6, 6, 4, 4, 0.8], [1, 1, 1, 1, 0.08], (ne, 5))\n",
    "    else:  # 2025: means/scales shifted AND a feature relationship flipped\n",
    "        benign  = rng.normal([7, 7, 2, 2, 0.6], [1.2, 1.2, 1, 1, 0.12], (ne, 5))\n",
    "        malware = rng.normal([2, 2, 7, 7, 0.2], [1.2, 1.2, 1, 1, 0.12], (ne, 5))\n",
    "    X = np.vstack([benign, malware])\n",
    "    y = np.array([0] * ne + [1] * ne)\n",
    "    order = rng.permutation(len(y))\n",
    "    return X[order].astype(float), y[order]\n",
    "\n",
    "# 2023 world: training data + a held-out test set.\n",
    "X2023_train, y2023_train = make_year(2023, 2000, 1)\n",
    "X2023_test,  y2023_test  = make_year(2023, 2000, 2)\n",
    "\n",
    "# 2025 world: ONLY 500 labeled samples for adaptation + a large held-out test set.\n",
    "X2025_labeled, y2025_labeled = make_year(2025, 500, 3)\n",
    "X2025_test,    y2025_test    = make_year(2025, 4000, 4)\n",
    "\n",
    "# The deployed 2023 baseline detector (seeded / reproducible).\n",
    "baseline_2023 = RandomForestClassifier(n_estimators=200, random_state=0)\n",
    "baseline_2023.fit(X2023_train, y2023_train)\n",
    "\n",
    "def eval_2023(model):\n",
    "    \"\"\"Accuracy on the held-out 2023 test set.\"\"\"\n",
    "    return accuracy_score(y2023_test, model.predict(X2023_test))\n",
    "\n",
    "def eval_2025(model):\n",
    "    \"\"\"Accuracy on the large held-out 2025 test set.\"\"\"\n",
    "    return accuracy_score(y2025_test, model.predict(X2025_test))\n",
    "\n",
    "TARGET_2025 = 0.80   # you must beat this on 2025 test accuracy\n",
    "FLOOR_2023  = 0.70   # ...without letting 2023 accuracy collapse below this"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "def check(adapted_model):\n",
    "    \"\"\"Grades a candidate adapted model against the success condition.\"\"\"\n",
    "    acc25 = eval_2025(adapted_model)\n",
    "    acc23 = eval_2023(adapted_model)\n",
    "    print(\"2025 test accuracy:\", round(acc25, 3), \" (must be >\", TARGET_2025, \")\")\n",
    "    print(\"2023 test accuracy:\", round(acc23, 3), \" (must stay >\", FLOOR_2023, \")\")\n",
    "    if acc25 > TARGET_2025 and acc23 > FLOOR_2023:\n",
    "        print(\"SUCCESS! Flag: CIS350{d0main_sh1ft_needs_fine_tun1ng}\")\n",
    "    elif acc25 > TARGET_2025:\n",
    "        print(\"HINT: 2025 looks great, but 2023 collapsed - you 'forgot' the old\")\n",
    "        print(\"      domain. Keep the 2023 training data in the mix (do not train\")\n",
    "        print(\"      on the 500 new samples alone).\")\n",
    "    else:\n",
    "        print(\"HINT: 2025 accuracy is still too low. Adapt the detector by refitting\")\n",
    "        print(\"      on the COMBINED 2023 + 500 labeled 2025 samples, then re-run.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - see the problem (domain shift)\n",
    "First confirm the 2023 detector was great in 2023 and is failing in 2025."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "print(\"Baseline 2023 accuracy on 2023 test:\", round(eval_2023(baseline_2023), 3))\n",
    "print(\"Baseline 2023 accuracy on 2025 test:\", round(eval_2025(baseline_2023), 3))\n",
    "print()\n",
    "print(\"The 2023 model is nearly perfect on 2023 data but collapses on 2025 - domain shift.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - your attack: adapt the detector (TODO)\n",
    "Use the **500 labeled 2025 samples** to build an `adapted_model` that clears BOTH thresholds.\n",
    "\n",
    "Scaffolding / hints (you fill in the TODOs):\n",
    "- Domain adaptation with limited labels: do NOT throw away the 2023 data. Combine the original 2023 training set with the 500 new 2025 samples and fit a fresh detector on the union.\n",
    "- Use `np.vstack([...])` to stack the feature matrices and `np.concatenate([...])` for the labels.\n",
    "- Reuse the same model type as the baseline: `RandomForestClassifier(n_estimators=200, random_state=0)` (set the seed so it is reproducible).\n",
    "- Training on `X2025_labeled` *alone* is a trap: great 2025, ruined 2023."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# TODO 1: build the COMBINED training set (2023 training data + 500 labeled 2025).\n",
    "X_combined = np.vstack([____, ____])\n",
    "y_combined = np.concatenate([____, ____])\n",
    "\n",
    "# TODO 2: fit a fresh, seeded detector on the combined data.\n",
    "adapted_model = RandomForestClassifier(n_estimators=200, random_state=0)\n",
    "adapted_model.fit(____, ____)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - grade yourself\n",
    "Run `check(adapted_model)`. When both conditions pass, it prints your flag - paste it into Canvas."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "check(adapted_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - reflection (answer briefly)\n",
    "Double-click and replace each *your answer here*.\n",
    "\n",
    "1. **Why did the 2023 detector fail in 2025 even though it was near-perfect in 2023?** *your answer here*\n",
    "2. **Why does training on the 500 new samples ALONE fail the check?** *your answer here*\n",
    "3. **Name TWO defenses that would catch this domain shift in production before it hurts you.** *your answer here*"
   ]
  }
 ]
}