{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.x"
  },
  "colab": {
   "provenance": []
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Were You in the Data?  (Membership Inference)\n",
    "### CIS 350: AI for Cybersecurity - CTF Challenge\n",
    "\n",
    "**Category:** Privacy   **Points:** 20   **Difficulty:** Hard\n",
    "**Runs in:** Google Colab or Jupyter (numpy, pandas, scikit-learn, matplotlib - all preinstalled in Colab).\n",
    "\n",
    "## Ethics and scope (read first)\n",
    "This is an AUTHORIZED, sandboxed, defensive exercise. The 'model' is a TOY classifier trained on SYNTHETIC data inside this notebook - never a real system, dataset, or person. Techniques here are for learning how to DEFEND privacy; do not apply them to systems you do not own or have permission to test.\n",
    "\n",
    "## Scenario\n",
    "A company published a machine-learning model trained on its private customer records. They swear the model 'does not reveal who was in the data.' You are the privacy auditor. A model that is a little OVERFIT is more confident on rows it was trained on (**members**) than on rows it has never seen (**non-members**). That confidence gap is a leak.\n",
    "\n",
    "## Objective\n",
    "Build a **threshold membership-inference attack**: pick a confidence threshold `t` and predict a row is a **member** when `confidence(row) >= t`.\n",
    "\n",
    "## Success condition (stated plainly)\n",
    "Your attack is scored on a labeled mixed set (half members, half non-members). **You win when your attack accuracy is greater than 0.60** - clearly better than a 0.50 coin flip. The provided `check(t)` prints the flag only when that condition is genuinely met.\n",
    "\n",
    "## What to submit\n",
    "This completed `.ipynb` with the `check(t)` cell showing your threshold and the printed flag line.\n",
    "\n",
    "Run cells top to bottom with **Shift + Enter**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Set up the toy model (provided, do not edit)\n",
    "This builds the synthetic data, trains the slightly-overfit model, and defines `confidence(row)`, the labeled `eval_rows` / `eval_labels`, and calibration splits."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")   # keep output clean\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "SEED = 42\n",
    "np.random.seed(SEED)\n",
    "\n",
    "# ---- Toy synthetic \"customer\" dataset (NOT real data) ----\n",
    "X, y = make_classification(\n",
    "    n_samples=800, n_features=20, n_informative=6, n_redundant=2,\n",
    "    n_classes=2, flip_y=0.15, class_sep=0.7, random_state=SEED)\n",
    "\n",
    "# Half the rows TRAIN the model  -> these are the \"members\".\n",
    "# The other half are held out, never seen -> the \"non_members\".\n",
    "X_members, X_nonmembers, y_members, y_nonmembers = train_test_split(\n",
    "    X, y, test_size=0.5, random_state=SEED)\n",
    "\n",
    "# A slightly OVERFIT model: deep-ish trees on limited data memorize members,\n",
    "# so it is more confident on rows it has already seen.\n",
    "model = RandomForestClassifier(\n",
    "    n_estimators=60, max_depth=12, min_samples_leaf=1,\n",
    "    random_state=SEED).fit(X_members, y_members)\n",
    "\n",
    "def confidence(row):\n",
    "    \"\"\"Model confidence for one row = max class probability (0.5 - 1.0).\"\"\"\n",
    "    return float(model.predict_proba([row])[0].max())\n",
    "\n",
    "# ---- Labeled MIXED evaluation set: half members (1), half non-members (0) ----\n",
    "N_EVAL = 150\n",
    "eval_rows = np.vstack([X_members[:N_EVAL], X_nonmembers[:N_EVAL]])\n",
    "eval_labels = np.array([1] * N_EVAL + [0] * N_EVAL)   # 1 = member, 0 = non-member\n",
    "\n",
    "# ---- Calibration split (SAFE to inspect) to help you pick a threshold ----\n",
    "cal_members = X_members[N_EVAL:N_EVAL + 80]\n",
    "cal_nonmembers = X_nonmembers[N_EVAL:N_EVAL + 80]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - The check function (provided, do not edit)\n",
    "`check(t)` runs your threshold attack on the labeled eval set and reveals the flag only if accuracy beats 0.60."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "def check(threshold):\n",
    "    \"\"\"Threshold membership-inference attack (self-grader).\n",
    "    Rule: predict \"member\" when confidence(row) >= threshold.\n",
    "    Reveals the flag ONLY if attack accuracy on the labeled eval set beats 0.60\n",
    "    (meaningfully better than a 0.50 coin flip).\"\"\"\n",
    "    threshold = float(threshold)\n",
    "    preds = np.array([1 if confidence(r) >= threshold else 0 for r in eval_rows])\n",
    "    acc = float((preds == eval_labels).mean())\n",
    "    print(\"threshold =\", round(threshold, 4), \" attack accuracy =\", round(acc, 4))\n",
    "    if acc > 0.60:\n",
    "        print(\"SUCCESS! Flag: CIS350{c0nf1dence_gap_leaks_membership}\")\n",
    "    else:\n",
    "        print(\"Not there yet - accuracy must beat 0.60 (better than a coin flip).\")\n",
    "        print(\"Hint: MEMBERS tend to have HIGHER confidence than non-members.\")\n",
    "        print(\"Compare the two distributions and pick a threshold BETWEEN their means.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 - Look at the confidence gap\n",
    "Members were memorized, so their confidence tends to be HIGHER. Confirm the gap before attacking."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "mem_conf = np.array([confidence(r) for r in cal_members])\n",
    "non_conf = np.array([confidence(r) for r in cal_nonmembers])\n",
    "print(\"members    -> mean confidence:\", round(mem_conf.mean(), 3))\n",
    "print(\"non-members-> mean confidence:\", round(non_conf.mean(), 3))\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "plt.hist(mem_conf, bins=15, alpha=0.6, label=\"members\")\n",
    "plt.hist(non_conf, bins=15, alpha=0.6, label=\"non-members\")\n",
    "plt.xlabel(\"confidence (max predict_proba)\"); plt.ylabel(\"count\")\n",
    "plt.legend(); plt.title(\"Confidence: members vs non-members\"); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4 - TODO: build your threshold attack\n",
    "Use the two distributions above (and the calibration split) to choose a threshold `t` that separates members from non-members, then call `check(t)`.\n",
    "\n",
    "Scaffolding:\n",
    "1. Members have HIGHER confidence than non-members.\n",
    "2. A good `t` sits BETWEEN the two means (a decent first guess is their midpoint).\n",
    "3. `t` must be a number between 0.5 and 1.0.\n",
    "4. Call `check(t)` - if accuracy beats 0.60 it prints the flag."
   ]
  },
  {
   "cell_type": "code",
   "metadata": {},
   "execution_count": null,
   "outputs": [],
   "source": [
    "# TODO: replace ____ with your chosen confidence threshold (0.5 to 1.0).\n",
    "# Tip: try the midpoint of the two means you printed in Part 3, then adjust.\n",
    "t = ____\n",
    "check(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5 - Reflect (short answers)\n",
    "Double-click and replace each *your answer here*.\n",
    "\n",
    "1. **Why is the overfit model more confident on members than non-members?** *your answer here*\n",
    "2. **What single number is your whole attack based on, and why is that a privacy risk?** *your answer here*\n",
    "3. **Name one defense (hint: Lab 11) that would shrink the confidence gap and lower your attack accuracy.** *your answer here*"
   ]
  }
 ]
}