Week 11: Prompt Injection and LLM Security

Dates: Mar 22-26  ·  Reading: Handout 9: LLM Security and Prompt Injection

Everyone is deploying chatbots and AI assistants - and they carry a brand-new, top-ranked weakness: prompt injection. We study how large language models (LLMs) are attacked so we can build safer LLM applications. Monday: how LLMs actually work, then prompt injection - direct, indirect, jailbreaks, and prompt leakage - and why it works. Wednesday: defensive prompting, guardrails, whole-app hardening, and a beginner Capture the Flag (CTF) mini-event. Lab 9 (CTF) is due this week. No quiz this week.

📄 Read Handout 9 (PDF)

Learning Objectives

• Explain how large language models (LLMs) work and their role in business
• Describe prompt injection attacks: direct, indirect, jailbreaks, and prompt leakage
• Identify defensive prompting techniques, guardrails, and least privilege
• Understand LLM security best practices (OWASP Top 10 for LLM Applications)

Lecture Slides

⬇ Monday: Prompt Injection & LLM Security (PPTX) ⬇ Wednesday: Defensive Prompting & Hardening (PPTX)

Two full 75-minute decks, CSU-branded, with diagrams of how an LLM works, direct and indirect injection, an attack taxonomy, a layered-defense stack, a hardened-app architecture, naive-vs-guardrailed code, activities, and detailed speaker notes on every slide.

Monday Session

Demystifying LLMs: they predict the next word - they do not understand or verify, and they follow instructions in their input. Their business role and why access raises risk. Then prompt injection (OWASP LLM risk #1): direct injection, indirect injection hidden in documents the model reads, jailbreaks, and prompt leakage - plus the root cause: no wall between trusted instructions and untrusted data.

Wednesday Session

Defending LLM apps in depth: a strong system prompt, input and output guardrails, keeping data separate from instructions, least privilege (limit the blast radius), and monitoring with human oversight. We finish with a beginner CTF mini-event on sanctioned practice platforms - practicing the attacker mindset that makes a better defender.

Lab

Lab 9: CTF Challenge. A beginner-friendly Capture the Flag on Hacker101 or the course CTFd instance (authorized platforms only). Solve a few challenges, record how you found each flag, and map each one to a defense from this week. Graded on participation and reflection. Purely educational - no attacks on external systems.

⬇ Lab 9 Guide & Worksheet (PDF)

Complete the worksheet (flag log + 5 reflections) and submit in Canvas. AI tools are not permitted.

Quiz / This Week

No quiz this week. Lab 9 (CTF) is due Wednesday. Quiz 9 is next week (Week 12).

In-Class Activity

Prompt Injection: Break and Defend an LLM Assistant — CTFd capture-the-flag challenge (Colab + browser). Graded, 20 points.

⬇ Activity slides (PPTX) 📄 Student worksheet (PDF)

🚩 CTF challenge starter (.ipynb)

Graded in-class activity with a worksheet. Full rotation on the Activities page.

---

← All lecture notes  ·  Assessments unlock on the Quizzes and Labs pages when the instructor releases them.