Course Outline
Module 1 — The Attack Surface of AI Applications
Lab: None — Architecture walkthrough & discussion
Developing a builder’s mental model of the AI attack surface.
Topics:
- LLM, RAG, and agent architectures from a developer’s perspective
- The request/response lifecycle of an AI feature
- Prompt flow: system, developer, user, and tool messages
- Points where untrusted data enters (or re-enters) the model
- Trust boundaries owned by developers versus those inherited
- Why AI attacks are semantic rather than syntactic
- Mapping the OWASP LLM Top 10 to actual code implementation
Key Insight: Any point where untrusted text reaches the model, or where model output reaches your code, is a boundary under your direct control.
Module 2 — Prompt Injection for Developers
Lab: Lab 01 — 01-Prompt-Injection
The AI equivalent of the “SQL injection moment”—though complete escape is not possible.
Topics:
- Direct vs. indirect prompt injection
- Hidden instructions within documents, web pages, and tool output
- Jailbreaks and role-confusion techniques
- The importance of separating instructions from data
- Defensive prompt design (delimiters, structure, minimal authority)
- Why prevention is partial—designing for containment
Hands-on:
- Attacking your own chatbot
- Bypassing naive filters
- Restructuring prompts to minimize blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The most underestimated bug class among developers.
Topics:
- Model output as untrusted input to the rest of the application
- Insecure output handling (LLM02): XSS, SSRF, and downstream command/SQL injection
- Avoiding eval/exec/rendering of raw model output
- Structured outputs and schema validation
- Output encoding and allowlisting
- Safe rendering in web/UI contexts
Hands-on:
- Identifying and fixing insecure-output-handling vulnerabilities
- Enforcing JSON schemas on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
A major new attack surface that developers are responsible for securing.
Topics:
- Vector DB and retrieval threats
- Ingestion sanitization
- Document provenance and trust scoring
- Retrieval scoping and metadata isolation
- Hidden instructions in retrieved content (indirect injection)
- Data exfiltration via retrieval
Hands-on: - Poison a RAG pipeline with a malicious document - Add ingestion sanitization and retrieval scoping for defense
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
Where bugs translate into actions.
Topics:
- Excessive agency (LLM06) and tool abuse
- Least privilege for agents
- Tool allowlists and argument validation
- Approval gates and human-in-the-loop mechanisms
- Sandboxing tool execution
- Scoped, short-lived credentials for agents
- Limiting autonomous loops and chaining
Hands-on:
- Locking down an over-permissioned agent
- Adding allowlists and approval gates to dangerous tools
Module 6 — Secrets, Identity & Cost
Lab: Lab 05 — 05-Secrets-and-Cost
The operational errors that cause the most immediate damage.
Topics:
- API key and secret management (excluding them from prompts, code, and logs)
- Per-user authentication and authorization for AI features
- Propagating user identity to tools and retrieval
- Denial-of-wallet: unbounded token/cost consumption
- Rate limits, token budgets, and timeouts
- Logging without leaking secrets or PII
Hands-on:
- Removing secrets from the prompt/code path
- Implementing per-user rate limits and token/cost budgets
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Deciding between buying or building for input/output safety.
Topics:
- Capabilities and limitations of guardrail frameworks
- Input guardrails: injection/PII/topic classifiers
- Output guardrails: validation, filtering, and grounding checks
- When to use guardrails versus custom deterministic checks
- Layering guardrails with previous module controls
- Performance, false positives, and failure modes
Hands-on:
- Adding an input/output guardrail layer to an AI feature
- Measuring detection effectiveness and misses
Module 8 — Red-Teaming Your Application
Lab: Lab 07 — 07-Red-Teaming
Shipping as if an attacker already has the code.
Topics:
- Building abuse/test suites for AI features
- Automated prompt injection and jailbreak tests
- Regression-testing guardrails and policies
- Integrating AI security checks into CI
- Model and dependency supply chain (provenance, pinning)
- Pre-release security checklists for AI features
Hands-on:
- Writing automated red-team tests for an AI feature
- Integrating these tests into CI checks
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: None — Scoring exercise (uses the Capstone app)
Converting built features into a repeatable security score.
Topics:
- The AI Security Hexagon: six questions replacing “is it secure?”
- Six scored categories (Data, Prompt, Agent, Supply Chain, Detection, Governance)
- The 100-point rubric and its weightings
- Verdict bands and single-category override rules
- The Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- Scoring before and after hardening as a metric
Hands-on:
- Scoring the Capstone app on the 100-point scale
- Identifying the single change that most improves the score
Key Insight: The three highest-weighted categories align with trust boundaries developers own—ensuring the score measures exactly what this course teaches.
Capstone
Students harden a deliberately vulnerable AI application end-to-end.
The starter app includes:
- An injectable prompt
- Insecure output handling
- An unscoped RAG pipeline
- An over-permissioned agent
- Secrets in the prompt path
- No cost limits
Students apply course concepts to:
- Restructure prompts for containment
- Validate and encode model output
- Sanitize and scope retrieval
- Apply least privilege and approval gates to agents
- Remove secrets and add cost/rate limits
- Add guardrails and automated red-team tests
Deliverable: A hardened app plus a concise OWASP LLM Top 10 self-assessment.
Module - Lab Map
Labs follow module order. The course comprises 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion and Module 9 is a scoring exercise, so neither includes a dedicated lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (The Attack Surface of AI Applications) has no lab—it functions as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder—it serves as a scoring exercise against the Capstone app.
Requirements
- Intermediate skill level.
- Students should be proficient in: building and consuming REST APIs, a scripting language (labs utilize Python), basic application authentication, git, and the CLI.
- No machine learning background is required—this is an application security course for those building with LLMs, not training them.
Audience
- Software / backend engineers developing LLM features
- Full-stack and API developers
- AI/ML application engineers
- Platform engineers deploying copilots and agents
- Tech leads and senior engineers responsible for AI features
Testimonials (2)
I really enjoyed learning about AI attacks and the tools out there to begin practicing and actively using for security testing. I took a lot of knowledge away which I didn't have at the beginning and the course met what I hoped it would be. My favorite part shown from the training was Comet Browser and was amazed at what it could do. Definitely something will be looking into more. Overall it was a great course and enjoyed learning all OWASP GenAI Top 10.
Patrick Collins - Optum
Course - OWASP GenAI Security
The profesional knolage and the way how he presented it before us
