Threat Modeling LLMs: Adam’s talk at Black Hat USA
Kymberlee Price, Shostack + Associates COO
PHANTOM-B is a practical tool built for threat modeling AI systems
This week at Black Hat USA, Adam gave a talk called "Threat Modeling LLMs: The PHANTOM-B Approach." It ran in three parts: a quick level-set on threat modeling, a broader look at threat modeling LLMs, and the PHANTOM-B approach we've built to make that process fast and focused. Here's the recap.
Context: why bother?
The talk opened with a level-set, because the words "threat modeling" are used in so many different ways by so many different people. At its core, it's using a consistent method to anticipate problems. It applies to the software you write, the software you assemble from other people's parts, and the LLMs you train or pull down from Hugging Face. All of it benefits from asking the questions: what are we working on, what can go wrong, what are we going to do about it, and did we do a good job. The Four Question Framework is now standard across industry and government alike, from Anthropic and Google to MITRE and the FDA.
Then came the underlying reason for the current AI race: executives are scared of missing the AI train the way old, once-dominant tech companies got left behind. That fear, whether or not the underlying AI-bubble debate ever resolves, is what's driving the pressure to ship. That's what makes threat modeling essential right now: of everything being reshuffled by AI, it's the security technique holding up best under the disruption, because the questions in the Four Question Framework don't care what's on the other end of the pipeline.
Threat modeling LLMs, broadly
From there the talk clarified that we use AI in four scenarios : offense, defense, software development, and business. Not knowing which we’re talking about leads to confusion. The existing landscape of answers to "what can go wrong" here is crowded: Berryville's ML and LLM risk analyses, OWASP's Top 10 for LLMs and AI Exchange, MITRE ATLAS, NIST's AI/ML catalog, Google's SAIF. All more organized than "we'll red team it and ship it," but each with real friction in practice that collapse the return on investment:
- High training cost.
- Findings that are academic, duplicative of standard security engineering, or simply can't be acted on.
- Teams working through a catalog and rediscovering things STRIDE already covers.
Those gaps spawned PHANTOM-B.
The PHANTOM-B approach
PHANTOM-B isn’t an effort to add one more catalog to the pile, but a response to what clients kept asking for: AI-focused approaches that's faster to teach and actually gets used. It was built iteratively, first on our own projects, then with hyperscalers, global banks, and other clients, and earlier versions didn't clear our own bar for release. Inspired by STRIDE, it's free under a Creative Commons license and designed to fit on a wallet card.
PHANTOM-B names eight things: prompt injection, hallucination, anthropomorphization, non-explainability, training issues, overreliance, missing security engineering, and bias. All of them are explained in [the whitepaper] but a few of them got bigger reactions from the crowd. Prompt injection isn't really about funny jailbreak stories. Instead, it's an attacker getting the LLM to do their bidding, and with today's systems, that bidding can do quite a lot. Anthropomorphization is what happens when we treat a token generator like it has a model of right and wrong, or of user versus attacker. It doesn't, and instructions phrased as "don't do X" tend to backfire because of it. And non-explainability means the LLM's account of its own mistake is a plausible-sounding story, not the actual reason. Remember that the next time an LLM tells you why it missed something.
The framework is deliberately narrow by design. We stripped out anything your standard security engineering should already catch, so PHANTOM-B complements STRIDE, kill chains, and SDLs rather than duplicating them.
Why this, why now?
Underneath the framework is the same point the talk opened with: the pressure to ship AI isn't going away, and it isn't a reason to skip the questions that keep systems trustworthy. If anything, it's the reason to ask them faster and more consistently than before. Threat modeling doesn't slow delivery down but it does let teams move quickly without flying blind into the kind of headline nobody wants to be in.
Thanks to everyone who came to the microphones with questions, and to the clients and collaborators who spent real time helping shape PHANTOM-B into something worth putting our name on.
The whitepaper is here, you can download the Black Hat slides here. Adam will be reprising the PHANTOM-B talk at 12:30pm on Saturday at the DEF CON AppSec Village Main Stage.
Speaking of PHANTOM-B, our partners at Cybersec Games are now selling 10-packs of the PHANTOM-B Wallet cards, joining the classic Threat Modeling Wallet Cards to support your team's threat modeling.
PHANTOM-B Wallet Cards give your team a simple physical reference for design reviews, architecture discussions, sprint meetings or quick conversations at the whiteboard. Pre-order yours before your next design review!
Photo by Kymberlee Price