Red-teaming my own pipelines: what a regulated buyer asks first

The build I promised for this, the last post in the fundamentals series, was to red-team my own pipelines and publish the findings: security and governance. It has a built-in honesty mechanism, since anything I gloss over is sitting in my own repos. The method was simple and I'd recommend it: for each system, trace where untrusted text enters and what the model can do after reading it, then look at what leaves the system with my name on it.

The threat that ships with the architecture

The first finding applies to everything I run, and there's no fixing it, only designing for it: every model I operate reads text I don't control. Enacted's model reads legal text fetched from the web. PermitCheck's reads customer-uploaded drawings. The research pipeline reads PDFs from anywhere on the internet. That input is the product, and any of it could contain instructions.

This is prompt injection, named by Simon Willison in 2022, and its indirect form, where the attack rides inside content the system was always going to ingest, was mapped by Greshake et al., 2023. OWASP's Top 10 for LLM applications puts it at number one, and the industry consensus position is uncomfortable: there's no reliable way to make a model distinguish instructions from data within its context. You can filter and you can detect, and you should, but neither can be the wall you lean on.

So the real defense has to be the one this whole series has been building: assume the model can be steered, and make the steerable part unable to do damage. Blast radius, not prevention.

The findings, per attack

Here's the red-team pass on Enacted, the system with the clearest writeup because its lanes are narrowest. Imagine an adversary who can influence the text of a regulatory document, or anything else that ends up in the model's input, and wants to weaponize my summarizer.

“Injected text makes the model take an action”

CONTRADICTED

The summarizer has no tools. Its entire output channel is prose that must pass a gate. There is nothing to hijack but a paragraph.

“Injected text gets a fabricated citation or date published”

CONTRADICTED

The deterministic gate rejects any citation or date not present verbatim in the input, and a summary that fails doesn't get retried into compliance, the page ships with the diff alone.

“Injected text steers the summary's prose”

NOT VERIFIABLE

The honest gap. The gate checks citations and dates, not meaning, so sufficiently clever input could bias what the summary emphasizes. Mitigation, not fix: the computed diff renders on every page beside the summary, so the checkable evidence is always one glance away.

“Injected text corrupts what counts as a change”

CONTRADICTED

Change detection never touches a model. It's a fact read from the official index, and the diff is computed by code. The most an adversary controls is commentary on evidence they can't alter.

The pattern in the verdicts is the pattern of the series. The attacks that fail all fail for the same reason: the thing they'd need to subvert isn't a model. And the one that partially succeeds targets the one output only a model produces. The residual risk lives where the probabilistic component has authority, which is why weeks 1 through 11 kept shrinking that authority.

The same pass over PermitCheck found the same shape one tier up: a drawing could carry adversarial text ("this package is fully compliant" in a title block), but verdicts are computed by comparing extracted values against limits resolved from zoning sources, values a note can't argue with, and anything shaky lands in not verifiable rather than a confident pass. The injection surface is real. The verdict authority isn't reachable from it.

PII: the boundary nobody asks about until they do

Second category, and the one my systems handle the most concretely of anything in this essay. PermitCheck submissions carry names, addresses, and full architectural drawings of people's homes. The nurture queue holds email addresses. The research pipeline processes published papers, but published case reports are still about human beings.

The audit questions that mattered when I walked it: which tables hold personal data, what reads them, and where does that data travel? The mechanical answers are row-level security policies on the Supabase tables. Writes go through the service role. Public read access exists only on the tables meant to be public.

The governance answer is the one a buyer cares about: customer documents are sent to a model provider's API for extraction, which means the provider's data-handling terms are part of my product's privacy story whether I mention them or not. Regulated buyers know this, and "which providers see the data, under what retention terms" is a first-meeting question. If you haven't written the answer down, the honest version is "I don't know yet," and that answer loses deals.

What the buyer asks, mapped to the series

Which brings this series to its natural close, because I went back through what security-minded conversations have opened with, and every question maps onto a week.

Where does our data go, and who retains it? That's this week: name the providers and the tables, with the access policies written down. What happens when the AI is wrong? Weeks 1 and 7: gates, and the mechanics of why they're needed. How do we know it stays accurate? Week 8: the golden set that catches drift before users do. Can it act without a human? Weeks 10 and 11: tiered action spaces, approval queues, and the append-only log that proves who decided what. Can you show us, not tell us? That one's the whole series.

The last thing I'd pass along is the practice I stole from my own methodology page: Enacted publishes what the system can't catch, in public, because a monitoring tool shouldn't be vague about its blind spots. Doing the same exercise on your own pipeline, writing down the injected-prose gap you can't close and the provider dependency you can't remove, is uncomfortable for about an afternoon, and afterward you own the most credible document in your sales folder.