One boundary, and what crosses it.
Which parts of your data reach the model, where they stop, and the command that proves each claim. What this will not tell you is that nothing reaches the model — that would not be true, and an unfalsifiable promise is worth less than a narrow one you can test.
The reviewer's browser
their laptopThe question
A batch and a market, as a sentence.
The dossier
Streamed back as it is assembled.
Your network
your servers, your databasesThe guard
Fail-closed on an API key. No key configured means refuse.
The agent loop
Holds the conversation. The only thing that talks to the model.
Six systems of record
ERP · MES · QMS · HCM · REG · TMS. Reads only.
The passage index
Your procedures, chunked and embedded, in your Postgres.
The question log
What was asked and answered, for the reviewer to return to.
The answer contract
Validates shape, then coherence, before anything is shown.
Azure OpenAI
swedencentral · one host, one regiongpt-5-mini
Decides which tool to call, then writes the answer.
text-embedding-3-small
Indexing only, never during a question. Replaceable with a local model.
One question, every hop.
A real run against LOT-IBU200-2609-B, in the order it happened, with what actually moved at each step. Tool and argument names are the ones in the source; the timings are from the recorded run.
5 turns means 5 requests to the model — countable, and checkable against the cost line the desk prints under every answer.
Before any model is involved
A request that is not allowed never reaches a database, let alone a model.
- 1
The reviewer asksbrowser
POST /api/ask { "question": "Can LOT-IBU200-2609-B be released to EU?", "loop": "sdk" }Sent as POST rather than in a URL: a query string is kept by proxies and access logs, and there would be nowhere to carry the key.
- 2
The guard runs firstyour network
authorize({ configuredKey, presentedKey, isDev })Fail-closed. No key configured means refuse, not allow. Nothing is parsed and nothing is connected to until this passes.
Turn 1 — what should I do?
The model is given the question and the tools it may call. No data from your systems yet.
- 3
The question and the tool definitions go outcrosses to the model
messages: [ system prompt, the question ] tools: [ assess_release, search_procedures ] store: false
The prompt is an ordered list of rules; the tools are names, descriptions and argument shapes. No batch data has been read at this point, so none can be sent.
- 4
It asks for the assessmentfrom the model
assess_release({ lot_id: "LOT-IBU200-2609-B", market: "EU" })The model chooses the tool and the arguments. It cannot reach a database; it can only name a tool that the loop then runs.
The walk — entirely inside your network
This is where almost all the data is. Almost none of it leaves.
- 5
Twenty-nine reads across six databasesyour network9.6s
mrd_erp product_lots · material_lots · market_authorisations · spec_limits mrd_mes batch_records · equipment · qualifications mrd_qms qc_results · batch_dispositions · deviations mrd_hcm employees · training_records · signature_authority mrd_reg sop_revisions · standard_clauses · sop_clause_links mrd_tms shipments · telematics_readings select … only — asserted by pnpm pharma:sql-check
Six separate databases on separate connections that no single query can join, which is why this is code and not SQL. Every one of these reads happens inside your network.
- 6
Rows become findings — the reduction that mattersyour network
{ "lotId": "LOT-IBU200-2609-B", "market": "EU", "product": "Ibuprofen 200 mg film-coated tablets", "governingSpecVersion": "SPEC-IBU200-v4", "basis": { "assessedOn": "2026-09-12", "builtToSpecVersion": "SPEC-IBU200-v4", "certification": { "dispositionId": "DISP-26-0001", "decidedOn": "2026-09-04", "decidedByRef": "EMP-0103", "certifier": "Eva Vos", "citedSopRevision": "SOP-QC-014 Rev 7", "governingSopRevision": "SOP-QC-014 Rev 7", "qpCertified": true } }, "findings": [{ "code": "CERTIFIER_TRAINING_LAPSED", "severity": "blocker", "summary": "Eva Vos (Qualified Person) signed on 2026-09-04 with TRN-GMP-REF expired since 2026-08-24.", "evidence": [ { "ref": "mrd_hcm.training_records#(EMP-0103, TRN-GMP-REF)", "asOf": "2026-09-04" }, { "ref": "mrd_qms.batch_dispositions#DISP-26-0001", "asOf": "2026-09-04" }, { "ref": "mrd_reg.sop_revisions#SOP-QC-014 Rev 7", "asOf": "2026-09-04" } ] }], "missing": [], "releasable": false }This is the whole object, and it is worth reading for what is ABSENT. Twenty-nine queries went in; what comes out is one finding, one sentence of summary, and three references. No table contents. No employee record beyond the id and the name that appears in the finding. No connection string, no schema, no other lot. A reference like mrd_hcm.training_records#(EMP-0103, TRN-GMP-REF) is an address you can look up yourself — it is not the row.
Turn 2 — reading the rule
The findings cross. The rows behind them do not.
- 7
That object — byte for byte — is what goes outcrosses to the model
role: "tool", name: "assess_release" content: <the JSON above, verbatim>
Nothing is added to it and nothing is expanded. What you read in the previous step is the complete message the model receives about your batch: 1 finding and 3 references out of 29 queries. If a fact is not in that object, the model has no way to know it.
- 8
It asks for the written procedurefrom the model
search_procedures({ query: "training required before certification…", sop_id: null, as_of: "2026-09-04", k: null })as_of is the date of the act being judged, not today. The revision that governs an act is the one in force when it happened — answering from the current revision is the easiest way to be confidently wrong.
- 9
Hybrid search runs against your indexyour network2.1s
5 of 50 passages — top hit: SOP-QC-014 Rev 7 > 7. Disposition and certification > 7.3 Personnel precondition to certification "…the certifying Qualified Person must hold a valid, unexpired GMP refresher training record on the date of certification. A certification made by a Qualified Person whose GMP refresher training had expired on that date is invalid. The batch does not become released…"
Meaning and keyword together, over vectors in your own Postgres. Note the revision: Rev 6 §7.3 is "Records" — a filing requirement. Rev 7 §7.3 is this rule. Searching as of the date of the act is what gets the second one. The other 45 passages are not sent.
Turn 3 — writing the answer
The last crossing. Everything after this is yours again.
- 10
The matched passages go outcrosses to the model
role: "tool", name: "search_procedures" content: [ 5 passages, each with its clause trail and text ]
Now the model has the facts and the rule, and can quote the clause rather than assert it from memory.
- 11
The answer comes back as structured datafrom the model
{ "lot_id": "LOT-IBU200-2609-B", "market": "EU", "summary": "On 2026-09-04 the Qualified Person Eva Vos signed the disposition for LOT-IBU200-2609-B, but her GMP refresher training record had expired on 2026-08-24. …", "blockers": [{ "code": "CERTIFIER_TRAINING_LAPSED", "in_short": "Certifying QP's GMP refresher had expired on certification date", "why_it_blocks": "SOP-QC-014 Rev 7 §7.3 requires the certifying Qualified Person to hold a valid, unexpired GMP refresher on the date of certification; a certification made by a QP whose refresher had expired is explicitly declared invalid …", "citations": [{ "ref": "mrd_hcm.training_records#(EMP-0103, TRN-GMP-REF)", "as_of": "2026-09-04", "claim": "EMP-0103's GMP refresher had expired before the certification date.", "detail": "Training record shows expiry_on 2026-08-24." }, … ] }], "concerns": [], "missing": [], "what_would_clear_it": [ … ], "escalate": { "reason": …, "suggested_owner": "Qualified Person, DEPT-QA" } }Every citation carries the record it came from AND the date it was read as of, because "SOP-QC-014 Rev 7" is a different rule from Rev 6 and a reference without its date cannot be checked by the person who has to countersign. Note what the schema has no field for: there is no verdict, no score, no "releasable" — and it is strict, so one cannot be smuggled in as an extra key.
- 12
Validated before it is shownyour network
shape → coherence → render
JSON Schema checks the shape. A second layer checks sense: a finding with no named human to own it is rejected, because that is the system quietly deciding.
- 13
Recorded, in your databaseyour network
insert into ask_history (kind, question, answer, run, trace)
So the reviewer can come back to it. Written to mrd_kb, which is ours — a write cannot reach the six systems of record because it resolves a different connection entirely.
- 14
The dossier appearsbrowser
event: answer
Tool steps stream as they happen so a ninety-second walk does not look broken. The answer itself is sent once, whole, after it has passed validation.
The same question, split three ways.
A supplier work list can be produced by one agent reading all 23 lots, by 23 agents each reading one, or — for the lots where the procedure genuinely does not say — by two arguing and a third writing up the disagreement. All three are built. They answer the same question against the same contract, so they can be compared rather than asserted.
The desk runs the first one. The walk above is what happens when somebody asks a question on this site. The other two run from the command line, against the same cases — a question typed into the supplier desk never starts twenty-four model calls.
Measured once, over all 23 lots of SUP-04
| One loopthe desk | Fan-outCLI | ||
|---|---|---|---|
| Calls to the model | 3 | 24 | 23 sub-agents and one assembler |
| Output tokens | 10,354 | 17,819 | 72% more, for the same 23 rows |
| Time | 82s | 50s | the fan-out is the faster one |
| Cost | ~$0.008 | $0.0398 | five times dearer |
| Input billed as cached | 87% | 55% | see the caching note below |
Faster and five times dearer — and the cost is not orchestration overhead, it is verbosity. Twenty-three agents each writing one row are each more thorough than one agent writing its twenty-third.
What splitting buys, on the wire
A sub-agent is handed one lot’s evidence and asked for one row. It has no tool that reaches another lot, so the most any single request can expose is a single lot — a narrower crossing than the single loop can offer. The rows come back markedly more worked: every one names its consignee, shipment id and unit count.
And what it costs
Isolation buys depth and costs coherence. Across the 23 rows, four different spellings of the same escalation owner appeared, because no agent can see what the others wrote. Somebody has to reconcile that, and that somebody is the assembler.
The assembler got worse as the list got longer
At four lots it caught the cross-lot pattern that justifies having an assembler at all — multiple deliveries already going to Piedmont Regional Hospital. At twenty-three lots, four lots went to Piedmont and it did not consolidate them. The participant that exists to see across items sees less well when there is more to see, which is the same attention problem the fan-out was meant to solve, relocated rather than removed.
Fanning out destroys prompt caching
The single loop runs at 87% of its input billed as cached. The first debate runs reported 0 of 5,825 input tokens billed as cached, because five calls with five different system prompts share no prefix. When the adjudicator gained a retrieval turn — two calls sharing one prefix — caching reappeared. So: fanning out across different prompts destroys caching; extra turns within one sub-agent restore it. A real economic property of the design, and only visible because per-request cost was already logged.
The debate, and what it is allowed to conclude
Most of a work list is not arguable — whether a lot shipped to a hospital is on file, and two agents disputing it is theatre that costs money like the real thing. What is arguable is what SOP-SCM-004 §7.3 leaves open on purpose, and the argument is about sequence: protect then test, or test then act. For SUP-04 that is 11 of the 23 lots. Each contested lot costs 5 calls, about 30 seconds and ~$0.011 — roughly $0.12 for all eleven, about fifteen times the single loop for the same question. Not fatal; also not free.
The adjudicator cannot issue a verdict. The answer contract has no field that could carry one and the coherence layer rejects the sentence in prose. What it produces is an escalation carrying both cases to the person allowed to decide: the strongest version of each argument, what exactly has to be decided, and what would settle it. A single model asked “recall or not?” returns a confident-sounding middle. Two arguing makes the disagreement visible instead of dissolving it.
An agent asked to cite a source it cannot reach will invent one
The first debate run produced “an initial decision must be made within 72 hours … per company Field Action SOP timeline”. There is no Field Action SOP and no 72-hour rule; SOP-SCM-004 §7.4 says five working days. The prompt had asked the adjudicator to cite the clock and given it no tools and no procedure text, so it produced something plausible. That is a prompt bug, not a model failure — and an invented regulatory deadline is worse than an absent one, because it gets acted on.
Fixed by giving the adjudicator search_procedures and nothing else, and by changing the instruction from “cite the rule” to “if you cannot find it, write that no timescale was found — do not supply a number from general knowledge”. The next run quoted §7.4 verbatim. The advocates still get no tools: two advocates retrieving their own passages is a debate about two different documents.
Not yet guarded. No eval cases cover the fan-out or the debate. They are reachable only from their own commands, so a regression in either is silent — which is why the desk runs the single loop, the one route the eval suite does cover.
Run them yourself — pnpm pharma:fanout SUP-04 · pnpm pharma:debate SUP-04 · pnpm pharma:fanout-check for the offline half. The first two spend real tokens. Figures above are from docs/pharma/NEXT.md §N6 and docs/PROGRESS.md §17.
Never leaves your infrastructure
- The six systems of record
- ERP, MES, QMS, HCM, REG and TMS are queried in your own network. The model never receives a connection string, a schema or a table.
- Every row not in a finding
- An assessment runs tens of queries and sends only what it concluded. The release walk runs 29 and sends one finding with three references; the supplier walk reads 23 lots and sends the ranked list. Rows read and discarded never leave.
- The document corpus
- Procedures live in your database. Only the passages that matched a question are sent, never the library.
- Credentials
- Database URLs and API keys are read server-side. Nothing reaches the browser or the model.
Crosses to the model, and stops there
Azure OpenAI · gpt-5-mini · swedencentral- The question
- As typed. It names a batch and a market.
- Retrieved passages
- The clauses hybrid search returned from your procedure documents — typically five.
- Tool results
- The findings the assessment produced: test results, signatures, training dates, shipment conditions.
- The answer
- Back over the same connection.
Indexing happens once, not per question.
Everything above is one question. This is the separate flow that runs when your documents change — and the only place the embedding model is involved at all.
Indexing
On change, not on question- 1
Documents load from your storestays
Procedures and specifications. 3 documents today.
- 2
Split into passagesstays
By heading, so a clause keeps its context. 50 passages, none orphaned.
- 3
Each passage is embeddedcrosses
text-embedding-3-small — 1536 numbers per passage. The passage text is sent; the numbers come back. This is the only time document text crosses, and it is not during a question.
- 4
Stored as an index you ownstays
Vectors land in your own Postgres beside the text. Rebuildable from the documents at any time.
Run it yourself.
Each claim below is asserted by a check that captures the real outgoing request. Each check also plants a violation and proves it catches it — a test never seen to fail is a test you cannot read a pass from.
The provider is told not to retain anything
pnpm pharma:compliance-checkverified here
store: false on every request. Left unset, the provider default is to retain — so this is asserted on the captured wire request, not trusted from a comment.
No conversation is held on their servers
pnpm pharma:compliance-checkverified here
No server-side conversation state. Each request carries its own context and nothing persists between them at the provider.
Transcripts are not shipped to a third party
pnpm pharma:compliance-checkverified here
The Agents SDK's tracing exporter defaults ON and posts to api.openai.com. It is pinned off, and the check runs with a key present so it proves tracing stays off even when there is something to export with.
Exactly one host is contacted
pnpm pharma:compliance-checkverified here
One endpoint, in one region. Nothing else is reached during a request.
The same holds on the second engine
pnpm pharma:compliance-mastraverified here
A property proved on one library says nothing about another: they build different requests. Both are checked.
Nothing on the answer path can write to your systems
pnpm pharma:sql-checkverified here
Every SQL string reachable from the tools, the agent, the CLI and the evals is scanned and must be a read. The guarantee is source-level and says so in its own output: it proves no write is written, not that the database would refuse one.
The model resource is in the EU
az cognitiveservices account list -o tableverified here
Read from Azure rather than from a diagram: the Foundry account reports swedencentral. The database is Frankfurt. The whole path stays in the EU, and this is the one claim on the list you can check without our code at all.
And what this page will not claim.
Two of these are the provider's word rather than ours, and one is a default that does not favour us. All three are the first questions a real security review reaches, so they are here rather than in an appendix.
Abuse-monitoring retention is ON by default
must be arrangedAzure OpenAI retains prompts and completions for up to 30 days for abuse monitoring, readable by authorised Microsoft reviewers on a flagged case. store: false does not turn it off — different mechanism. Modified Abuse Monitoring has not been applied for here. On a corpus that names unreleased batches that application belongs before the first real run, not after.
The endpoint is reachable from the internet
must be arrangedThe resource has public network access enabled. An Entra token is required, so it is not open — but network isolation is a separate control, and a regulated customer usually wants it: Private Endpoint, public access off. That is a configuration change rather than a code change, and it has not been made here.
“They do not train on your data” is a contract, not a measurement
vendor statementWe can show what our code sends — that is every other claim on this page. We cannot show what the provider then does with it. That one belongs to Microsoft’s terms and your agreement, and it should be quoted from a current source with a date on it, not repeated from memory by an engineer. Repeated from memory is how a review gets an answer that is right today and wrong at renewal.
What is kept, and by whom.
The model provider keeps nothing
store: false on every request, no conversation held on their side, and no transcript exported anywhere. Asserted on the wire, not assumed.
You keep two things, on purpose
A per-request cost line, and the question-and-answer history a reviewer returns to. Both live in your own database, under your retention policy rather than ours.
Embeddings currently run on Azure alongside the language model. They do not have to: the same pipeline runs against a local embedding model with no network call at all, which is the route to take if passages may not leave your own hardware. EMBEDDINGS=local — the index is rebuilt, and nothing else changes.