AI you can defend to a regulator: the human-approval pattern for AI-initiated mutations.

How we let AI suggest mutations to regulated state without ever letting AI execute them. The approval workflow, the "approval-as-identity" trick, the OPA ai_actions policy package, and what we learned shipping the first three governed-AI use cases into production tenants.

"AI" is a feature label. "AI mutating regulated state" is a control problem. The first is exciting; the second is what gets you fined. Modir ships AI features that mutate state — service requests get classified and assigned, suitability narratives get drafted, next-best-actions get queued — but no AI ever mutates regulated state. That separation, simple to state and stubborn to maintain, is the core of what we call "governed AI."

This post walks through the architectural pattern that lets us ship those features and still defend the system to a regulator. The pattern has three parts: the input/output guards (well-trodden territory; we'll keep it brief), the OPA ai_actions policy that gates tool calls, and the human-approval workflow that turns an AI suggestion into a human-executed action.

The temptation

The first AI feature any wealth platform ships is an "assistant" — usually a chat that knows about the client. It is exciting to demo. The product manager asks the obvious next question: "Can the assistant do things?" Could it open a service request? Update the client's preferences? Place an order?

The temptation is to give the AI an API key and let it call the same endpoints the human would. Don't. The day a regulator asks "who initiated this transaction?" you do not want the answer to be "the AI did, on behalf of the user." That is a category mistake — the regulated frame is "the licensed advisor took this action," and the audit log has to support that frame structurally.

The principle

The architectural rule is: AI never mutates regulated state. AI produces suggestions. Humans execute. The implementation looks like this:

1. An AI graph reaches a node where it would, naively, want to mutate state.
2. Instead of mutating, it produces an AIInteraction row with status='pending_approval'. The row contains the proposed mutation, the rationale, the cited sources, and the OPA decision the AI consulted.
3. An aiInteractionWorkflow in Temporal waits for an approval signal (default 24-hour timeout).
4. A human reviewer sees the proposal in their UI, with the rationale and sources visible. They approve or reject.
5. On approval, the actual mutation runs as an ordinary API call — but with the approver's identity, not the AI's. The audit log shows the human took the action; metadata.aiInteractionId points back to the AI source for traceability.

The pattern preserves three properties simultaneously: the AI gets to suggest concretely (not just as text), the human is unambiguously responsible for what happens, and the audit chain reflects reality.

The OPA ai_actions policy

Even before the suggestion lands in front of a human, OPA's ai_actions package decides whether the AI is allowed to propose the mutation in the first place. Some mutations are off-limits to AI entirely (e.g. closing an account, transferring funds). Some are allowed for one role but not another. The policy makes those decisions explicit:

package modir.ai_actions

# Allow proposals for read-only enrichment (no approval needed).
allow if {
  input.use_case == "briefing"
  input.target.kind == "report"
}

allow if {
  input.use_case == "commentary"
  input.target.kind == "report"
}

# Suggestion-only — never auto-execute.
require_approval if {
  input.use_case in {"triage", "next_best_action", "suitability_draft"}
}

# Disallow entirely — AI can't propose certain things.
deny[reason] if {
  input.target.kind == "account"
  input.target.action == "close"
  reason := "AI cannot propose account closure"
}

deny[reason] if {
  input.target.kind == "transfer"
  reason := "AI cannot propose fund transfers"
}

The policy lives in infra/opa/policies/ai_actions.rego alongside the other policies. The compliance team can edit it the same way they edit any other Rego file. Tests in ai_actions_test.rego are part of CI.

The approval-as-identity trick

The subtle part of the design is what happens when the human approves. Naively, you might commit the mutation as a system action that references the human approver. But that produces an audit trail where the actor is "system" and the human is metadata — which is not what the regulator wants to see.

Instead, we run the actual mutation as a normal API call with the approving user's identity. The user's session is the actor. The mutation is exactly the same as if the user had typed it in themselves — same OPA decisions, same OpenFGA relationship checks, same audit event with the user as subject. The AI source becomes metadata: aiInteractionId pointing back to the proposal and its rationale.

The implementation requires care. The approving user must have permission to perform the mutation independently — if they don't, the OPA decision rejects it, exactly the same way it would reject the user's manual attempt. Approval cannot grant new authority; it just turns "AI proposed; human reviewed; mutation performed" into a single coherent transaction with the right subject.

The Temporal workflow

The waiting and timeout logic lives in Temporal:

@workflow.defn
class AIInteractionWorkflow:
    @workflow.run
    async def run(self, interaction_id: str) -> InteractionResult:
        signal = await workflow.wait_condition(
            lambda: self.approval_signal is not None,
            timeout=timedelta(hours=24),
        )
        if not signal:
            return InteractionResult(status="expired")
        if self.approval_signal == "approved":
            return await workflow.execute_activity(
                "commit_as_approver",
                interaction_id,
                approver_id=self.approver_id,
            )
        return InteractionResult(status="rejected")

The workflow holds the proposal alive across deploys, restarts, and operator intervention. If the worker restarts, the interaction picks up exactly where it was. If the timeout expires, the proposal is marked expired (and the audit chain records it). Nothing is lost.

RAG without privilege escalation

The other place AI tries to violate access boundaries is retrieval. A naive RAG implementation indexes all the documents in the tenant and queries them with a user prompt. If the user can't read certain documents — say, another advisor's client notes — the AI can leak them through the RAG response.

Modir's RAG is per-tenant (the index itself is scoped) and per-user-relationship (every chunk is filtered through OPA's data_access package using the requesting user's OpenFGA relationships). A chunk the user can't read in the database, they can't read in the AI either. The cost is one OPA call per retrieved chunk. The benefit is no privilege escalation through retrieval. We pay it gladly.

Cost capping is governance, too

The other dimension of "governed AI" is "AI you can budget for." LiteLLM is the single LLM gateway; per-tenant daily and monthly caps live in Redis. When a cap is hit, AI requests degrade gracefully — cached briefings continue to serve, new generation pauses, and the next day resumes.

Cost capping also limits blast radius. If a misbehaving prompt would otherwise generate millions of tokens against the wrong model, the cap stops it at the fixed dollar threshold. Cost capping is part of the governance posture; we treat unbounded LLM spend as a similar category of risk to unbounded data egress.

Langfuse: the audit trail for AI

Every AI request passes through Langfuse, which captures the full trace span: prompt, retrieved chunks, model, tokens in, tokens out, cost, latency. Traces are tenant-scoped and queryable from the compliance console. When a regulator asks "show me the full provenance of this AI suggestion," the answer is a Langfuse trace with content hashes referencing the audit chain.

The combination — OPA decision logs, Langfuse traces, audit events with aiInteractionId back-pointers — gives us an end-to-end provenance graph for every AI-influenced state change. The human reviewer's identity, the AI suggestion's rationale, the data the AI saw, the policies that gated it, and the mutation that actually happened are all linked.

What we got wrong the first time

Two things, on separate occasions.

The first was letting the AI sign approval payloads as itself. We thought it would be elegant: the AI proposes, signs the proposal with a service identity, the human approves with their identity, and the mutation runs with the human as actor. In practice, the service-identity signature confused our auditors more than it helped. We dropped it. The AI source is metadata; the human is the actor; the proposal is just a row in the database referenced from the audit event.

The second was running approval as a synchronous API call. Some approvers take their time — and 30-second HTTP timeouts are not where you want to wait for a human's decision. Moving to Temporal made the asynchronous nature of approval explicit. Approvals can take 18 minutes, 8 hours, or never; the workflow handles all three cases and always exits cleanly.

Why this is a regulator-favorite design

Regulators care about two questions. First, "who took this action?" Second, "could this action have happened without the licensed party knowing?" The governed-AI pattern answers both crisply: the human took the action; the licensed party knew about it because the licensed party is the one who approved it. The AI was a tool the licensed party used; it was not an independent agent.

The most rewarding moment in our last regulator review was when the auditor asked "what stops the AI from making changes the user didn't approve?" and we walked them through the policy, the workflow, and the audit chain. The auditor said: "I have not seen this in a wealth platform before. Please send me a write-up." This post is, in some sense, that write-up.

Engineering posts represent the views of the authors.