Enterprise LLM Governance at the Point of Generation

Regulatory Framework

The regulatory perimeter for enterprise LLM deployment is no longer a single statute or a single auditor's checklist. It is a layered set of obligations that intersect at the moment a model produces text. NIST AI RMF 1.0 establishes the GOVERN, MAP, MEASURE, and MANAGE functions, and the Generative AI Profile released in 2024 extends those functions with risks specific to foundation models: confabulation, data leakage, harmful bias amplification, dangerous content generation, and downstream misuse. The Profile's measurement guidance asks deployers to demonstrate not only that risks were identified but that controls operate at the point where risk materializes. For generation, that point is inference time.

ISO/IEC 42001, the AI Management System standard published in late 2023, requires documented controls over the AI system lifecycle, with Annex A controls covering data quality, model behavior, and operational oversight. Clause 8 operational planning requires that risk treatments be implemented as controls that can be evidenced. A statistical filter that occasionally permits prohibited content cannot be evidenced as an effective Annex A control because its failure mode is non-deterministic.

The EU AI Act creates additional, sharper obligations. Article 26 imposes deployer duties for high-risk systems including human oversight, input data relevance, and monitoring. Article 50 imposes transparency duties for systems that generate or manipulate content, requiring that synthetic outputs be machine-detectable and that deployers prevent prohibited outputs in interactions with natural persons. Article 53 imposes general-purpose AI obligations on providers, including technical documentation of training, evaluation, and risk-mitigation measures. The Code of Practice that operationalizes Article 53 expects providers to demonstrate that systemic risks are mitigated by design rather than by post hoc moderation.

NIST CSF 2.0 adds the GOVERN function as a peer to IDENTIFY, PROTECT, DETECT, RESPOND, and RECOVER, and the PROTECT function's PR.DS data security and PR.PS platform security categories now apply to inference pipelines. SOC 2 Trust Services Criteria CC6 logical access, CC7 system operations, and CC8 change management apply to LLM inference as a system of record when generated outputs are used in customer-facing decisions. FedRAMP moderate and high baselines from NIST SP 800-53 require AC-3 access enforcement, AC-4 information flow enforcement, and SI-10 information input validation, all of which apply to model inputs and outputs in cloud LLM deployments. FFIEC examination guidance for model risk treats LLM outputs as model output subject to validation. HIPAA's minimum necessary standard at 45 CFR 164.502(b) requires that protected health information disclosures be limited to the minimum necessary, a standard that statistical filters cannot deterministically meet.

Architectural Requirement

Translated into engineering terms, these frameworks converge on a single architectural requirement: governance must be enforceable at the moment of output production, not at the moment of output delivery. The distinction is not academic. The ungoverned output, if it exists, has been written to GPU memory, may have traversed inter-process boundaries, may have been logged by an inference observability stack, may have been cached for prompt-caching efficiency, and may have been replicated across availability zones. Each of those locations is a potential disclosure surface that a post-generation filter cannot retroactively scrub.

The architectural requirement therefore has three components. First, the governance evaluation must occur prior to the commitment of the candidate output to any persistent or observable state. Second, the evaluation must be deterministic with respect to a documented policy so that audit evidence can be produced. Third, the evaluation must be contextual, parameterized by the agent's identity, trust scope, current task, and the data sensitivity of the surrounding session, because the same string of tokens may be permissible in one context and prohibited in another under HIPAA minimum necessary, EU AI Act Article 50, or SOC 2 CC6.

This requirement cannot be satisfied by a filter bolted onto the model's output stream because the filter is causally downstream of the generation event. It cannot be satisfied by training-time alignment because training produces statistical tendencies, not guarantees. It can only be satisfied by a control surface that participates in the generation process itself, evaluating each candidate semantic transition before that transition becomes part of the output.

Why Procedural Compliance Fails

The dominant procedural response to LLM governance is a stack of policies, training requirements, prompt-engineering guidelines, and a post-generation safety classifier. This stack fails as a compliance artifact for reasons that are now well-documented in regulator guidance and enforcement actions.

Post-generation filtering produces ungoverned artifacts as a matter of course. The output is generated, exists in memory, is evaluated, and is then either delivered or suppressed. For a HIPAA-covered entity, the existence of a generated string containing protected health information is itself a use of that information, regardless of whether the string was delivered to the requester. For an EU AI Act Article 50 deployer, the production of prohibited synthetic content is a regulated event regardless of whether downstream delivery was suppressed. The procedural pattern treats suppression as equivalent to prevention, and regulators increasingly do not.

RLHF and constitutional AI techniques reduce the frequency of policy violations but cannot supply deterministic guarantees. The training signal is statistical, the alignment is incomplete, and adversarial prompting reliably surfaces ungoverned behaviors. ISO/IEC 42001 Clause 8 and NIST AI RMF MEASURE 2.7 require that controls be evidenced under realistic operating conditions including adversarial inputs. A control that holds 99.5 percent of the time is, for compliance purposes, a control that fails on a known schedule, and that schedule must be documented as residual risk.

Filter-and-model coordination introduces a two-system failure model. A filter outage, a filter version skew, or a misconfigured rule set permits ungoverned outputs to reach users. NIST CSF 2.0 PR.PS-01 requires that platform security configurations be managed and monitored, and a filter whose absence is invisible to the model fails this requirement. SOC 2 CC7.2 requires that anomalies be detected, but a filter that silently allows a class of outputs through cannot be detected by the model that generated them.

Procedural compliance also fails the auditability test. Regulators want to see what was evaluated, against what policy, with what outcome, for each generated output. A post-generation filter produces a binary outcome: blocked or delivered. It does not produce a record of the policy evaluation that justifies the outcome at the granularity of the decision. NIST AI RMF MANAGE 4.1 and ISO/IEC 42001 A.6.2.6 both require traceability of decisions affecting AI system behavior.

What the AQ Primitive Provides

AQ inference control inserts a semantic admissibility gate inside the inference loop. The gate operates on candidate transitions, the unit at which the model proposes the next semantic step, rather than on completed outputs. Each candidate is evaluated against the agent's persistent state, the governance policy bound to that agent, the trust scope of the current session, and the entropy budget allotted to the response. If the candidate would violate any of these, it is not committed. The inference engine explores an alternative transition or terminates the response with a governed fallback that is itself produced through the same gate.

The evaluation is semantic rather than lexical. Token-level filters miss meaning that is composed across tokens and over-block phrases that share surface form with prohibited content but are permissible in context. Semantic admissibility evaluates the transition's meaning against the policy, so a transition that would disclose protected health information is inadmissible whether the disclosure is phrased clinically, colloquially, or in a foreign language.

The evaluation is contextual. The agent carries persistent state including the identities of the current user, the data classifications it is authorized to handle, the obligations it is permitted to make on behalf of the enterprise, and the audit handle of the session. The same model serving a customer service agent and a clinical decision support agent produces different governed outputs because the agents carry different policies and trust scopes. Governance becomes an agent property evaluated at inference time, not a model property frozen at training time.

The entropy-bounded inference component prevents circumvention through information density. A semantic budget caps the total information content of each response, preventing prohibited information from being smuggled through paraphrase, encoding, or compression. This addresses a class of jailbreaks that classifier-based filters consistently miss because the surface form of the smuggled output does not match the classifier's training distribution.

The control surface produces auditable evidence at the granularity regulators are asking for. Every candidate transition, the policy version evaluated against it, the constraints checked, the outcome of the check, and the alternative chosen are recorded in the agent's lineage. This produces an evidentiary trail that satisfies NIST AI RMF MANAGE 4.1 traceability, ISO/IEC 42001 A.6.2.6 documentation, EU AI Act Article 12 record-keeping, and SOC 2 CC7.2 anomaly detection in a single artifact.

Compliance Mapping

Inference control maps directly onto the control language of the major frameworks. Under NIST AI RMF, the GOVERN function's policy enforcement is realized as the admissibility policy bound to each agent; MAP context-of-use is realized as the persistent agent state; MEASURE is realized as the lineage of evaluations and outcomes; MANAGE is realized as the policy update and rollback path. The GenAI Profile risks of confabulation, data leakage, and harmful content map to admissibility predicates that are evaluated at every transition.

Under ISO/IEC 42001, Clause 6 risk treatment is implemented as the policy itself, Clause 8 operational planning is implemented as the inference-time enforcement, Clause 9 performance evaluation is implemented as the lineage analytics, and Annex A controls A.6.2.5 verification of AI system behavior, A.6.2.6 documentation, and A.7.4 input data quality are evidenced by the per-transition record.

Under the EU AI Act, Article 26 deployer obligations of human oversight and monitoring are evidenced by the policy authoring and the lineage. Article 50 transparency duties for synthetic content are realized by admissibility predicates that mark or prevent synthetic content as required. Article 53 GPAI obligations including the Code of Practice expectations for systemic risk mitigation by design are realized by the structural prevention of prohibited outputs at generation time. Article 12 record-keeping is satisfied by the lineage.

Under NIST CSF 2.0, GOVERN GV.PO and GV.OC are evidenced by the policy lifecycle. PROTECT PR.DS-01 and PR.DS-02 data-in-use and data-in-transit protections are realized by the prevention of generation rather than by post hoc redaction. DETECT DE.AE anomaly detection is realized by lineage analytics on transition rejections. SOC 2 CC6.1 logical access is realized by the trust scope check at each transition; CC7.2 system monitoring is realized by lineage; CC8.1 change management is realized by policy versioning. FedRAMP AC-3, AC-4, and SI-10 are realized respectively by trust scope enforcement, transition-level information flow enforcement, and admissibility checks on inputs and outputs. FFIEC model output validation is realized by the lineage's per-output evaluation record. HIPAA minimum necessary at 45 CFR 164.502(b) is realized by predicates that limit PHI disclosure to the minimum necessary for the agent's authorized purpose.

Adoption Pathway

Enterprises adopting inference control begin by wrapping an existing LLM inference pipeline with the semantic state layer. The agent identity, trust scope, and policy bindings are externalized from the prompt and held as persistent agent state. Existing prompt-engineered guardrails are translated into admissibility predicates that operate on transitions rather than on completed outputs. The post-generation filter is retained during the transition period as a defense-in-depth layer but is no longer the primary control of record.

The next phase binds the policy to the regulatory framework the enterprise is evidencing. For a HIPAA-covered entity, the policy expresses minimum necessary in terms of disclosure predicates over PHI categories. For an EU AI Act high-risk deployer, the policy expresses Article 26 oversight obligations and Article 50 transparency duties. For a SOC 2 reporting entity, the policy expresses the relevant Trust Services Criteria as predicates whose evaluation is logged in lineage suitable for auditor review.

The third phase integrates lineage into the existing GRC and audit tooling. The per-transition record becomes the primary evidence artifact for control operation. Existing model risk management processes, which previously consumed sampled output reviews, instead consume the lineage as a continuous control attestation. The shift from sampled to continuous evidence is itself a maturity advance recognized by NIST AI RMF MEASURE 4 and ISO/IEC 42001 Clause 9.

The final phase extends the control surface to multi-agent and tool-using deployments where one agent's output is another agent's input. The admissibility gate evaluates not only the textual output but the action proposed by the agent, including tool calls, external API invocations, and downstream agent dispatches. This closes the governance loop across the full agentic system rather than at the boundary of a single model invocation, satisfying the systemic-risk expectations of EU AI Act Article 53 and the GenAI Profile's downstream-misuse risk category.