Training-Level Memorization Detection

Mechanism

The mechanism observes per-example signals during training and aggregates them into memorization scores that are evaluated against governance thresholds. Three signal classes are computed in concert. The first is gradient sensitivity: the norm of the gradient produced by an example at a given training step indicates how strongly that example is reshaping parameters, and a sustained pattern of disproportionately large gradients on a specific example is a leading indicator of memorisation. The second is parameter localisation: the system tracks which parameters are being moved by which examples, and an example whose contribution localises to a small parameter subset is more likely to be memorised than distributed across general representations. The third is reconstruction fidelity: at periodic checkpoints the model is queried with prefixes of training examples and the verbatim continuation rate is measured against a held-out reference, providing a direct behavioural signal of memorisation.

The three signals are fused into a per-example memorisation score. Scores are accumulated across training steps and compared against a threshold derived from the example's content class, sensitivity band, and the governance policy in force. When an example crosses the threshold, the system raises a memorisation event carrying the example identifier, the contributing signal magnitudes, the implicated parameter subset, and the training step at which the threshold was crossed. The event is dispatched to an intervention controller that selects a response from a configured menu.

Intervention responses include rate-limiting, in which the example's effective learning rate is reduced for the remainder of training so that further absorption is slowed; localised regularisation, in which an L2 or low-rank penalty is applied to the parameters most strongly modified by the example; example excision, in which the example is removed from the remaining training mixture and its contributions are partially reversed by counter-gradient steps against a synthetic neutralising target; and quarantine, in which the example continues to participate in training but the resulting model is deployed only with output filters configured to detect and suppress reproduction of the example.

The system records every memorisation event and every intervention in a training-time governance log. The log is the input to a post-training memorisation manifest that accompanies the model artifact and lists, for each retained risk, the example class, the residual memorisation score, and the intervention applied.

Operating Parameters

Operating parameters control the sensitivity, scope, and remediation aggressiveness of the detector. The sampling-rate parameter sets the fraction of training steps at which per-example signals are computed; computing on every step gives full coverage at significant compute cost, while sampling at lower rates trades coverage for efficiency. The window-length parameter controls how many recent steps contribute to the accumulated score, balancing responsiveness against false positives from transient gradient spikes.

Threshold parameters are specified per content class and per sensitivity band. Examples drawn from corpora known to contain personally-identifiable information are subject to tighter thresholds than examples drawn from public reference texts; examples drawn from rights-governed corpora are subject to thresholds tuned to the licensing regime under which the corpus is held. The fidelity-probe parameter controls how often the reconstruction-fidelity signal is computed and against which prefix lengths, with longer prefixes producing stronger but more compute-intensive evidence.

Intervention parameters bind threshold breaches to responses. A graduated configuration applies rate-limiting on first breach, regularisation on sustained breach, excision on severe breach, and quarantine when excision is impossible because the example is required for downstream task performance. A strict configuration applies excision on any breach above a hard threshold, accepting the resulting capability cost in exchange for stronger guarantees. The manifest-disclosure parameter determines what residual risk information is published with the model: a maximal disclosure surfaces every retained event, while a minimal disclosure surfaces only events whose residual score remains above a deployment-time threshold.

Alternative Embodiments

One embodiment computes signals on every example at every step and accepts the overhead in exchange for full-resolution detection. A second embodiment computes signals on a stratified sample weighted toward high-sensitivity content classes, achieving most of the detection benefit at a fraction of the compute. A third embodiment computes signals only on examples flagged as high-risk by a pre-training classifier, focusing the detector on regions of the corpus where memorisation matters most.

Embodiments differ in how reconstruction fidelity is probed. A direct embodiment queries the model with literal prefixes of training examples; this gives the strongest signal but assumes that the prefix is not itself sensitive. An indirect embodiment queries with paraphrased prefixes generated by a probe model, reducing the chance that the probe itself becomes a leakage vector. A canary embodiment inserts synthetic memorisation targets into the training corpus and tracks their reconstruction as a proxy for organic memorisation, accepting that the canary signal is correlated with but not identical to the quantity of interest.

Alternative embodiments vary in remediation venue. A purely training-time embodiment intervenes during training and ships a model whose memorisation has been suppressed at source. A post-training embodiment leaves training intact and applies machine-unlearning procedures after the fact, accepting weaker guarantees in exchange for compatibility with existing pipelines. A serving-time embodiment leaves the model unchanged and installs an output filter informed by the memorisation manifest, suppressing reproduction at inference time without altering parameters. Hybrid embodiments combine training-time intervention for the most severe risks with serving-time filtering for residual risk.

Composition With Other Primitives

The detector composes with the provenance-tracing primitive in two directions. Forward, every memorisation event references the provenance record of the implicated example, so that a downstream rights or privacy claim can be resolved against the responsible source. Backward, the provenance record is annotated with the memorisation outcome for examples drawn from it, supporting source-level governance — corpora that produce disproportionate memorisation events can be reweighted, recurated, or retired.

Composition with the intergenerational-burden primitive imports memorisation residue from the manifest as a burden component on descendants, ensuring that distillation does not silently transmit memorisation from teacher to student. Composition with the deployment-policy primitive uses the manifest to gate deployment: a model with residual memorisation above a deployment threshold is restricted to environments instrumented with the corresponding output filters. Composition with the audit primitive produces machine-readable evidence that governance was applied during training, supporting regulatory or contractual disclosure obligations.

Distinction From Prior Art

Existing approaches to memorisation detection operate primarily after training, querying a finished model to estimate how much of its training data can be reconstructed. These approaches identify a problem only after parameters have been frozen and remediation requires expensive unlearning or retraining. Existing differential-privacy mechanisms add calibrated noise during training to bound memorisation in expectation, but they do so uniformly and at a capability cost that is often unacceptable for production-scale models. Existing data-deduplication preprocessing reduces memorisation risk by removing duplicate examples but does not detect memorisation of unique sensitive examples and does not produce a per-example governance log. The present approach differs in that it operates during training, produces per-example evidence, supports targeted interventions that preserve general capability, and emits a structured manifest that supports both pre-deployment gating and serving-time enforcement.

Implementation Considerations

Production deployment requires attention to three operational concerns. The first is compute budget. Per-example signal computation introduces overhead proportional to the sampling rate; implementations should expose this overhead as a tunable parameter and provide guidance on the points along the cost-coverage curve that satisfy common deployment classes. The second concern is signal calibration. Gradient norms and parameter-localisation measurements depend on architecture, optimiser configuration, and training-stage; thresholds derived for one configuration may not transfer directly to another. Implementations should ship with calibration utilities that fit thresholds to a reference run on the target configuration and should record calibration provenance alongside the manifest so that downstream consumers can validate the threshold choices.

The third concern is manifest ergonomics. The memorisation manifest must be useful to multiple downstream audiences: deployment authorities making gating decisions, serving systems configuring output filters, auditors verifying that governance was applied, and counterparties evaluating residual risk under contractual obligations. Implementations should adopt a layered manifest format in which a compact summary is suitable for routine consumption and a detailed appendix preserves event-level evidence for forensic use. Privacy-sensitive details, including raw example text, must not appear in the manifest itself; the manifest references provenance identifiers that resolve, under appropriate authorisation, to the underlying corpus entries.

A fourth concern is adversarial robustness. An adversary controlling part of the training corpus may attempt to insert examples that evade the detector while still being absorbed into model parameters, for example by spreading the memorisation signal across many near-duplicate examples rather than concentrating it in one. Implementations should include cross-example aggregation that detects clustered memorisation across nominally distinct items and should retain raw signal data long enough to support post-hoc analysis when adversarial behaviour is suspected. Implementations should also support periodic recalibration against an evolving threat model, treating memorisation governance as an ongoing engineering practice rather than a one-time configuration.

A fifth concern is the relationship between memorisation governance and other training-time policies. Differential-privacy noise, data-deduplication, and content filtering each interact with memorisation signals in ways that can confound detection if not accounted for. Implementations should expose the active configuration of all interacting policies in the manifest so that a downstream consumer can interpret residual scores against the full governance posture rather than treating memorisation in isolation. This integrated reporting also supports regulators and auditors evaluating whether the combination of policies meets a domain-specific standard for responsible model training.

Disclosure Scope

The disclosure encompasses methods for computing per-example memorisation scores by fusing gradient sensitivity, parameter localisation, and reconstruction fidelity signals; methods for binding scores to threshold-driven interventions including rate-limiting, localised regularisation, excision, and quarantine; methods for emitting a structured memorisation manifest accompanying the trained model; and methods for composing memorisation governance with provenance, lineage, deployment, and audit primitives. Embodiments addressing full, stratified, and risk-targeted sampling fall within scope, as do direct, indirect, and canary fidelity probes and training-time, post-training, and serving-time remediation venues.