Determined AI Orchestrates Compute, Not Learning Depth

1. Vendor and Product Reality

Determined AI was founded in 2017 by Evan Sparks, Neil Conway, and Ameet Talwalkar — UC Berkeley AMPLab and Carnegie Mellon researchers whose academic work on Spark MLlib, hyperparameter search (the ASHA algorithm), and adaptive resource allocation became the intellectual basis for the product. The company open-sourced its core platform in 2020 and was acquired by Hewlett Packard Enterprise in June 2021, where it now sits inside the HPE Machine Learning Development Environment alongside HPE Cray Supercomputing and the Ezmeral data fabric, serving HPE's enterprise and federal HPC customers as well as the residual open-source community.

The product's scope is the operational complexity of distributed deep-learning training. It handles GPU scheduling across heterogeneous clusters (NVIDIA H100, A100, older V100, AMD MI300 in HPE deployments), elastic scaling that grows and shrinks training jobs as cluster pressure changes, automatic checkpoint management with fault-tolerant resumption after node failures, and distributed hyperparameter search using ASHA and population-based training to prune unpromising trials early. The PyTorch and TensorFlow integration is non-trivial: Determined wraps the user's training loop, transparently parallelizes it across devices using DDP or Horovod, and surfaces a web UI and REST API for experiment tracking, metric visualization, and model artifact management.

The platform's strengths are real and the engineering is mature. Customers running large-model pre-training, fine-tuning, and architecture-search workloads use Determined to keep clusters at high utilization, to recover automatically from the inevitable hardware faults of multi-week training runs, and to compress hyperparameter search wall-clock time by a factor of three to ten relative to grid or random search. Within its scope — operational orchestration of training compute — Determined is one of the few credible open-source-rooted alternatives to vendor-specific stacks like NVIDIA's Base Command Platform and AWS SageMaker. What it does not do, and structurally was never designed to do, is govern what the model learns at each layer during training. That is delegated entirely to the user's training code.

2. The Architectural Gap

The structural gap is the boundary between compute governance and learning governance. Determined governs which hardware runs which job, how gradients are aggregated efficiently across devices, and how trials are pruned in hyperparameter search. It does not govern which training examples influence which model layers, how that influence is provenance-tracked, or how depth-specific policies (freeze early layers, route only public-domain gradients to embedding tables, restrict copyrighted-corpus updates to LoRA adapters) are enforced during the gradient step. The platform treats gradients as opaque tensors to be aggregated quickly and correctly; it cannot treat them as credentialed observations subject to a routing policy because that conceptual layer does not exist in the architecture.

The gap is invisible when training proceeds normally and becomes visible the moment regulators, customers, or rights-holders ask depth-and-provenance questions. The EU AI Act's general-purpose AI obligations, in force from August 2025, require that providers of GPAI models (which include any model above the 10^25 FLOP training threshold and many below) maintain technical documentation of training data, including a "sufficiently detailed summary" of copyrighted content used. The U.S. Copyright Office's 2025 guidance on AI training, the ongoing Authors Guild v. OpenAI and New York Times v. Microsoft litigation, and the emerging Japanese and UK text-and-data-mining safe harbors all converge on the requirement that training providers be able to demonstrate which content influenced which model behaviors. A platform that cannot route gradients by provenance cannot produce that demonstration except by retraining from scratch.

The gap also bites in the contemporary memorization-and-leakage problem. Models trained without depth-selective governance routinely memorize personally identifiable information, copyrighted text, and proprietary data verbatim, especially in early training and especially in lower transformer layers that capture surface form. The mitigations available to a compute-only orchestrator — differential privacy noise on the whole gradient, full-corpus deduplication, post-hoc unlearning — are blunt instruments that either degrade overall model quality or fail to reach the layers where memorization concentrates. Depth-selective gradient routing — the policy primitive that says "this batch's gradients may update layers 24 and above but not embedding or shallow layers" — is implementable today only by hand-rolled training code that bypasses the platform's optimizations and forfeits the platform's fault tolerance, because the platform's checkpoint-and-resume semantics do not understand layer-depth policies.

Determined cannot patch this from within its current architecture without re-conceiving what an experiment is. A run today is "code plus data plus hyperparameters yields metrics and a checkpoint." A run under learning governance is "code plus credentialed-data-with-provenance plus hyperparameters plus depth-policy yields metrics, checkpoint, and a provenance-coverage attestation." The latter requires governance state to be a first-class object the platform serializes, restores, and presents in its UI — which is a different shape of platform.

3. What the AQ Training-Governance Primitive Provides

The Adaptive Query training-governance primitive specifies that every gradient update in a conforming pipeline is admitted as a credentialed observation with a depth-routing policy applied before aggregation. Each training example carries provenance credentials — corpus identifier, license class, jurisdiction tag, sensitivity level — and the depth policy is a function from credential class to the set of layers that may receive its gradient contribution. The aggregation step is governance-aware: gradients from a public-domain batch may flow to all layers, gradients from a licensed-corpus batch may flow only to higher layers permitted by the license, gradients from a redaction-restricted batch may flow only to LoRA adapters that can later be removed.

The primitive composes with the rest of the AQ governance stack. The provenance credential is itself an authority-credentialed observation in the governance-chain sense, the depth policy is published and signed in the same authority taxonomy, and the resulting model carries a lineage record that allows any downstream user to verify which corpora influenced which layers. Memorization is detected as an entropy-profile deviation — a learned representation whose entropy at a given layer is below the policy-expected band signals memorization rather than generalization, which the governance layer can flag, gate, or roll back. Hyperparameter search is governance-aware: trials that achieve low loss through memorization are dominated by trials that achieve governed learning even at slightly higher loss.

The primitive is technology-neutral with respect to framework (PyTorch, JAX, TensorFlow), aggregator (DDP, FSDP, ZeRO, Horovod), and accelerator (NVIDIA, AMD, TPU, custom), and it composes hierarchically across layers, modules, and model partitions. The inventive disclosure is depth-selective gradient routing under credentialed-observation provenance with governance-state checkpointing as a structural condition for governed model training.

4. Composition Pathway

Determined integrates with AQ as a compute-orchestration substrate running under the training-governance layer. What stays at Determined: the GPU scheduler, the elastic-scaling logic, the fault-tolerant checkpointing, the experiment-tracking UI, the ASHA hyperparameter search, the PyTorch and TensorFlow wrappers, the HPE customer relationship and federal HPC qualification. Determined's investment in operational distributed-training expertise — communication-aware placement, gang-scheduling on InfiniBand fabrics, fault-recovery semantics — remains its differentiated layer.

What moves to AQ as substrate: provenance-credentialed batches, depth-routing policies, governance-aware aggregation, governance-state checkpointing, and provenance-coverage attestations. The integration points are well-defined. The Determined data loader is wrapped by an AQ provenance loader that attaches credentials to each batch. The aggregation hook (currently used by Determined for DDP-style averaging) is replaced by a governance-aware aggregation that consults the depth policy before reducing across devices. The checkpoint serializer is extended to include governance state — active provenance chains, depth policies in effect, entropy profile expectations per layer — so that resumption from a fault preserves not just model weights but governed-learning context. The hyperparameter search is augmented with governance metrics in its objective, so ASHA prunes trials that achieve loss through ungoverned memorization.

The new commercial surface is governance-credentialed model training for HPE's federal and regulated-industry customers — DoD, intelligence community, financial services, healthcare — that increasingly cannot deploy models trained without provenance-coverage attestations. The substrate is portable across compute vendors, survives platform migrations, and produces models whose lineage is verifiable independently of HPE or Determined's internal records.

5. Commercial and Licensing Implication

The fitting arrangement is an embedded substrate license: HPE / Determined embeds the AQ training-governance primitive into the Machine Learning Development Environment and sub-licenses governance-credentialed training to enterprise and federal customers as part of the platform contract. Pricing is per-governed-GPU-hour or per-attested-model-artifact rather than per-cluster-seat, which aligns with how regulated customers actually consume training capacity.

What HPE gains: a structural answer to the EU AI Act GPAI documentation requirement and the U.S. copyright-and-training litigation environment that platform-only competitors cannot match without re-architecture; a defensible position against NVIDIA Base Command Platform and AWS SageMaker by elevating the architectural floor on training governance rather than competing on raw cluster economics; and a forward-compatible posture against the next regulatory layer, which is widely expected to require provenance-coverage attestations as a condition of model deployment in regulated sectors. What the customer gains: portable, audit-grade lineage on every model artifact, the ability to satisfy depth-and-provenance requests from regulators, rights-holders, and downstream auditors without retraining, and a single governance chain spanning data ingestion, training, fine-tuning, and deployment under one authority taxonomy. Honest framing: the AQ primitive does not replace Determined's compute orchestration. It gives Determined the learning-layer governance the platform was never architected to provide and the regulatory environment now insists on.