Assurance · calibration, drift, and accountability

Any model will emit a probability. Almost none of them are ever scored.

A warning system's real metric is not accuracy. It is whether the stated probability means what it says, what a false one costs, and how many the watch floor absorbs before it stops reading the display. This page carries the machinery that answers those questions — reliability decomposition, false-alarm economics, drift triggers, and the accreditation path — because a capability that cannot be scored cannot be trusted, and one that cannot be trusted will not be used twice.

Live scoring

Current backtest

—Brier score

—Skill vs base rate

—Reliability term

—Resolution term

—Base rate

—Scored events

Synthetic corpus

3,000 held-out seeded events. These numbers prove the scoring machinery exists and works. They are not evidence of real-world calibration, which would require real outcomes.

Reliability diagram

Does 70% mean 70%?

Perfect calibration is the diagonal. Points below it are overconfidence: the model said it more often than it happened.

Murphy decomposition

Where a good Brier score hides a bad habit

Stated band	n	Mean stated	Observed	Gap

The aggregate number lies by omission. The reliability term across all bands sits inside the retraining trigger, so an automated drift check would report this model as healthy. Read the top bands on their own and the picture changes: the model is materially overconfident exactly where the consequential decisions get made. An aggregate score that passes while the decision band fails is the most common way calibration monitoring gives false comfort.

False-alarm economics

Pick a threshold. Watch what it costs.

The operating point is a policy choice, not a technical one, and it is the choice that determines whether the system survives contact with a watch floor.

Modeled from backtest

Alert threshold 0.60 Warning events per month 40

—False alarms / mo

—Missed / mo

—Precision

—Recall

—Operator trust @ 6 mo

Every threshold trades a missed warning against a false one. There is no setting that avoids both, and the honest version of this product says so on the page instead of in a footnote.

Model registry

Every model in the stack, its limits, and who signed for it

A model with no owner, no stated limits, and no recalibration date is not a component. It is a liability with an API.

Model	Role	Kind	Trained	Stated limits	Drift

Retraining triggers

Retraining is an event, not a cadence. A fixed quarterly retrain is a calendar habit that ignores whether anything changed. These triggers fire on measured conditions — novelty rate, reliability drift, catalog change, or a single containment failure — with a 90-day backstop so that a quiet period cannot let a stale model run indefinitely.

Verification, validation, accreditation

How this would actually become usable

The gap between a working demonstrator and a fielded capability is almost entirely this column. It is usually left off the slide.

Planning framework

Gate	What it establishes	Evidence this build already produces	Still required
Model verification	The code solves the equations it claims to solve	Atmosphere reproduces USSA-76 tabulated density to 3–4 significant figures; propagator is deterministic and seeded	Independent re-implementation and cross-comparison against a reference trajectory code
Model validation	The equations describe reality closely enough for the decision	None. All corpora are synthetic.	Comparison against instrumented flight-test or range data. This is the gate that cannot be closed with open sources.
Simulation accreditation	An authority accepts the model for a stated use	Stated limits, assumptions, and known omissions published per surface	Named accreditation authority, intended-use statement, and a formal acceptance decision
Algorithmic test & evaluation	The learned components behave across the input space	Reliability decomposition, novelty detection, out-of-distribution behaviour demonstrated live	Adversarial evaluation, subgroup performance, and a red-team against deliberately novel profiles
Responsible-AI review	Appropriate human judgment is preserved	Decision rules name a human authority; Target and Engage removed from scope	Formal review against the governing AI policy and a documented human-judgment analysis
Software authorisation	The system may run on a given network	Static site, no runtime backend, no data at rest	Full control assessment against the applicable risk-management framework for the target enclave

The validation row is the honest one. Verification can be closed with care and open references. Validation cannot. No amount of open-source modelling substitutes for measured flight data, and any vendor claiming a validated hypersonic impact-point model on open sources alone should be asked which range provided the truth data.

Classification and releasability

What the model inherits from its training data

A model trained on controlled data produces controlled outputs and carries the classification of its training corpus in its weights. That is a deployment constraint, not a paperwork detail: it determines which networks the model can run on and which partners can ever receive it. A tiered architecture has to be designed in from the start, because it cannot be retrofitted by declassifying a trained model.

Tier	Corpus	Where it can run	Shareable with
Tier 0 — open	Open-source catalogs, published atmosphere, synthetic tracks	Unclassified networks, laptops, coalition demo	Any partner. This is what the current build is.
Tier 1 — controlled	Adds controlled technical parameters	Controlled-unclassified enclaves	Case-by-case, per the owning authority
Tier 2 — national	Adds intelligence-derived observations	National classified networks only	Not releasable as a model
Tier 3 — coalition variant	Tier 2 architecture retrained on releasable data only	Partner networks	Requires a separate training run, not a filter on outputs

Post-event scoring

The loop nobody builds

Without a scoring loop there is no calibration, no learning, and no accountability — only a system that has been confidently wrong an unknown number of times. Each stage below is cheap individually and is almost always skipped collectively.

One rule is worth the whole loop. A single observed impact outside the 90% containment ellipse forces a recalibration review. Not a ticket, not a trend line — a review. Containment claims are the product; a breached one is a defect.

Sustainment

Who keeps this alive at 0300

A two-person team is a compelling origin story and an unacceptable steady state. The question an evaluator will ask is not who built it — it is who answers the phone eighteen months from now when it disagrees with the radar.

Function	Why it cannot be zero	Minimum
Model owner / analytic lead	Owns calibration, signs retraining decisions, defends the assessment record	1.0
Data pipeline & labelling	New systems enter the catalog continuously; an unmaintained catalog silently expands the out-of-distribution rate	1.5
Software sustainment	Dependency, security, and enclave maintenance for whatever network it lands on	1.5
Test & evaluation	Independent scoring — the model owner cannot grade their own work	1.0
Operator training & docs	Comprehension is a design surface; an untrained crew reads a footprint as a target	0.5
Watch support	Someone reachable when the display and the radar disagree at 0300	2.0

Operator comprehension

Designing against both failure modes

Automation bias and automation distrust are the same design problem approached from opposite ends, and most decision-support tools accidentally optimise for one by ignoring the other.

Against over-trust. Confidence is displayed separately from probability. Out-of-distribution tracks are visually distinct and cannot be narrowed by the classifier. Rules that cannot execute in the remaining window turn red rather than recommending the impossible.

Against under-trust. Every number carries its drivers, its alternatives, and what would change it. Seeds are shown so any judgment can be reproduced. The known-limits list is published on the console itself, not buried in documentation.

The 0300 test. Everything on the console has to be readable by a tired operator in one pass: the footprint is a shape and it is either shrinking or it is not, the rule is either green or it is red, and the track is either recognised or it is marked novel. Anyone who needs the reliability diagram to act on a warning has already been failed by the display.