One quality score can't tell mis-elicited preference from broken reality. Two posteriors can.
Keep preference and reality on separate evidence channels, weight every grader by measured reliability, and watch the gap between them for Goodhart.
A Bayesian Alignment Kernel makes machine judgment underwritable. Every output leaves the loop as a self-similar atom of evidence - the claim, the channel that produced it, and the measured reliability of every grader that scored it - the same typed contract from a single verdict up to a portfolio of pipelines. That is the business consequence: autonomy extended the way a lender extends credit, against measured loss rates instead of demos, with a stated alarm for the day the book is being marked to model instead of market.
Mechanically, the kernel runs on two posteriors instead of the single number most loops optimize. One “quality” score silently merges four objects - what the operator wants, what reality rewards, what the system can produce, and how much each grader can be trusted - and while they share one number, reward hacking is not even statable: there is nothing for the hacked metric to diverge from. The kernel is a typed tuple that keeps the four apart: disjoint evidence channels feeding two posteriors, a reliability ledger that gates every update, and a divergence statistic between the posteriors that rings when the proxy is being gamed.
One round of a kernel: select by preference, judge by reality. The generator proposes; the preference posterior and the reliability-weighted verifier battery select; shipped outcomes update a separate reality posterior, which deliberately does not feed selection. The dashed edge between the two posteriors is δ - the divergence you watch.
The Insight
The kernel's entire move is typing the concerns apart and forcing each claim to carry its evidence channel. Preference evidence (tournaments, keep/drop verdicts) only ever updates the preference posterior. Realized outcomes only ever update the reality posterior. Every evaluator - human, LLM judge, or deterministic check - carries a reliability entry that scales how far its evidence can move anything.
The separation is load-bearing because a single quality score makes three failure modes indistinguishable: the operator's preferences were mis-elicited, the preferences are real but reality disagrees with them, or the evaluators measuring either one are too noisy to trust. Each demands a different response - re-elicit, re-anchor, or recalibrate - and a single latent variable cannot tell you which one you are in. The framework is not a new alignment algorithm; it is a typed contract for composing the mechanisms you already run.
The Object
Nine typed components; the load-bearing pair is Θ_op and Θ_real - two posteriors that never share an evidence channel.
A, B - typed task and output spaces. π_θ : A → Δ(B) - the generator (model, pipeline, or whole agent session); the only component that is literally a Markov kernel.
Θ_op - prior and posterior over the latent operator-preference function R_op, updated only by elicited preference evidence: pairwise comparisons under a Bradley–Terry likelihood, keep/drop verdicts under a Rasch-style threshold, ordinal verdicts under a cumulative logit. Pairwise evidence identifies differences only; mixed elicitation pins the scale.
Θ_real - prior and posterior over the latent reality-outcome function R_real, updated only by realized outcomes after shipping. One caveat is structural: outcomes are observed only for outputs that were selected, so the channel inherits selection bias unless you ship an exploration fraction or at least log the selection score next to every outcome.
V_ψ, ρ - the verifier battery (deterministic checks first, LLM judges only from a different provider than the generator) and the per-evaluator reliability ledger: agreement with prevalence, calibration, test-retest variance, cross-provider disagreement. ρ is not a dashboard - it enters the math four ways, below.
R̂_φ, U - a cheap learned proxy for R_op (the thing best-of-n actually calls), and the update operator that revises prompts, proxies, rubrics, and both posteriors on a stated cadence - gated by ρ, with the rollback condition declared in advance.
A posterior over functions is infinite-dimensional, so an instantiation must commit to a representation: a grid over item strengths for small pools, a Laplace approximation over Bradley–Terry logits for tournament data, or an ensemble of proxy heads. If you cannot name which one you are running, you are not running the kernel.
The Selection Rule
Selection is best-of-n with an uncertainty penalty: the winner is the candidate the preference posterior both rates highly and actually knows about, with verifier evidence weighted by precision.
The rule on an illustrative candidate field (the candidates are invented; the rule is exact). The raw argmax of E[R_op] chases the high-mean, high-uncertainty candidate - rose X; the γ·sd penalty moves the pick to b*, a candidate the posterior actually knows. Slanted dashes are iso-score levels of E − γ·sd; b* sits on the highest one.
The pessimism term is the overoptimization mitigation: candidates the preference posterior is unsure about get discounted, which blunts the proxy-hacking incentive. λ and γ are not vibes - you calibrate them by grid search against realized outcomes on a probe set, picking the regret-minimizing cell and tie-breaking toward larger γ. Precision weights handle noise, not validity: a reliable-but-invalid verifier is confidently wrong, so validity is audited separately against realized outcomes, and a verifier with near-zero deattenuated validity is dropped regardless of how consistent it is.
Reliability Is Load-Bearing
The ledger ρ is wired into the math at four points, so an untrusted evaluator cannot move the system no matter how loud it is. Most pipelines treat evaluator reliability as a report you read and then ignore; here it is a constraint:
1. Attenuation correction
Measurement noise biases observed correlations toward zero (Spearman, 1904). Claims about a verifier's validity use the deattenuated estimate - corr_obs / √(rel_V · rel_outcome) - with correspondingly widened intervals, so a good-but-noisy verifier is not unfairly dropped and a noisy comparison is never read as precise.
2. Noisy-evaluator likelihoods
Each verdict is modeled with its evaluator's sensitivity and specificity (Dawid–Skene-style), or with measurement-error variance estimated from test-retest runs. An unreliable evaluator yields a flatter likelihood, so the posterior moves less on its say-so - automatically, with no manual discounting step to forget.
3. Update gating
Evidence from an evaluator below its reliability floor contributes nothing to any update. The default floor is κ ≥ 0.6 - the Landis–Koch substantial-agreement boundary, a defensible default to override per domain via decision economics, not taste. Below-floor evaluators are quarantined and recalibrated, not silently kept.
4. Support gating
A ρ entry is local evidence, not a passport. Its support is the tuple (surface, construct, evidence channel, labeler mechanism, domain). Before an evaluator can move an update, the caller must show the requested judgment is inside that support. Out-of-support evaluators resolve to unmeasured for bootstrap claims and below-floor for hard autonomy gates. A transfer gate is judged against the gold channel's reliability ceiling, not the default κ floor. The canonical failure is a judge calibrated on self-contained artifacts applied to registry-grounded inference: the model can be confident and anti-correct because the required evidence is not in the artifact.
Bridge 1 is pure formula - no data points exist or are implied. Observed validity = true validity · √(rel_V · rel_outcome), so a noisy evaluator's correlation with outcomes shrinks toward zero even when the true relationship is strong: at a reliability product of 0.49, a true 0.80 reads as 0.56. The kernel divides the shrinkage back out before judging any verifier.
The Divergence Monitor
The divergence δ between the two posteriors is the Goodhart alarm - the statistic the single-score loop cannot even state, available here because preference and reality never shared a number. The two latents live on different scales, so every estimator is scale-free:
Schematic, after Gao, Schulman & Hilton (2022) - shape, not data, and no numeric axes on purpose. Under continued optimization the preference-side proxy keeps rising while the realized outcome turns down; the shaded region is δ_trend, the slope-opposition window the kernel alarms on.
The third is the canonical proxy-hacking signature - the shape Gao, Schulman & Hilton (2022) measured for reward-model overoptimization. Rising δ under active optimization means the proxy is being hacked relative to reality. The response ladder is ordered by cost: freeze updates, re-elicit preferences (the preference model may be stale rather than wrong), raise the uncertainty penalty γ, and only then treat the verifier battery as compromised.
Worked example: selecting a visual language
The kernel instance that picked this site's visual language earned its keep twice: the preference posterior selected the winning language, and the reliability ledger quarantined a worse-than-chance instrument before its labels touched anything. The task space A is a render brief (subject, surface, topic); the output space B is rendered figures; the generator is an SVG pipeline; the committed posterior representation is a Bradley–Terry fit. Sixty candidate languages entered a tournament; 209 confidence-weighted pairwise labels produced a winner at strength 2.17 against 1.61 for the strongest alternative, with 13 of 20 head-to-head wins in the deciding round. Every figure on this page, including the diagram above, renders in the language that fit selected.
Measured - the only numbers drawn are the measured ones. The Bradley–Terry fit also makes a checkable prediction: σ(2.17 − 1.61) ≈ 0.64 head-to-head win rate, against 13/20 = 0.65 observed in the deciding round. Preference-side evidence only, per the thin reality channel declared below.
Parameter caveat. This instance had no Θ_real probe set, so λ and γ were not calibrated by the selection-rule procedure above. The visual-language pick used the preference posterior and the tournament fit only; any uncertainty penalty was preference-side caution, not outcome-validated regret minimization. Until per-figure outcomes are linked, this example demonstrates Θ_op selection and bridge-3 quarantine, not calibrated best-of-n against reality.
The quarantine episode is bridge 3 executing exactly as specified. A test-retest run on one elicitation instrument - an ungrounded “pure gut, no metadata” classification prompt - returned κ = −0.212: worse than chance, far below any floor, so the instrument was quarantined and its labels never reached a posterior. The rebuilt instrument grounds the judgment (subject, surface, and topic are shown) and blinds the bias surface (language name, renderer, provider, and prior rankings are hidden), and recollection is gated on κ ≥ 0.5 over a stratified 20-item retest. The episode predates the kernel's write-up; the framework names the discipline rather than inventing it.
A measured instrument on a published scale: the ungrounded instrument retested at κ = −0.212 and was quarantined; the rebuilt instrument must clear κ ≥ 0.5 on a stratified 20-item retest; the default update-gating floor stays at the Landis–Koch 0.6 substantial-agreement boundary.
The verifier battery follows the deterministic-first rule: a geometric gate computes edge crossings, lane collisions, box incursions, and label overflow from the exact layout the renderer draws (every figure on this page passes it with zeros across the board), and a vision grader from a different model provider than the generator handles the perceptual reads. The reality channel is the honest gap: site analytics exist, but no per-figure outcome signal is linked yet - so Θ_real is thin, δ is not yet computable, and every claim in this example is stated preference-side only. The kernel's own rules require saying that out loud.
Was this page itself produced by a kernel? Only partially, and the gap is still statable in the kernel's own types. Since publication, published posts fail by default without a fresh VerificationProof sidecar, and one transcript-to-proof-graph mint path now refuses below-floor or quarantined hard-gate evaluators. That closes part of the public-tier crack. It does not make this page a full kernel: no preference posterior selected among candidate pages, the essay/page graders still carry no ρ entries, unmeasured evaluators still warn rather than refuse, and the proof ratchet is not a Θ_real outcome channel.
When the kernel breaks
The kernel degrades in six typed ways, and naming the active one is part of the contract - each is a falsifiable commitment the framework generates:
- -Entangled channels. If preference and reality evidence cannot be separated in a domain, the two posteriors collapse into one and δ is undefined. The kernel degrades to the single-latent loop it was built to replace - an instantiation must say so rather than pretend.
- -Selective labels. Reality outcomes arrive only for shipped outputs. Without an exploration fraction, importance weights, or at minimum logged selection scores, the reality posterior quietly learns the selection policy instead of reality.
- -Reliable-but-invalid verifiers. Precision weighting amplifies consistency, including consistent wrongness. If validity is not audited against outcomes, the battery converges on confident error - the verification trap.
- -Construct-transfer drift. A verifier's ρ does not transfer just because the prompt still runs. If the construct changes, or if the target label requires evidence outside the artifact, an in-domain reliable judge can become high-confidence anti-signal. The test is held-out agreement at the construct/evidence-channel boundary; negative κ under high confidence means stop, not discount. Add the missing channel or build a new instrument, then make it climb its own reliability ladder. (Held-out example: a self-contained-evidence severity judge applied to registry-grounded clinical inference, κ = −0.23, the inversion the held-out Cochrane test caught at the boundary before any production caller saw it.)
- -Assumed composition. Generators compose; nothing else in the tuple is known to. For a conjunctive pipeline, the product of per-stage agreement rates is only a lower bound on raw composite agreement, and deep conjunctions drift toward “almost always fails,” where raw agreement looks high and κ collapses - the prevalence pathology in its purest form. Measure composite reliability; do not derive it.
- -Mistaking it for an algorithm. The kernel proves nothing by itself. It is a contract that makes existing mechanisms - tournaments, verifier batteries, update gates - compose without lying about what their evidence supports. Ensemble uncertainty is a lower bound; γ compensates, imperfectly.
Connection to Other Frameworks
The kernel is the composition layer: each neighboring framework fills a slot in the tuple.
The Deity Problem - its three evidence channels for learning operator intent are exactly what feeds Θ_op. The kernel adds the likelihoods and the reliability ledger on top of the elicitation.
Quality Hillclimb - the special case where the verifiers are assumed reliable and the gate is treated as ground truth. The kernel is what you graduate to when those assumptions need auditing.
The Capital Value of Verifiers - V_ψ is the appreciating asset; ρ is its audited book value. A verifier without a reliability entry is an asset carried at an invented mark.
Designed Convergence - the update operator U, with its declared rollback condition and ρ-gating, is the convergence mechanism stated as a typed component instead of a hope.
Prior art & the name21 entries · what is imported, what is newshow ↓
Every mathematical component on this page is an import; the assembly is the contribution. The register below names where each part comes from. What is assembled - two posteriors with disjoint evidence channels, a reliability ledger wired into update gating, and a divergence alarm, packaged as one typed object that lives at selection time in an operator's loop rather than inside a training run - is, as far as I can find (checked 2026-06-12), unclaimed as a name and unassembled as one contract. That is a claim about my search, not about the literature - it is one counterexample away from a correction, and the changelog will record any predecessor a reader surfaces.
Two elements I would defend as more than renames: evaluator test-retest reliability wired into update gating for LLM-evaluator pipelines, with its falsifiable companion claim that below-floor evaluators predict downstream reality failures; and the divergence alarm computed between two live posteriors, rather than against a held-out gold model.
Links are included only where the source was verified directly; classical references are cited bibliographically and left unlinked rather than guessed.
Preference evidence
- Bradley & Terry (1952), Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons - Biometrika 39 - The pairwise-comparison likelihood the preference posterior is fit with. Within a context it identifies strength differences only - the reason mixed elicitation is needed to pin the scale.
- Rasch (1960), Probabilistic Models for Some Intelligence and Attainment Tests - Danish Institute for Educational Research - The threshold model behind keep/drop verdicts: P(keep) = sigma(R_op - c_e), with the evaluator threshold estimated jointly.
- McCullagh (1980), Regression Models for Ordinal Data - J. Royal Statistical Society B 42 - Cumulative logit with ordered thresholds, for approve / revise / kill verdicts.
- Christiano et al. (2017), Deep Reinforcement Learning from Human Preferences - arXiv:1706.03741 - Pairwise preference as the canonical evidence type for a learned objective. The kernel keeps the evidence type and moves the fit to selection time.
- Ouyang et al. (2022), Training Language Models to Follow Instructions with Human Feedback - arXiv:2203.02155 - The training-time preference pipeline whose modeling conventions the operator loop inherits and contrasts with.
Evaluator reliability
- Spearman (1904), The Proof and Measurement of Association between Two Things - American J. Psychology 15 - Bridge 1: the attenuation correction that deattenuates validity correlations for measurement noise.
- Cohen (1960), A Coefficient of Agreement for Nominal Scales - Educational and Psychological Measurement 20 - Kappa, the chance-corrected agreement statistic the ledger reports for binary verdicts.
- Landis & Koch (1977), The Measurement of Observer Agreement for Categorical Data - Biometrics 33 - The agreement benchmarks behind the default kappa >= 0.6 update-gating floor.
- Dawid & Skene (1979), Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm - Applied Statistics 28 - Bridge 2: sensitivity/specificity marginalization over noisy evaluators, so reliability enters the posterior automatically.
- Shrout & Fleiss (1979), Intraclass Correlations: Uses in Assessing Rater Reliability - Psychological Bulletin 86 - The ICC variants used for continuous-score test-retest entries in the ledger.
- Feinstein & Cicchetti (1990), High Agreement but Low Kappa - J. Clinical Epidemiology 43 - The prevalence paradox: why the ledger reports kappa with prevalence, and why deep conjunctive pipelines drift toward collapsed kappa at high raw agreement.
- Gwet (2008), Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement - British J. Mathematical and Statistical Psychology 61 - AC1, reported alongside kappa when verdicts are skewed.
- Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena - arXiv:2306.05685 - Measured judge biases and judge-human agreement rates: the empirical reason LLM evaluators get ledger entries instead of trust.
Selection + divergence
- Kendall (1938), A New Measure of Rank Correlation - Biometrika 30 - The tau inside the rank-divergence estimator.
- Cobbe et al. (2021), Training Verifiers to Solve Math Word Problems - arXiv:2110.14168 - Verifier-gated best-of-n: the lineage of the lambda term in the selection rule.
- Lightman et al. (2023), Let's Verify Step by Step - arXiv:2305.20050 - Process supervision beats outcome supervision on multi-step tasks: why the composite-verifier question is posed as per-stage concatenation versus final-output checking.
- Gao, Schulman & Hilton (2022), Scaling Laws for Reward Model Overoptimization - arXiv:2210.10760 - The proxy-versus-gold divergence curves. The trend estimator is this signature watched live.
- Coste et al. (2023), Reward Model Ensembles Help Mitigate Overoptimization - arXiv:2310.02743 - Ensemble-conservative objectives: the lineage of the gamma uncertainty penalty.
- Manheim & Garrabrant (2018), Categorizing Variants of Goodhart's Law - arXiv:1803.04585 - The failure taxonomy the divergence alarm ladder operationalizes.
Reality channel
- Lakkaraju, Kleinberg, Leskovec, Ludwig & Mullainathan (2017), The Selective Labels Problem - KDD 2017 - Outcomes are observed only for selected items. The exploration-fraction and logged-selection-score mitigations come from here.
The name
- Cristianini et al. (2001), On Kernel-Target Alignment - NeurIPS 14; Kornblith et al. (2019), Similarity of Neural Network Representations Revisited - arXiv:1905.00414 - Disclosure, not lineage: "kernel alignment" in kernel methods measures similarity between representations and is unrelated to this object. The overload runs in the other word order, and only the generator here is literally a kernel (a Markov kernel).
Changelog2 entries · building in publicshow ↓
- Cross-provider adversary pass (gpt-5.5, high effort) against the page given recent shipped evidence (REFUTED Cochrane transfer test, 3-provider judge panel port, ρ-floor enforce on a real mint path, site publish-proof ratchet). Four findings applied: (1) added a 4th reliability bridge - Support gating - because a ρ entry is local evidence, not a passport; (2) added a 6th failure mode - Construct-transfer drift - distinct from reliable-but-invalid, with the κ = -0.23 inversion as the held-out example; (3) added a parameter caveat to the worked example - λ and γ were not calibrated, the visual-language pick was Θ_op + bridge-3 only; (4) refreshed the "was this page produced by a kernel?" disclosure to reflect the publish-proof ratchet and the live ρ-floor enforce.
- The tuple, the three reliability bridges, the divergence monitor, and the visual-language worked example. Five computed figures (selection-rule field, attenuation curves, overoptimization schematic after Gao et al., Bradley-Terry outcome bars, the kappa gating line carrying the real quarantined instrument) - every coordinate computed from a data table and machine-verified for containment, label keepouts, monotonicity, and argmax separation. Prior-art register: every imported component cited with what it contributes, the novelty claim stated falsifiably, the kernel-methods name collision disclosed. Every section leads with its bottom line.
Rosetta Stone
Four circles, four readings of the same object. Each role reads the artifact through its own lens.
Two ledgers that must reconcile. Optimizing the preference posterior while the reality posterior diverges is marking to model instead of marking to market - delta is the spread, and a rising spread under active optimization means the book is overstated.
Stop trusting graders on reputation. Every evaluator - human reviewer, LLM judge, automated check - carries a reliability entry, and evidence from a below-floor evaluator moves nothing until the instrument is rebuilt. The quarantine is the discipline.
A typed interface over components you already run: a generator, two posteriors with disjoint evidence feeds, a precision-weighted verifier battery, a gated update operator. The missing pieces in most stacks are the reliability ledger and the divergence monitor - build those, reuse the rest.
The likelihoods are written down: Bradley-Terry for pairwise preference, Rasch thresholds for verdicts, noisy-rater marginalization for every evaluator. Claims are posterior statements carrying the rho of each evaluator that shaped them, and composition across pipeline stages is stated as an open problem with a measurable conjecture, not assumed.