Conformal Calibration for Multi-modal Regression with Missing Modalities

Each auxiliary predictor g_k outputs one plain number: its own guess of the target, in target units. In the experiments these are plain XGBoost regressors, one per modality. For a SRED listing:

• tabular  tabular variables (size, rooms, location…) → g₁ → “log(rent) ≈ 7.60”
• text  description → frozen text encoder (E5 / MiniLM) → embedding vector → g₂ → “7.90”
• image  photos → frozen DINOv2 → embedding vector → g₃ → “6.85”

The raw text and pixels are never fed to g_k directly. A frozen pretrained encoder turns them into a feature vector first, and the regressor maps that vector to a rent guess. Then d(x) = sd{7.60, 7.90, 6.85} ≈ 0.44, in the same units as the target. Think of three appraisers, each reading only one part of the dossier, each writing down a rent estimate: disagreement is the spread of the three numbers. No labels, no intervals, no probabilities involved.

Keep two roles separate (a common confusion). The base predictor, the model being calibrated, is a different object. It outputs the interval (ℓ, u). That interval comes from a single point prediction ŷ (then ℓ = u = ŷ), from a pair of fitted 2.5% / 97.5% quantiles, or from an SEMF ensemble of latent-propagated decoder outputs summarized into an interval. The g_k's never produce the interval. They exist only to produce d(x).

No, they don't. Conformal prediction never invents a prediction. It corrects one. So the wrapper needs some prediction rule to correct, called the base predictor, and it accepts that rule in any of three forms:

• A point predictor (plain XGBoost). Output: one number ŷ. We treat it as the degenerate interval ℓ = u = ŷ, so the conformal layer builds the entire interval around it: [ŷ − q̂·a_γ, ŷ + q̂·a_γ].
• A quantile predictor (XGBoost trained at the 2.5% and 97.5% levels). Output: a rough interval. Fitted quantiles carry no guarantee. Standard signed CQR may stretch or shrink them; our disagreement-weighted rule clips scores at zero and therefore only expands them.
• An ensemble-output base (the SEMF-derived diagnostic). It samples latent variables and retains 50 decoder outputs, whose empirical quantiles form a rough interval. These are latent-propagated decoder outputs, not draws from the full response distribution, because the cached run omitted SEMF's conditional-response draw.

The base predictor decides the interval's location and shape (where it sits, whether it is asymmetric, whether it is already wider in noisy regions). The conformal layer decides its size, so that 95% coverage is guaranteed. A quantile or ensemble-output base is worth having because it adapts the shape to each x. With a point base, all the width has to come from the padding. The experiments use all three base-output types, and that is exactly what “predictor-agnostic” means here.

Is the base predictor a separate model, and must it be multi-modal? It is separate because it is the model you would deploy anyway, and the wrapper never trains or modifies it. It only needs to be multi-modal in a minimal sense. No fusion architecture is required, and the main base predictor in the experiments is one XGBoost trained on all features concatenated (the tabular columns and the cached embeddings side by side as one wide table). What genuinely must be per-modality are the auxiliaries, because d(x) is the spread of their predictions. In the missing-modality experiments the base should consume all modalities, since the interesting failure is a model that relied on images suddenly receiving zeros. A tabular-only base would not care, and there would be nothing to repair.

Illustrative numbers. SRED, with an XGBoost point predictor as the base.

1. Train once, on the fit split (6,498 listings). The base predictor is one XGBoost trained on all features together: the tabular columns plus the cached text embedding plus the cached image embedding. The auxiliaries g₁, g₂, g₃ are three plain XGBoost regressors trained on one modality each.
2. Calibrate once, on the calibration split (1,999 listings). The base predictor makes a rent guess ŷ_i for each calibration listing. The error is e_i = |y_i − ŷ_i|. The marginal recipe takes the 95th-percentile error, say q̂ = 0.40. The weighted recipe first divides each error by that listing's a_γ(X_i), then takes the quantile, giving say q̂ = 0.31 (smaller, because dividing by a scale ≥ 1 shrinks the scores).
3. Predict for a new listing. 3.5 rooms, 80 m², Lausanne, five photos. Base predictor: ŷ = 7.62. One number, no interval. Auxiliaries: 7.60 / 7.90 / 6.85, so d = 0.44 and a_γ = $\sqrt{1+8·{0.44}^2}$ = 1.60.
4. Build the interval. Baseline (marginal): 7.62 ± 0.40 → [7.22, 8.02], the same padding as every other listing. Weighted: 7.62 ± 0.31·1.60 = 7.62 ± 0.50 → [7.12, 8.12], wider because the sources conflict. A peaceful listing with d = 0.02 would instead get 7.62 ± 0.31, narrower than the baseline's ± 0.40.

Notice what the base predictor produced: only the number 7.62. Everything interval-shaped came from the calibration layer. The baseline and weighted method use the same fitted model, prediction, and calibration listings; they differ in score normalization, the resulting quantile, and the padding applied to each test point.

Model count for this point-base example. With 3 modalities, four regressors are trained: three auxiliaries (one per modality, each outputting a point guess) and one base predictor on all modalities. More generally the wrapper needs K auxiliary prediction rules plus one base prediction system; a quantile system can itself contain several fitted quantile models. Nothing else in the layer is trained: q̂ is a sorted quantile of calibration errors, γ comes from a small grid search on the tuning split, and the Mondrian groups are just labels.

It is the letter e, not an epsilon, and it is not noise. For one calibration listing, e_i answers a single question: how far did the true label land outside the base interval?

With base interval [7.40, 7.80] and true log(rent) y:

• Truth above it, y = 8.05 → e = 0.25. The interval missed by 0.25 on top.
• Truth below it, y = 7.10 → e = 0.30. Missed by 0.30 underneath.
• Truth inside it, y = 7.60 → e = −0.20. Negative means covered with slack to spare.
• Point predictor (ℓ = u = ŷ) → e = |y − ŷ|, the plain absolute error.

Compute e_i for every calibration listing (their true rents are known), sort the values, and take the ⌈(n+1)(1−α)⌉-th smallest, roughly the 95th percentile. That number is q̂, the padding that would have been just enough for 95% of the calibration set. The disagreement-weighted rule makes two linked changes: clip to e_i⁺ = max{e_i, 0}, then divide by a_γ(X_i). The quantile is therefore taken over non-negative, disagreement-normalized errors.

• γ = 0 recovers a scale-free pure-expansion rule. The scale becomes 1. For a point-prediction base interval (ℓ = u), this is ordinary absolute-residual split conformal. For an interval base, it is clipped CQR. It has the same endpoints as signed CQR whenever the signed calibration quantile is non-negative and does not contract the base interval otherwise.
• Never below 1. A purely multiplicative scale like γ·d can hit zero and collapse the normalized score. This form is bounded below, so no example gets a degenerate scale.
• Identifiable one-parameter family. The natural two-parameter form $\sqrt{{\mathrm{\lambda }}_0+{\mathrm{\lambda }}_1{d}^2}$ is redundant: multiplying the scale by any constant is absorbed by q̂. Setting λ₀ = 1 removes that redundancy, and γ plays the role of λ₁/λ₀.

• Signed CQR scores can be negative. Standard CQR and our unscaled Mondrian variant may therefore contract an overconservative base interval. The weighted method specified here instead uses e⁺, so its q̂ is non-negative, greater disagreement cannot cause a stronger contraction, and an ordered base interval stays non-empty.
• Empirical scope. Every cross-dataset weighted final quantile was non-negative, so the non-negative specification produces the reported endpoints. The SEMF-derived SRED results are presented as empirical diagnostics rather than formal proposition instances.
• What is not claimed: coverage conditional on a specific x (impossible distribution-free, Barber et al., 2021), and no training-conditional claim. Prop 2 gives group-conditional coverage, the strongest honest middle ground.
• Pooling small strata is an engineering fallback, and we explicitly note that the group guarantee is lost when you do it.
• Some SRED rows are labelled “diagnostics”: results reusing saved SEMF outputs also reuse validation information for model selection and are therefore not claimed as formal instances of the propositions.

The baseline is the same fitted model under the standard calibration rule. In every paired comparison (the 60 runs above, and the chart below) the base predictor is trained once and its predictions are reused on both sides. Only the calibration rule applied on top changes:

• Marginal baseline: the standard recipe. One global q̂ from the whole calibration split, and every test example gets the same padding.
• Weighted / Mondrian (ours): the same predictions and the same calibration listings, but the padding is scaled by d(x), or computed per group.

That is what makes the 60 comparisons clean: same fitted model, same splits, same seed on both sides, so any difference is attributable to the calibration layer alone. The baseline "produces" intervals of the same shape and location as the method's. Only their widths differ.

The shortest way to say it: the marginal comparator removes the disagreement-dependent scale, while the Mondrian comparator removes the group split (one pooled q̂ instead of one q̂_h per group). There is one sign nuance. Our weighted specification clips scores at zero, so γ = 0 is the pure-expansion marginal rule. It coincides with the usual signed CQR comparator whenever the signed final quantile is non-negative (as it was in every cross-dataset weighted run), but unlike signed CQR it cannot contract when that quantile is negative.

Concretely, the nine rows of the SRED table are 3 trained base predictors × 3 calibration rules. Each trio of rows shares one fitted model (XGBoost point, augmented SEMF, or XGBoost quantile) and compares three ways of computing the padding from the same calibration errors: marginal (the baseline), disagreement-weighted, and disagreement-Mondrian. Mondrian is one of our two proposed methods, not a baseline. Whether a row says "split conformal" or "CQR" only reflects the base model's output format (point vs interval) and applies identically to baseline and method.

Two other comparison points appear once each and should not be confused with the main baseline: the pooled quantile in the missing-modality table (one quantile over all masked calibration examples together, a deliberately blunt sanity check), and the point-accuracy baselines (XGBoost on concatenated features, MLP, cross-attention) in the R² table, which exist to scope the point-prediction claim and play no role in the calibration claims.

Values are log(rent). CHF equivalents appear in the Overview example. Regenerated with the repository pipeline (run_predagn_ablation.py, SRED, seed 0, XGBoost point base). Aggregates of this rerun match the released seed-0 artifact to the second decimal. On this seed the weighted layer trims mean width from 0.523 to 0.513 log units while coverage moves 0.947 → 0.942 against the 0.95 target: five conflicted listings are newly covered, ten calm ones newly missed. Width moves to where the sources disagree. The guarantee remains marginal; it does not promise coverage within either subset of listings.

Mercari is nearly neutral on average (width 2.439378 → 2.439120 before rounding). The mechanism needs the source predictors to induce a reliability ordering. When that ordering is weak, d(x) carries little useful signal. This is not an automatic test-set no-harm guarantee. Six of the 15 Mercari runs became slightly wider, while 9 matched or improved. We report the nearly neutral average as an informative negative result.

The point-prediction table also shows that strong neural fusion leads on SRED, Mercari, and IMDB-WIKI, while homogeneous XGBoost leads on Pawpularity. The claim is deliberately about the calibration wrapper, not about winning point accuracy.