Conformal Calibration for Multi-Modal Regression with Missing Modalities

Abstract

Prediction intervals for multi-modal regression with tabular variables, text, images, or other input sources are difficult to calibrate when those sources disagree or one is missing. A single global quantile averages these regimes together instead of calibrating to the modality pattern observed at test time. We address this through a modality-aware conformal calibration layer. The layer trains or reuses one predictor per modality, computes a disagreement score from their predictions, and uses that score in split conformal calibration under a strict split protocol. We use the score in two complementary ways. First, a continuous disagreement-scaled method reallocates interval width across examples while preserving the usual marginal split-conformal guarantee. Second, a Mondrian (stratified) method calibrates within groups defined by disagreement or modality availability fixed before calibration, giving group guarantees under joint exchangeability of the calibration and test examples. Across four multi-modal datasets, the disagreement-scaled layer matches or improves the marginal conformal baseline in 59 of 60 paired runs for interval continuous ranked probability score (CRPS) and in 52 of 60 for interval width, while keeping empirical coverage near the 95% target. In stress tests with missing modalities, mask-matched recalibration recovers up to 19.5 percentage points of coverage in the hardest fixed-mask regime. The result is a simple, model-agnostic reliability layer for multi-modal regression systems.

Overview

2 min

Models that predict from tables, text, and images together usually get one global correction to turn their outputs into 95% prediction intervals. That single correction can allocate width inefficiently across levels of source disagreement and can become miscalibrated when the test-time modality pattern shifts because one source is missing. We add a small, model-agnostic calibration layer that measures how much the per-modality predictors disagree. Standard conformal machinery then turns that signal into intervals that put width where it is needed, without losing the formal coverage guarantee.

Example

Our running example throughout the page is Swiss apartment listings (SRED; Azizi and Rudnytskyi, 2022). Each listing brings three information sources: the table (size, rooms, location), the written description, and the photos plus a satellite view.

We hand-picked these four test listings after seeing the results, purely to illustrate the mechanism. They are not extra performance evidence.

Pick a listing below to see what happens to the intervals.

Tabular (all 4 fields)

Tabular fields for the selected listing
rooms
living space
latitude
longitude

Text (header and description)

Images (montage and satellite)

Photo montage of the listing, the image encoder's first input

Satellite view of the listing, the image encoder's second input

Both image fields are used. DINOv2 embeds the photo montage and the satellite view separately, and the two embeddings are mean-pooled into one image block.

Tabular model
Text model
Image model
Spread of the reads
d(x), log units
Width vs. marginal

Contributions

The formal coverage statements use the fixed-pipeline and exchangeability conditions stated precisely in § 6.

Disagreement-scaled intervals. The spread of the per-source reads, d(x), scales each listing's padding, so width moves to conflicted listings like the family flat above while retaining the usual marginal coverage guarantee.
Calibration by modality regime. One quantile per pre-declared group (full, no text, no image, tabular only) gives a finite-stratum coverage guarantee within each group; the fixed-mask stress tests show the corresponding empirical repair.
A model-agnostic layer. The deployed base predictor is never retrained or modified, the conformal machinery itself stays deliberately standard, and we state that scope explicitly.

The problem

1 min

The running example predicts log(rent) from the three sources introduced in the Overview. Standard practice, split conformal prediction (Vovk et al., 2005; Lei et al., 2018), computes one number, a quantile of the calibration errors, and pads every interval with it. That guarantees at least 95% coverage on average. But “on average” hides three very different regimes:

Sources agree. The global padding tends to over-cover these listings, so width is wasted there, though low disagreement alone does not make an example easy.
Sources conflict. Text says luxury, photos say renovation project. The global rule can under-cover this subgroup even while retaining its marginal guarantee.
A source is missing. Text, images, or both may be absent. In the hardest fixed-mask stress regime, IMDB-WIKI with only tabular inputs left, the full-regime quantile covers only about 76% of examples despite the nominal 95% target.

We ask whether we can keep the distribution-free guarantee while making the calibration aware of the modality situation of each test point.

The core idea: disagreement as a signal

2 min

Train one cheap auxiliary predictor per modality: g₁ sees only the table, g₂ only the text, g₃ only the images. Each predicts the rent alone. Then define the disagreement score d(x) as the standard deviation of the available source predictions, meaning how spread out their estimates are.

d(x) = sd{ ŷ₁, ŷ₂, …, ŷ_K }

Here sd uses the average across the K available predictions (the divisor is K), which gives the values shown below.

Sources agree → small d(x)

7.57 / 7.60 / 7.63 ⇒ d(x) = 0.02

All three views land close together, so this listing has low observed disagreement.

Sources conflict → large d(x)

6.85 / 7.60 / 7.90 ⇒ d(x) = 0.44

The image predictor sees something the others don't, so this listing has high observed disagreement. The calibration step in § 4 uses that signal.

Disagreement is a hint that a listing is hard, not proof that the prediction is right: the sources can agree and all be wrong. The signal earns its keep when bigger conflicts tend to come with bigger errors, and the coverage guarantee holds whether or not they do.

It is a useful calibration covariate because it is observable at test time (no labels needed), cheap, and model-agnostic. It works identically whether the main predictor is XGBoost (Chen and Guestrin, 2016), a neural network, or an SEMF-derived decoder-output ensemble. We feed d(x) into two standard conformal constructions, described in § 4 and § 5 and summarized by our overview figure.

Figure 1 of the paper. Top, the calibration layer: three per-modality predictors feed the disagreement score, a fixed base predictor supplies the interval, and one calibration step follows, disagreement-scaled CQR or Mondrian. Bottom, a worked example with three regimes. — Figure 1 of the paper. The worked example in its bottom band is the same one the live demo in § 4 reproduces, with the same quantiles and intervals.

What exactly does each model output? (the numbers d(x) is computed on)

Each auxiliary predictor g_k outputs one plain number: its own guess of the target, in target units. In the headline cross-dataset benchmark these are plain XGBoost regressors, one per declared source block (one block per modality there). For a SRED listing:

tabular tabular variables (size, rooms, location…) → g₁ → “log(rent) ≈ 7.60”
text description → frozen text encoder (E5 / MiniLM) → embedding vector → g₂ → “7.90”
image photos → frozen DINOv2 → embedding vector → g₃ → “6.85”

The raw text and pixels are never fed to g_k directly. A frozen pretrained encoder turns them into a feature vector first, and the regressor maps that vector to a rent guess. Then d(x) = sd{7.60, 7.90, 6.85} ≈ 0.44, in the same units as the target. Think of three appraisers, each reading only one part of the dossier, each writing down a rent estimate: disagreement is the spread of the three numbers. No labels, no intervals, no probabilities involved.

Keep two roles separate (a common confusion). The base predictor, the model being calibrated, is a different object. It outputs the interval (ℓ, u). That interval comes from a single point prediction ŷ (then ℓ = u = ŷ), from a pair of fitted 2.5% / 97.5% quantiles, or from an SEMF ensemble of latent-propagated decoder outputs summarized into an interval. The g_k's never produce the interval. They exist only to produce d(x).

Disagreement-scaled intervals

5 min

Take any base interval (ℓ(x), u(x)), from a quantile model or from a point prediction with ℓ = u. The quantile case is conformalized quantile regression (Romano et al., 2019). Instead of padding every example equally, scale the padding by a factor that grows with disagreement:

a_γ(x) =

\sqrt{1 + γ \cdot {d (x)}^{2}}

Calibrated interval = [ ℓ(x) − q̂·a_γ(x), u(x) + q̂·a_γ(x) ]

Here q̂ is the usual split-conformal quantile, but computed on non-negative normalized scores e_i⁺ = max{e_i, 0} and s_i = e_i⁺ / a_γ(X_i) (Papadopoulos et al., 2008). Clipping rules out a structural inversion in which a negative quantile would make greater disagreement cause a stronger contraction. The hyperparameter γ ≥ 0 is chosen on a separate tuning split and fixed before calibration. This keeps the same marginal coverage target while allowing padding to vary with disagreement. In our experiments, width generally moves from low-disagreement examples toward conflicted ones.

Why a base interval at all? Do my models have to output intervals?

No, they don't. Conformal prediction never invents a prediction. It corrects one. So the wrapper needs some prediction rule to correct, called the base predictor, and it accepts that rule in any of three forms:

A point predictor (plain XGBoost). Output: one number ŷ. We treat it as the degenerate interval ℓ = u = ŷ, so the conformal layer builds the entire interval around it: [ŷ − q̂·a_γ, ŷ + q̂·a_γ].
A quantile predictor (XGBoost trained at the 2.5% and 97.5% levels). Output: a rough interval. Fitted quantiles carry no guarantee. Standard signed CQR may stretch or shrink them; our disagreement-scaled rule clips scores at zero and therefore only expands them.
An ensemble-output base (the SEMF-derived diagnostic). It samples latent variables and retains 50 decoder outputs, whose empirical quantiles form a rough interval. These are latent-propagated decoder outputs, not draws from the full response distribution, because the cached run omitted SEMF's final response draw.

The base predictor supplies the interval's location and starting shape (where it sits, whether it is asymmetric, whether it is already wider in noisy regions). The conformal layer adds a calibrated correction, and the corrected interval carries a guarantee of at least 95% marginal coverage, under the assumptions laid out in the validity section below. A quantile or ensemble-output base is worth having because it adapts the shape to each x. With a point base, all the width has to come from the padding. The experiments use all three base-output types, so the same wrapper is exercised across point, quantile, and ensemble bases.

Is the base predictor a separate model, and must it be multi-modal? It is separate because it is the model you would deploy anyway; the conformal calibration wrapper treats its fitted output rule as fixed and does not modify it. The theory permits any fixed base predictor. No fusion architecture is required. One headline base predictor (used in the cross-dataset missing-modality stress test) is XGBoost on all concatenated features (the tabular columns and cached embeddings side by side as one wide table). The auxiliaries must instead be per source block, because d(x) is the spread of their predictions; a source block may be a semantic modality or a separately modeled field within one. In the reported missing-modality experiment the base consumes all modalities, since the measured failure is a model that relied on a source suddenly receiving a zeroed feature block. A tabular-only base would be unaffected by masking text or images, so it would not measure that failure mode.

One listing end to end (who produces what)

Illustrative numbers. SRED, with an XGBoost point predictor as the base.

Fit on the fit split (6,498 listings). The base predictor is one XGBoost trained on all features together: the tabular columns plus the cached text embedding plus the cached image embedding. The auxiliaries g₁, g₂, g₃ are three plain XGBoost regressors trained on one modality each. Early stopping is monitored on the separate tuning split.
Calibrate once, on the calibration split (1,999 listings). The base predictor makes a rent guess ŷ_i for each calibration listing. The error is e_i = |y_i − ŷ_i|. The marginal recipe takes roughly the 95th percentile of the errors, say q̂ = 0.40. The disagreement-scaled recipe first divides each error by that listing's a_γ(X_i), then takes the quantile, giving say q̂ = 0.31 (smaller, because dividing by a scale ≥ 1 shrinks the scores).
Predict for a new listing. 3.5 rooms, 80 m², Lausanne, five photos. Base predictor: ŷ = 7.62. One number, no interval. Auxiliaries: 7.60 / 7.90 / 6.85, so d = 0.44 and a_γ = $\sqrt{1 + 8 \cdot {0.44}^{2}}$ = 1.60.
Build the interval. Baseline (marginal): 7.62 ± 0.40 → [7.22, 8.02], the same padding as every other listing. Scaled: 7.62 ± 0.31·1.60 = 7.62 ± 0.50 → [7.12, 8.12], wider because the sources conflict. A peaceful listing with d = 0.02 would instead get 7.62 ± 0.31, narrower than the baseline's ± 0.40.

Notice what the base predictor produced: only the number 7.62. Everything interval-shaped came from the calibration layer. The baseline and disagreement-scaled method use the same fitted model, prediction, and calibration listings; they differ in score normalization, the resulting quantile, and the padding applied to each test point.

Model count for this example with a point base. With 3 modalities, four regressors are trained: three auxiliaries (one per modality, each outputting a point guess) and one base predictor on all modalities. More generally the wrapper needs K auxiliary prediction rules plus one base prediction system; a quantile system can itself contain several fitted quantile models. Nothing else in the layer is trained: q̂ is a sorted quantile of calibration errors, γ comes from a small grid search on the tuning split, and the Mondrian groups are just labels.

What is e exactly? (the error that q̂ is a quantile of)

It is the letter e, not an epsilon, and it is not noise. For one calibration listing, e_i answers a single question: how far did the true label land outside the base interval?

e_i = max{ ℓ(X_i) − Y_i, Y_i − u(X_i) }

With base interval [7.40, 7.80] and true log(rent) y:

Truth above it, y = 8.05 → e = 0.25. The interval missed by 0.25 on top.
Truth below it, y = 7.10 → e = 0.30. Missed by 0.30 underneath.
Truth inside it, y = 7.60 → e = −0.20. Negative means covered with slack to spare.
Point predictor (ℓ = u = ŷ) → e = |y − ŷ|, the plain absolute error.

Compute e_i for every calibration listing (their true rents are known), sort the values, and take the ⌈(n+1)(1−α)⌉-th smallest, roughly the 95th percentile. That number is q̂, the padding at a rank just above 95% of the calibration set. The extra notch (the n+1 in the rank) is what makes the guarantee hold for a fresh listing rather than just describing the calibration data. The disagreement-scaled rule makes two linked changes: clip to e_i⁺ = max{e_i, 0}, then divide by a_γ(X_i). The quantile is therefore taken over the non-negative, normalized errors.

The worked example from Figure 1

Base interval [7.40, 7.80], γ = 8, with Figure 1's quantiles of 0.25 global and 0.20 for normalized non-negative scores.

Drag the disagreement d(x)

d(x)
0.02 Scale a_γ
1.00 Calibrated interval
[7.20, 8.00] Width
0.80 Width vs. global
−11%

The dashed teal line is the base interval from the model. The faint gray bar is the global CQR interval, whose raw-score quantile is 0.25. The violet bar is the disagreement-scaled interval, built from the smaller normalized-score quantile 0.20. These are Figure 1's numbers: at d = 0.02 the scaled interval is [7.20, 8.00], 11% narrower than the global one, and at d = 0.44 it is [7.08, 8.12], about 16% wider (displayed width 1.04 versus 0.90 for the global interval). "Width vs. global" is a per-example comparison, not an improvement score. Positive means this example's interval is wider than the one-size-fits-all interval, a cost paid deliberately where the sources conflict. The method's gain lives in the average, where width moves to conflicted examples and the mean width and CRPS improve while empirical coverage stays near the 95% target.

Why the form

\sqrt{1 + γ \cdot d^{2}}

γ = 0 recovers a rule that only expands. The scale becomes 1. For a base interval from a point prediction (ℓ = u), this is ordinary split conformal on absolute residuals. For an interval base, it is clipped CQR. It has the same endpoints as signed CQR whenever the signed calibration quantile is non-negative and does not contract the base interval otherwise.
Never below 1. A purely multiplicative scale like γ·d can hit zero, collapsing the scale and making score normalization undefined. This form is bounded below, so no example gets a degenerate scale.
Identifiable one-parameter family. The natural two-parameter form $\sqrt{λ_{0} + λ_{1} d^{2}}$ is redundant: multiplying the scale by any constant is absorbed by q̂. Setting λ₀ = 1 removes that redundancy, and γ plays the role of λ₁/λ₀.

Group-wise Mondrian calibration

4 min

Mondrian coverage in simple words. Ordinary conformal coverage is a marginal probability promise: “at least 95% of future listings land inside their intervals.” That promise can hide a failing group, the way a school can boast a 95% pass rate overall while one class passes only 75% of its students. Mondrian coverage upgrades the promise: name your groups in advance (full, no text, no image, tabular only) and guarantee at least 95% inside each group separately. The recipe is almost disappointingly simple. Run the usual calibration once per group, so each group's padding comes from its own peers. A no-photo listing is padded like other no-photo listings, not like the average listing. The name comes from the Mondrian taxonomies of Vovk et al. (2005); the labelled boxes evoke Piet Mondrian's grids.

Disagreement scaling keeps a marginal guarantee. Sometimes a stakeholder needs coverage for a named group. For example, listings without photos may still need 95% coverage. For that you need a stronger construction (Vovk et al., 2005; Vovk, 2012; Boström and Johansson, 2020). Split the calibration set into pre-declared groups and compute one conformal quantile q̂_h per group. We use two grouping rules:

H_dis: low / mid / high disagreement bins (edges fixed on the tuning split);
H_reg: the modality availability regime, meaning full, no text, no image, or tabular only.

Missing modalities are the headline use, not the only one. The same construction repairs the high-disagreement bin (marginal calibration averages 0.923 coverage there; per-bin calibration brings it to about 0.948), and another diagnostic applies it to bins of the decoder-output ensemble's own predicted uncertainty. In that weaker diagnostic, both methods remain near the target and the differences are small relative to descriptive variation across the five seeded runs. This is not an inferential comparison. Its bin edges are estimated from the calibration half's predicted widths, so it is a post-hoc diagnostic rather than a formal instance of Proposition 2. The formal recipe requires a group rule fixed before calibration, a label observable at test time without the outcome, enough in-group calibration examples for a finite interval, and joint exchangeability of the calibration and test examples conditional on the fixed pre-calibration pipeline. Care about a named group, use Mondrian. Care about smooth average efficiency, use the disagreement scale.

Here is the fixed-mask missing-modality stress test end to end. It is discrete (a modality is there or it isn't), which is why this demo has switches rather than a slider. Each displayed state applies one mask to every calibration and evaluation example in that run:

The frozen base predictor still runs after the absent modality's embedding block is filled with zeros (the stress-test convention). In Table 2 the base predictor produces a point prediction, and the conformal calibration step supplies the interval around that point.
No disagreement value is invented for the absent source. The selected mask is recorded, e.g. “no image”.
Every calibration example receives that same mask. Their errors are typically larger, and the mask-matched quantile is therefore larger.
Every evaluation interval uses padding calibrated under that mask. A no-photo listing is padded using errors from no-photo calibration listings, not full-modality errors.

Our cross-dataset stress table implements this as eight separate fixed-mask experiments: each row applies one deterministic mask to every calibration and test example in that run. Thus the label is constant within a row: mask-matched recalibration is the Mondrian special case with a constant stratum label, equivalent to ordinary split conformal on the masked distribution rather than a nontrivial partition of one sample with mixed missingness. A deployment with several naturally occurring patterns would need representative calibration examples from each pre-declared pattern under the same fixed pipeline.

A useful analogy is measuring with one eye closed. You can still see, but less precisely, so an honest margin of error must be larger. The layer finds out how much larger by rehearsing. It closes the same eye on the calibration listings, where the true rents are known, measures how much worse the guesses get, and turns exactly that damage into the new padding.

Knock out a modality

Schematic illustration only. The base interval and paddings shown here are illustrative, not fitted values. Table 2 itself uses a point predictor with one residual quantile fitted per mask.

Available at test time:

Applied mask
Full Global interval
[7.20, 8.00] Its real coverage
94.8% Mask-matched interval
[7.20, 8.00] Its real coverage
94.8%

All modalities are present, so the mask-matched quantile equals the global one, because the full mask is exactly what the global quantile was calibrated on.

The mask-state coverage numbers are the final five-seed SRED means underlying Table 2. Its three masked states are printed in the table; the full state uses the corresponding unmasked full-regime mean (94.8%, rounded). The displayed base interval and padding sizes are illustrative, with masked paddings drawn to scale with the reported width changes (+21%, +41%, +71%).

How the two methods divide the work: disagreement-scaled = continuous, aimed at efficiency, marginal guarantee. Mondrian = discrete, aimed at the guarantee, per-group validity. They are complementary, not competing, and we evaluate them in those distinct roles.

Understanding the propositions

4 min

For a reader who wants to inspect the math, there are exactly two propositions. Both are standard split-conformal exchangeability arguments applied to the new scores. What we add is the careful statement of when they apply.

Proposition 1 · scaled-score validity

Claim: the disagreement-scaled interval covers with probability ≥ 1−α, marginally, conditional on everything fixed before calibration.

Needs: (a) base interval, source predictors, availability rule, pre-processing, and γ all fixed before the calibration set is touched; (b) A(X) ≠ ∅ for every calibration and test example, so d can be evaluated; (c) calibration and test points exchangeable conditional on 𝒟_pre.

Proof shape: conditioning makes all functions fixed → the n+1 normalized scores are exchangeable → the test score exceeds the k-th order statistic (k = ⌈(n+1)(1−α)⌉) with probability ≤ α. Rank argument, nothing exotic.

Proposition 2 · per-group validity

Claim: the Mondrian interval covers ≥ 1−α within each group h, a genuinely conditional statement.

Needs: (a) the group rule H fixed using only pre-calibration data; (b) all n+1 calibration and test examples jointly exchangeable conditional on 𝒟_pre; (c) the named group has positive probability conditional on 𝒟_pre; (d) enough calibration points in the group, otherwise q̂_h = +∞ (an honest infinite interval); (e) A(X_i) ≠ ∅ whenever evaluating H or the chosen score requires d.

Proof shape: condition on the full membership vector. Because the fixed membership event is invariant to permutations within stratum h, joint exchangeability implies exchangeability of the in-stratum scores. Apply the rank argument there, then average over membership vectors.

Two conditions do the real work. First, every data-driven choice is fixed before calibration. Second, calibration and test examples are exchangeable conditional on that fixed pipeline. The four-way split, fit (fit the base and source learners) → tune (monitor early stopping, fit stacking or gating weights when used, and fix γ, bin edges, and every remaining data-driven decision) → calibration (scores and quantiles only) → test, enforces the first condition but does not establish the second. We collect everything fixed before calibration into 𝒟_pre and condition on it, which is what makes separate tuning of γ compatible with the rank argument.

Subtleties worth knowing before someone finds them

Signed CQR scores can be negative. Standard CQR and our unscaled Mondrian variant may therefore contract an overconservative base interval. The disagreement-scaled method specified here instead uses e⁺, so its q̂ is non-negative, greater disagreement cannot cause a stronger contraction, and an ordered base interval stays non-empty.
Empirical scope. The same non-negative score is used for scale selection and calibration, and the per-run outputs record the selected scales and quantiles. The SEMF-derived SRED results are presented as empirical diagnostics rather than formal proposition instances.
What is not claimed: coverage conditional on a specific x (impossible distribution-free, Barber et al., 2021) or conditional on the realized calibration scores. The propositions do condition on 𝒟_pre, exactly as stated; Proposition 2 additionally gives a finite-stratum group-conditional guarantee.
Pooling small strata changes the stated construction. If it is used as an engineering fallback, the guarantee conditional on the original stratum is lost.
Some SRED rows are labelled “diagnostics”: results reusing saved SEMF outputs also reuse validation information for model selection and are therefore not claimed as formal instances of the propositions.

Results

4 min

Four multi-modal regression datasets, three headline base predictors plus a ridge stacker as a robustness check, five seeds, and frozen pretrained encoders (DINOv2 for images, E5 / MiniLM for text): SRED (Swiss rents, tab+text+img; Azizi and Rudnytskyi, 2022), Mercari (product prices, tab+text; Mercari, 2018), Pawpularity (pet-photo popularity, tab+img; PetFinder.my, 2021), IMDB-WIKI (age from face+name, tab+text+img; Rothe et al., 2018).

59 / 60

interval CRPS: match or improve

46 strict improvements, 13 exact ties, 1 loss

52 / 60

interval width: match or improve

39 strict improvements, 13 exact ties, 8 losses

+19.5 pp

biggest coverage repair under a missing modality

IMDB-WIKI tabular only: 0.758 → 0.953

The headline counts aggregate three base predictors per dataset and five seeds. Overall mean PICP changes only from 0.94943 to 0.94899. Adding a fourth ridge stacker base as a robustness check gives 75/80 non-inferior CRPS comparisons and 68/80 non-inferior width comparisons.

What is “the baseline” in these comparisons?

The baseline is the same fitted model under the standard calibration rule. Every one of the 60 headline comparisons and the 80-comparison robustness check compares disagreement-scaled with marginal calibration. The base predictor is trained once and its predictions are reused on both sides. Only the calibration rule applied on top changes:

Marginal baseline: the standard recipe. One global q̂ from the whole calibration split, and every test example gets the same padding.
Disagreement-scaled (ours): the same predictions and calibration listings, but the padding varies through d(x). These are the pairs counted above.
Mondrian (ours): it likewise reuses the fixed predictions and calibration listings, but computes q̂_h per group. Its evaluations are separate from the 60/80 disagreement-scaling counts.

That is what makes the paired comparisons clean: same fitted model, same splits, same seed on both sides, so any difference is attributable to the calibration layer alone. The baseline "produces" intervals of the same shape and location as the method's. Only their widths differ.

The shortest way to say it: the marginal comparator removes the scale that varies with disagreement, while the Mondrian comparator removes the group split (one pooled q̂ instead of one q̂_h per group). There is one sign nuance. Our disagreement-scaled specification clips scores at zero, so γ = 0 is the marginal rule that only expands. It coincides with the usual signed CQR comparator whenever the signed final quantile is non-negative (as it was in every cross-dataset disagreement-scaled run), but unlike signed CQR it cannot contract when that quantile is negative.

Concretely, the nine rows of the SRED table are 3 trained base predictors × 3 calibration rules. Each trio of rows shares one fitted model (XGBoost point, augmented SEMF, or XGBoost quantile) and compares three ways of computing the padding from the same calibration errors: marginal (the baseline), disagreement-scaled, and disagreement-Mondrian. Mondrian is one of our two proposed methods, not a baseline. Whether a row says "split conformal" or "CQR" only reflects the base model's output format (point vs interval) and applies identically to baseline and method.

Two other comparison points appear once each and should not be confused with the main baseline: the pooled quantile in the missing-modality table (one quantile over all masked calibration examples together, a deliberately blunt sanity check), and the point accuracy baselines (XGBoost on concatenated features, MLP, cross-attention) in the R² table, which exist to scope the point prediction claim and play no role in the calibration claims.

Paired results across three base predictors (Table 1: marginal → scaled, five seeds)

Per-dataset paired results for marginal and disagreement-scaled calibration
Dataset	PICP	MPIW	Interval CRPS	Match/improve (MPIW, CRPS)
SRED (Tab+Txt+Img)	0.943 → 0.942	0.714 → 0.689	0.103 → 0.102	15/15 · 15/15
Mercari (Tab+Txt)	0.950 → 0.950	2.439 → 2.439	0.373 → 0.373	9/15 · 15/15
Pawpularity (Tab+Img)	0.953 → 0.952	84.426 → 83.020	12.791 → 12.717	14/15 · 14/15
IMDB-WIKI (Tab+Txt+Img)	0.952 → 0.951	40.616 → 40.464	6.489 → 6.478	14/15 · 15/15

Each row averages over three base predictors (XGBoost point, XGBoost quantile, source-wise quantile) and five seeds. Arrows compare marginal → disagreement-scaled calibration for the same fitted base predictor and seed. The last column counts runs where disagreement-scaled calibration matches or improves the metric. A fourth ridge stacker of per-source point predictions is included in the 80-comparison robustness check.

Four real listings from the test split

The same four listings as in the Overview, now picked by outcome. Gray is the marginal interval, violet the disagreement-scaled one, and the dashed line the realized rent (SRED, seed 0, XGBoost point base).

Real rent
Tabular read
Text read
Image read
d(x), log units

The same four listings as a table, with provenance

Four real SRED test listings with source predictions and calibrated intervals
Listing	Rooms	m²	ŷ tab	ŷ text	ŷ image	d(x)	Real rent	Marginal	Scaled

Values are log(rent). CHF equivalents appear in the Overview example. The listing rows were regenerated with the finalized pipeline (run_predagn_ablation.py, SRED, seed 0, XGBoost point base), and exact digits drift slightly across machines, which is why the repository ships the embedding caches. In the released seed-0 run the disagreement-scaled layer trims mean width from 0.528 to 0.508 log units while coverage moves 0.946 → 0.937 against the 0.95 target, and width moves to where the sources disagree. The guarantee remains marginal and does not promise coverage within either subset of listings.

The headline chart: coverage repair when a modality goes missing

The same trained model is evaluated under a fixed missing-modality mask. Reusing its full-regime quantile (gray) can fall far below the 95% target when the masked modality carried information; recalibrating on calibration examples under that same mask (violet) raises empirical coverage toward the target. Hover or focus a chart row, or use the table below, for exact values and the width cost.

One global quantile Mask-matched recalibration ┄ 0.95 target

Coverage before and after is the final five-seed Table 2 result. The repair is not free. The hardest regimes pay real width (SRED tabular-only +71.2%, IMDB-WIKI tabular-only +125.0%). The larger intervals are the price of recalibrating for masked inputs that carry less information. The full-regime quantile was never calibrated for that situation.

Numbers as a table (Table 2: all eight stress regimes)

Coverage and width changes for all eight missing-modality stress regimes
Mask	Global PICP	Pooled PICP	Mask-matched PICP	Gain	Width cost

Pooled = one quantile over the scores produced by every distinct non-full mask, excluding the unmasked regime. For SRED and IMDB-WIKI this stacks three dependent masked copies of each calibration example; for the two-source datasets there is only one non-full mask, so pooled and mask-matched coincide. Gain (mask-matched vs. global) is our five-seed mean of unrounded per-seed changes, so it can differ by 0.1 pp from a value recomputed from the rounded coverage columns.

Where the method does nothing, and why that's fine

Mercari is nearly neutral on average (width 2.439378 → 2.439120 before rounding), despite heterogeneous behavior across disagreement bins. This is not an automatic promise of no harm on the test set. Six of the 15 Mercari runs became slightly wider, while 9 matched or improved. We report the limited mean efficiency gain as an informative negative result.

The point prediction table also shows that among the multi-modal fusion models, the neural baselines trained with MSE lead on SRED, Mercari, and IMDB-WIKI, while concatenated XGBoost leads on Pawpularity. The overall Pawpularity leader in that table is the image-only solo model (R² 0.240 against 0.234 for concatenated XGBoost). The claim is deliberately about the calibration wrapper, not about winning point accuracy. These neural baselines pool fit and tuning examples for pre-processing, target standardization, and parameter training, then use a seeded random 90/10 internal train/validation split of that pool for early stopping; calibration and test examples remain untouched.

Limitations and scope

2 min

What we claim, and the caveats we disclose.

Validity and evaluation scope

The finite-sample guarantees require every learned or tuned choice to be fixed before calibration, followed by jointly exchangeable calibration and test examples. The cross-dataset benchmark uses separate fit, tune, calibration, and test splits.
The four-way split enforces data separation but does not establish exchangeability. We find that 37 of 1,109 SRED test rows exactly match a training row on latitude, longitude, living space, room count, and recorded rent, and that 43 of 3,922 IMDB-WIKI test rows share a normalized non-empty name with the training split. These overlaps may weaken the plausibility of row-level exchangeability.
Calibration groups need enough data. At a 95% target, each protected Mondrian group needs at least 19 calibration examples even to form the finite split-conformal quantile.
The missing-modality experiments zero the masked modality's feature block in calibration and evaluation. They are controlled stress tests, not evidence about naturally occurring or informative missingness.

Method limits and next steps

The scalar disagreement score gives each available source equal weight. It can miss cross-modal dependence and differences in source reliability, and correlated or substitute predictors can agree while all are wrong.
The disagreement score's behavior depends on the number and redundancy of the available sources. We therefore keep the source set fixed in the continuous experiments and handle missing modalities with a separate regime rule.
Fixing γ = 1 on a pre-specified standardized scale removes only the dedicated step of selecting the scale. Tuning the base predictor remains, while richer learned scales require sufficient independent tuning data. We use a simulation in the paper to quantify when spending labels on tuning the scale is worthwhile.
Priorities for future work are cross-fitting, disagreement weighted by source reliability, systematic studies of source count and redundancy, natural missingness, and corruption at deployment.

Questions & answers

What exactly is the contribution? It looks like standard conformal prediction.

The contribution is the use of source-wise disagreement and modality availability as explicit calibration variables for multi-modal regression, together with their continuous and stratified constructions and the empirical evaluation across four datasets. The validity statements follow standard split-conformal rank arguments under the fixed pipeline and exchangeability assumptions. Related work by Bose et al. (2024) constructs conformal intervals from internal neural features; this layer instead works with source-wise predictions from XGBoost, neural nets, or SEMF alike, and makes availability regimes explicit.

Walk me through the proof of Proposition 1.

Three moves: (1) condition on 𝒟_pre, meaning everything fitted or tuned beforehand, so ℓ, u, d, a_γ are fixed functions. (2) the n calibration scores and the test score s_n+1 = e⁺/a_γ are then exchangeable, so the standard rank bound for exchangeable scores applies. (3) coverage fails only if the test score exceeds the k-th smallest calibration score with k = ⌈(n+1)(1−α)⌉, which has probability at most α. Ties only make it conservative. This is the standard split-conformal argument (Lei et al., 2018).

γ is tuned on data. Doesn't that invalidate the guarantee?

No: γ is selected on a separate tuning split and fixed before the calibration set is ever scored. That is exactly what conditioning on 𝒟_pre formalizes. What would break it: tuning γ on the same labels used for q̂. We state this explicitly.

Why

\sqrt{1 + γ d^{2}}

and not something simpler like γ·d?

Three reasons. At γ = 0, the scale is 1 and the method recovers the clipped marginal rule (the same endpoints as signed CQR whenever its quantile is non-negative). The scale is bounded below by 1, so no example gets a collapsing normalization. Finally, it is the identifiable parametrization of $\sqrt{λ_{0} + λ_{1} d^{2}}$ : multiplying the whole scale by a constant is absorbed by q̂, so fixing the intercept at 1 removes a redundant degree of freedom.

What happens when a group has almost no calibration points?

If m_h = ⌈(n_h+1)(1−α)⌉ exceeds n_h, the quantile is +∞, an honest infinite interval rather than a fake finite one. If an application instead pools small groups as an engineering fallback, that changes the protected strata and forfeits the stated guarantee conditional on the original group.

Doubling the interval width to fix coverage: is that a win?

The narrow intervals were never calibrated for masked inputs: in the hardest fixed-mask regime they covered only about 75.8% against the 95% target. Width is the honest price of missing information. And the blunt alternative (one pooled quantile over all represented non-full masks) over-pays where it isn't needed. IMDB-WIKI no-text gets inflated to 99.8% coverage at about twice the mask-matched width, while mask-matched recalibration leaves that easy fixed mask nearly unchanged.

How does this relate to SEMF?

SEMF (Azizi et al., 2025) is the origin story and one of the base predictors, but the layer is model-agnostic. The cached runs retain 50 latent-propagated decoder outputs per example. They omit SEMF's final response draw and are therefore not samples from the full predictive distribution. The reported SEMF-derived results use five checkpoints generated after correcting the alignment between non-constant E-step weights and replicated training rows. The checkpoints and their downstream tables and figures were checked by the reproducibility pipeline; no reported result uses the legacy ordering. No conformal claim depends on this internal training detail.

Did you train a separate model for each missing-modality case? What does this cost?

No retraining. One full-modality base predictor is trained per dataset and seed. Each Table 2 row is a separate fixed-mask experiment. The absent modality's embedding block is zeroed for every calibration and test example in that run, and the model itself is never touched. The corresponding calibration errors are then sorted once to obtain that mask's quantile. The encoders are frozen and their embeddings cached; after predictor fitting, the calibration layer itself consists mainly of rescaling scores and taking quantiles.

Where does the method fail?

When source disagreement is not a stable guide to error size, d(x) offers little useful signal. Mercari shows limited mean efficiency gain despite heterogeneous bin behavior, and individual runs can still become slightly wider. The score is also a scalar from point predictions, so it can miss richer cross-modal structure. Learned modality-aware scales are a natural direction for future work.

Is it assumed that the per-modality models are good? What if one gives bad predictions?

No accuracy assumption on the auxiliaries is needed for validity. Once they and γ are fixed before calibration, they define a fixed score function and the usual rank argument applies. Their quality does affect sharpness. A wild source prediction raises d(x) and the local scale, but finite tuning and calibration do not turn this into an efficiency guarantee on the test set. The tuning grid includes γ = 0, so the tuner can always select the marginal rule, though that is not an automatic no-harm switch. What helps is an informative ordering. Examples with larger source disagreement should tend to land farther outside the base interval.

What happens if the modalities disagree systematically, everywhere?

If disagreement is constant, the disagreement-scaled method is algebraically identical to the clipped marginal rule. With constant scale A, every normalized score is e_i⁺/A, its calibrated quantile is the marginal quantile of the non-negative scores divided by A, and multiplying by A restores exactly the same padding. More generally, multiplying the entire scale function by a constant is absorbed by q̂. Rescaling d itself is instead absorbed by retuning γ. The method can improve efficiency only when variation in disagreement usefully tracks variation in error size.

How much do the trained models actually matter?

Two separate levers. The base predictor is the sharpness lever: in the SRED diagnostic, mean interval width across the three calibration rules is 1.827 for the XGBoost point base, 1.791 for the augmented SEMF base, and 2.165 for the XGBoost quantile base, while switching the calibration rule within a fixed base moves mean width by only 0.065 to 0.093. The calibration layer is the honesty lever: model quality alone gives no distribution-free 95% coverage guarantee. In the reported stress test, applying a full-regime quantile in the hardest IMDB-WIKI fixed-mask regime yields only 0.758 coverage (0.76 rounded) against the 0.95 target. No calibration rule can rescue a hopeless model's width. The auxiliaries matter through the reliability ordering their disagreement induces, not through point accuracy alone.

Why are there no significance tests, and when would Bonferroni be relevant?

The coverage results need no hypothesis test. The propositions are finite-sample guarantees under their stated exchangeability conditions. For efficiency, we report paired counts descriptively rather than attach naïve tests across repeated seeds. Base predictors within a dataset–seed cell share splits and source predictions, and five seeds from each of four datasets are not independent draws of datasets; Bonferroni does not repair that dependence. A different use of the union bound applies when protecting a pre-specified batch of B future intervals. Calibrating each at miscoverage α/B controls the probability of any batch error by α without assuming independence. Using α/|ℋ| is only the special case of one future interval per stratum. Ordinary per-stratum validity itself requires no multiplicity correction.

How does the number of modalities K matter, and what about substitute modalities?

Mechanically, one source departing from the rest by Δ produces d = Δ· $\sqrt{K - 1}$ /K, which is 0.50Δ, 0.47Δ, and 0.43Δ for K = 2, 3, and 4. This mild dilution as K grows can be absorbed by retuning γ. With K = 1, d(x) is zero and the disagreement-scaled rule reduces to its γ = 0 behavior; with K = 0, disagreement is undefined and a deployment needs a pre-specified fallback or abstention rule. What matters is the information structure, not only the count. Highly similar source predictions add little variation to d, while distinctive sources can create informative conflicts. Mercari's nearly neutral average reflects limited mean efficiency gain despite heterogeneous bin behavior; it does not show that its source blocks are redundant. Likewise, the masking stress test measures sensitivity to those artificial masks, not a general causal measure of modality substitutability. With large K, availability regimes can multiply up to 2ᴷ and leave fewer calibration examples per protected regime.

Can intervals cross or overlap?

Three separate cases. Fitted base quantiles can cross. The cross-dataset pipeline sorts each fitted quantile triple as part of the fixed rule before calibration. The SRED diagnostic retained its fitted quantile pair without rearrangement, but its final endpoints were checked to be ordered; a crossed calibration pair still defines a valid score and tends to inflate it. The disagreement-scaled interval specified here cannot cross when the base interval is ordered, because e⁺ makes q̂ non-negative and padding only expands. Contraction from a negative q̂, including a possible empty endpoint representation, belongs to signed CQR or the unscaled signed Mondrian variant, whose set-based definition remains valid. Intervals for different examples may overlap without issue, and each Mondrian example receives exactly one pre-declared group quantile.

Does the idea extend to classification?

Our paper is regression only (a real-valued target and intervals), but both methods have natural classification analogues. Conformal classification replaces intervals with label sets. Score each class with, for example, one minus its predicted probability, and under the usual conformal assumptions the calibrated set covers the true label at the target rate. The Mondrian construction carries over unchanged. Calibrate label sets within each modality availability regime, exactly as in the regression case. The disagreement score needs a new definition of d(x), because per-modality classifiers output probability vectors rather than one number. Natural candidates are the spread of per-modality probabilities for the predicted class, an entropy or divergence between the modality distributions, or simply the disagreement rate among modality votes. Working that out, and showing it helps, would be new material: a natural extension, not a claim of our paper.

Pocket glossary

Conformal prediction: Distribution-free recipe turning any predictor plus held-out data into intervals with finite-sample coverage, provided the rules are fixed before calibration and the data are exchangeable.
Split conformal: The practical variant: fit on one split, compute error quantile q̂ on a calibration split, pad test predictions with it.
Exchangeability: The only distributional assumption: calibration and test examples' joint law is order-invariant (weaker than i.i.d.).
CQR: Conformalized quantile regression (Romano et al., 2019). Base model outputs quantiles (ℓ, u). Score e = max{ℓ−y, y−u} measures how far outside you fell.
Non-conformity score: The per-example error the quantile is taken over. The disagreement-scaled score here is e⁺ / $\sqrt{1 + γ d^{2}}$ , where e⁺ = max{e, 0}.
Mondrian CP: Separate calibration quantile per pre-declared category. The name comes from Vovk et al. (2005); the per-group validity result used here follows Vovk (2012).
PICP / MPIW: Empirical coverage (want ≈ 0.95) / mean interval width (want small, given coverage).
Interval CRPS: Proper score combining coverage and sharpness in one number, computed as a uniform distribution over the interval (Gneiting et al., 2007).
Marginal vs conditional: Guaranteed on average vs guaranteed for each x. Exact conditional is impossible distribution-free. Groups are the honest middle.
SRED: Swiss Real Estate Dataset (Azizi & Rudnytskyi, 2022): ≈11,000 Swiss apartment rental listings with tabular attributes, description text, and photos. The target is log monthly rent.
SEMF: Supervised Expectation-Maximization Framework (Azizi et al., 2025). The cached diagnostic used here retains latent-propagated decoder outputs, not samples from the full predictive response distribution.
NCIW: Normalized calibrated interval width (Azizi et al., 2026): a post-hoc diagnostic that applies the smallest common scaling factor reaching at least the target empirical coverage, then normalizes the resulting width by the observed target range (max minus min of the test labels). It peeks at test labels, so it is not a deployable metric.
𝒟_pre: Everything decided before calibration (models, γ, bin edges). The propositions hold conditional on it.

References

Selected works this page draws on; the paper has the full bibliography.

Angelopoulos, A. N. and Bates, S. (2023). Conformal prediction: A gentle introduction. Foundations and Trends in Machine Learning, 16(4):494–591. [link]

Azizi, I. and Rudnytskyi, I. (2022). Improving real estate rental estimations with visual data. Big Data and Cognitive Computing, 6(3):96. [link]

Azizi, I., Boldi, M.-O., and Chavez-Demoulin, V. (2025). SEMF: Supervised expectation-maximization framework for predicting intervals. In Proceedings of the Fourteenth Symposium on Conformal and Probabilistic Prediction with Applications, PMLR 266:250–281. [link]

Azizi, I., Bodik, J., Heiss, J. M., and Yu, B. (2026). CLEAR: Calibrated learning for epistemic and aleatoric risk. In International Conference on Learning Representations, 157741–157813. [link]

Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, R. J. (2021). The limits of distribution-free conditional predictive inference. Information and Inference: A Journal of the IMA, 10(2):455–482. [link]

Bose, A., Ethier, J., and Guinand, P. (2024). Conformal prediction for multimodal regression. arXiv:2410.19653. [link]

Boström, H. and Johansson, U. (2020). Mondrian conformal regressors. In Proceedings of the Ninth Symposium on Conformal and Probabilistic Prediction and Applications, PMLR 128:114–133. [link]

Chen, T. and Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794. [link]

Gneiting, T., Balabdaoui, F., and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B, 69(2):243–268. [link]

Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523):1094–1111. [link]

Mercari, Inc. (2018). Mercari Price Suggestion Challenge. Kaggle competition. [link]

Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2024). DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research.

Papadopoulos, H., Gammerman, A., and Vovk, V. (2008). Normalized nonconformity measures for regression conformal prediction. In Proceedings of the IASTED International Conference on Artificial Intelligence and Applications, 64–69.

PetFinder.my (2021). PetFinder.my Pawpularity Contest. Kaggle competition. [link]

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of EMNLP-IJCNLP 2019, 3982–3992. [link]

Reimers, N. and Gurevych, I. (2020). Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of EMNLP 2020, 4512–4525. [link]

Romano, Y., Patterson, E., and Candès, E. J. (2019). Conformalized quantile regression. In Advances in Neural Information Processing Systems, 32:3538–3548. [link]

Rothe, R., Timofte, R., and Van Gool, L. (2018). Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer Vision, 126(2–4):144–157. [link]

Vovk, V., Gammerman, A., and Shafer, G. (2005). Algorithmic Learning in a Random World. Springer.

Vovk, V. (2012). Conditional validity of inductive conformal predictors. In Proceedings of the Asian Conference on Machine Learning, PMLR 25:475–490. [link]

Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. (2024). Multilingual E5 text embeddings: A technical report. arXiv:2402.05672. [link]

Citation

@misc{azizi2026modality,
  title         = {Conformal Calibration for Multi-Modal Regression with Missing Modalities},
  author        = {Azizi, Ilia},
  year          = {2026},
  eprint        = {2608.07795},
  archivePrefix = {arXiv},
  primaryClass  = {stat.ML},
  url           = {https://arxiv.org/abs/2608.07795}
}