The same three adapters, measured against a published result¶
Drift-Adapter evaluates the same three adapters this project fits — Orthogonal Procrustes, Low-Rank Affine and a compact Residual MLP — on the same problem: mapping a new model's query vectors into the space an old index already uses. It reports that they recover 95–99% of a full re-embedding's retrieval. The measured band reports a mean 0.714–0.722, and that bridging is worth doing in about one run in five.
This document was commissioned on the premise that the gap is a difference in what each protocol can see: that the published protocol is the more generous one, and that both numbers are right about different questions. The measurement says otherwise, and that is the finding. Running the paper's own protocol — on its own corpora, its own model pair, its own fit budget — this harness measures 0.24 to 0.50, not 0.95 to 0.99. The published band did not reproduce, under any of three readings of the protocol, and an adapter-independent ceiling puts it above what the old space can hold.
The paper's protocol is a legitimate protocol and it answers a legitimate question, which the paper states plainly: intrinsic metrics "provide a clean, direct, and application-agnostic assessment of the adapter's core function: preserving the nearest-neighbor structure" (section 6). Nothing below disputes that, and nothing below is a claim about the paper's conclusions on latency, cost or operational disruption.
What is in dispute is one set of numbers. Two measurements of the same protocol, on the same corpora and the same model pair, cannot both be right at 0.97 and 0.50 — so either this reproduction differs from the paper in a way neither document has identified, or the published figures do not describe what the published protocol produces. Section 9 says what would tell us which.
Harshil Vejendla, Drift-Adapter: A Practical Approach to Near Zero-Downtime Embedding Model Upgrades in Vector Databases. EMNLP 2025 Main, pages 15938–15949. arXiv:2509.23471 · ACL Anthology
1. What the paper measured¶
The ground truth is the new model's own neighbours. Section 4, "Query and Relevance Definition":
"The ground truth for retrieval (used to calculate Recall@k and MRR) is established by performing an exhaustive k-nearest neighbor search for each query within the 1M item database using embeddings generated by the new model (f_new) for both queries and database items. Adaptation Recall Ratio (ARR) is defined as the ratio of recall achieved by an adapter configuration to this ground truth recall."
The queries are documents. Same subsection:
"For the MTEB text datasets, we use 10,000 documents from their respective test sets as queries. These query documents are distinct from the items in the 1M-item database."
No human relevance judgement appears anywhere in the paper. That is rebasis' T0 tier, definition for definition, including the name ARR and the ratio-to-oracle form — and ADR 8 already records what T0 can and cannot see.
The rest of the setup, for the record. Text: AG-News, DBpedia-14 and Emotion, a
database of 1M items per dataset sampled from their training splits; f_old =
all-MiniLM-L6-v2, f_new = all-mpnet-base-v2. Image: 1M LAION-400M items, CLIP
ViT-B/32 → ViT-L/14. Adapters: OP solved in closed form; Low-Rank Affine at
r = 64; a residual MLP with 256 hidden units and GELU; an optional Diagonal
Scaling Matrix worth +0.005 to +0.015 ARR on LA and MLP and used by default
there. Np = 20,000 fit pairs (2% of the corpus), split 80/20, with query
embeddings "strictly held out". A single FAISS HNSW shard over the f_old vectors,
M = 32, ef_construction = 200, ef_search = 50. Results averaged over 5 seeds.
The headline table, section 5.1 — R@10 ARR:
| AG-News | DBpedia-14 | Emotion | LAION (CLIP) | |
|---|---|---|---|---|
| Misaligned (no adaptation) | 0.652 | 0.589 | 0.723 | 0.635 |
| OP | 0.974 | 0.968 | 0.953 | 0.942 |
| LA (r = 64) | 0.983 | 0.979 | 0.967 | 0.961 |
| MLP (256 hidden) | 0.992 | 0.990 | 0.984 | 0.978 |
Added query latency 3.0–9.8 µs. Standard deviations 0.001–0.005.
Section 5.3 evaluates one architecture jump, GloVe 300d → MPNet 768d on AG-News, with DSM on every variant: misaligned 0.213, OP 0.587, LA 0.632, MLP 0.715. The paper reads that number as a diagnostic rather than a failure — "is a 28.5% drop in recall an acceptable temporary bridge to avoid downtime, or is the model drift severe enough that an immediate full re-index is required?" — which is the same use rebasis puts ARR to.
Appendix A.4 reports a heterogeneous-drift experiment on a synthesised 500k DBpedia collection: one global MLP reaches 0.85, two class-routed MLPs reach 0.94. The paper calls it small-scale and preliminary. It is the strongest published support for the per-cluster adapters ADR 10 keeps on its list, and the drift in it is manufactured rather than observed.
2. The reproduction, and it does not reproduce¶
tools/bridge_band.py gained a --protocol flag. Under t0-knn the queries are
held-out documents, removed from the collection before anything is encoded, and
the judgements are the new model's exhaustive kNN over what remains. The adapter
still comes from the same probe_store → save_adapter path the rebasis fit
CLI runs and is still applied through the documented Bridge API.
The harness is not the variable. Its t1-judged leg, run from the same code
in the same pass, reproduces every published figure: retention at nDCG@10
0.717, the gain/retention anti-correlation −0.933, the naive swap at
0.151, bridging beating the status quo in 1 of 48 runs, and the
break-even predicting the outcome 48 times out of 48. Those are
bridge-band.md and cascade-band.md exactly.
The metric was then verified per query rather than in the mean. On AG-News, over
10,000 queries, recall@10 takes exactly the eleven values 0.0, 0.1 … 1.0 —
which is |retrieved ∩ relevant| / 10 with exactly ten relevant documents — and
the full_reindex configuration scores 1.0000 for every single query, which
is the paper's ARR = 1.0 oracle arriving as a check on the wiring.
On the paper's own three text corpora, its own model pair and its own Np:
| corpus | documents | bridged R@10 | status quo | ceiling | misaligned † | paper's OP |
|---|---|---|---|---|---|---|
| AG-News | 110,000 | 0.497 | 0.541 | 0.707 | 0.0001 | 0.974 |
| DBpedia-14 | 550,000 | 0.238 | 0.358 | 0.617 | 0.0000 | 0.968 |
| Emotion | 14,400 | 0.389 | 0.298 | 0.562 | 0.0005 | 0.953 |
† naive swap padded. MiniLM is 384-dimensional and MPNet is 768, so the new
query cannot enter the old index without a convention, and zero-padding is the
one this harness applies — see section 10 for
why that is a number rather than an option.
And on rebasis' own sixteen corpora and three-rung ladder, 48 runs under the same protocol: mean 0.408, range 0.175–0.706. Not one of the 48 reaches 0.90.
Two smaller things are worth a line each. The misaligned baseline — new query into the old index, the paper's own configuration — measures 0.000 to 0.001 on those three corpora against a published 0.589–0.723, and retains 0.024 of a full reindex across the 48 runs, where the dimension-matched rung contributes an honest swap and the other two a padded one. And Emotion is the only corpus in the set where bridging beats keeping the old model (0.389 against 0.298). The paper has no status-quo configuration, so nothing in it could have noticed either.
3. Three readings of the protocol, and all three eliminated¶
The paper's phrase is "an exhaustive k-nearest neighbor search"; it does not say what k is, and it does not describe how a 384-dimensional index is searched with 768-dimensional queries. So the reproduction ran every reading it could construct.
Reading one — the ground truth is the top-10 set. The natural reading of "Recall@10" against an exhaustive kNN, and the one measured above: 0.408 mean over 48 runs, 0.497 on AG-News. Eliminated.
Reading two — the ground truth is the nearest neighbour alone. Not a
strawman: it is what rebasis' own T0 does. rebasis.probe.groundtruth sets
SPARSE_RELEVANT = 1, because m0-findings.md section 3
measured both variants against real queries over 84 configurations and found the
strict one much the worse predictor:
| T0 ground truth | mean |ARR_T0 − ARR_T1| | worst | decision agreement |
|---|---|---|---|
| strict — the full top-10 kNN | 0.2613 | 0.5823 | 41.7% |
| relaxed — the nearest neighbour only | 0.0954 | 0.2750 | 53.6% |
Under that reading the numbers rise a long way and still do not arrive: AG-News 0.804, and over the 48 runs mean 0.684, range 0.312–0.971. One run in 48 reaches 0.95. Eliminated as an explanation of a 0.95–0.99 band.
Reading three — the corpus is duplicated. AG-News' training split holds 120,000 rows and the paper describes a database of 1,000,000 items sampled from it, which is reachable only with replacement, at roughly eight copies per document. Under that reading a query's top-10 fills up with copies of one document, and Recall@10 stops measuring neighbour-set overlap.
That mechanism is real and it was measured. --replicate copies each indexed
document after the query proxies are removed, so no proxy has a twin left, and
scoring collapses copies onto the underlying document so that retrieving a
different copy counts as a hit. The ceiling moves exactly as predicted —
Emotion at its implied 62× goes from 0.562 to 1.000, DBpedia-14 at its
implied 2× from 0.617 to 0.918. The top-10 really does collapse toward copies
of one document.
The adapter does not follow it up:
| documents | bridged R@10 | status quo | ceiling | |
|---|---|---|---|---|
| Emotion ‡ | 14,400 | 0.389 | 0.298 | 0.562 |
| Emotion ×62 ‡ | 930,000 | 0.337 | 0.314 | 1.000 |
| DBpedia-14 | 550,000 | 0.238 | 0.358 | 0.617 |
| DBpedia-14 ×2 | 1,100,000 | 0.309 | 0.481 | 0.918 |
‡ The two Emotion rows hold out different numbers of query proxies — 1,600 and 1,000 — so that pair is not a controlled comparison of replication alone. The DBpedia pair is.
Emotion goes down. The ceiling reaches 1.0 and the adapter stays near a third. Eliminated — and eliminated informatively, because the mechanism is confirmed and the consequence is not.
That elimination was predicted before the runs landed, on the paper's own numbers. The implied duplication factors are AG-News 8×, DBpedia-14 1.8× and Emotion 62×; the paper's OP scores on those corpora are 0.974, 0.968 and 0.953 — ordered inversely to duplication, with the most heavily duplicated corpus scoring lowest. If duplication were doing the lifting, that ordering would run the other way.
What could not be reconciled at all. The stated corpus sizes do not follow
from the stated splits, under either packaging. Upstream, AG-News is
120,000 train / 7,600 test, DBpedia-14 is 560,000 / 70,000, Emotion is
16,000 / 2,000 — so "1 million items … sampled from their respective training
sets" and "10,000 documents from their respective test sets" are both out of
reach for two of the three. MTEB's own packaging makes it worse rather than
better: there is no mteb/ag_news at all, mteb/emotion ships 15,956/1,988/1,986
and mteb/DBpediaClassification ships 2,048/2,048, because MTEB subsamples. This
is recorded as unreconciled, not as an accusation, and it is the reason reading
three was worth measuring at all.
4. The ceiling: the published band is above what the old space holds¶
This is the strongest statement available, because it does not depend on how good anybody's adapter is.
ceiling_old_space is a query built from the answer. For each query it uses the
normalised mean of the ground-truth documents' own old-model vectors — among
unit vectors, exactly the one maximising summed similarity to the target set. It
is not an adapter, not a method and not a result; nothing a user can run produces
it. It exists to bound the rows above it.
| bridged | ceiling | the paper reports | |
|---|---|---|---|
| AG-News, its own model pair | 0.497 | 0.707 | 0.974 – 0.992 |
| 48 runs, rebasis' ladder, mean | 0.408 | 0.567 | — |
| 48 runs, range | 0.175 – 0.706 | 0.255 – 0.802 | — |
Zero of the 48 ceilings reach 0.95, and the highest is 0.802. Under the strict reading the published band sits above what an oracle achieves — on the paper's own corpus and model pair, where a 384-dimensional MiniLM index does not hold MPNet's top-10 neighbourhoods in a form any single query point can retrieve, and on every one of the 48 runs besides, which span three dimension changes and sixteen corpora. Where that holds, no adapter of any family reaches the published band, however good, because the limit is the space and not the map. That is ADR 10's "retention is bounded by the source" arriving at a tier ADR 10 never claimed it for.
The caveat belongs beside the number every time it is quoted. The centroid maximises summed similarity to the target set, which is the natural relaxation of how many of them reach the top ten — so this estimates the maximum rather than proving it. A ceiling of 0.707 against a published 0.974 is a gap of 0.27, and the slack in that relaxation would have to be very large indeed to cover it; but a genuine upper bound would be a stronger object than this one, and this document does not have one.
Under reading two the ceiling is exactly 1.000, because a single target document is trivially retrievable by its own vector. So the published band is attainable in principle under the nearest-neighbour reading, and the measured number there is 0.684. That distinction matters: the disagreement is not everywhere a claim of impossibility.
5. The adapter family is worth 0.045 of it¶
Every candidate forced in turn, AG-News, the paper's model pair and Np, CSLS disabled so that selection cannot vary:
| adapter | parameters | R@10 | nDCG@10 | R@100 | the paper reports |
|---|---|---|---|---|---|
procrustes — the paper's OP |
589,824 | 0.4969 | 0.5655 | 0.8860 | 0.974 |
procrustes_centered+dsm — centred, with a diagonal scaling matrix |
591,744 | 0.4961 | 0.5648 | 0.8844 | — |
linear |
295,296 | 0.4833 | 0.5516 | 0.8711 | — |
residual_mlp — the paper's MLP |
590,848 | 0.4818 | 0.5501 | 0.8694 | 0.992 |
low_rank_affine, r = 192 |
221,568 | 0.4522 | 0.5176 | 0.8462 | — |
low_rank_affine, r = 64 — the paper's LA |
74,112 | 0.2196 | 0.2497 | 0.5759 | 0.983 |
| ceiling | — | 0.7065 | — | — | — |
Setting the paper's own rank aside, the whole family spread is 0.045 against a gap of 0.48. It is not the explanation.
Two things in that table stand on their own account.
The paper's ordering is reversed. It reports MLP > LA > OP and concludes the
MLP is best. On its own corpus, its own model pair and its own protocol, OP is
best here and the MLP is fourth of five. That is
ADR 10's finding —
procrustes_centered beating every more flexible candidate 15 times out of 15 —
reproducing at a tier it was not measured at, and it is the resolution of the
ADR-10-versus-the-paper tension that this reproduction can actually support: not
"the metric hides the MLP's advantage", but the ordering does not reproduce
either. One genuine surprise sits alongside it: uncentred procrustes beat
procrustes_centered on this pair, which ADR 10's 15-out-of-15 did not predict.
One pair at one tier is an exception to record, not a challenge to that ADR.
Rank 64 is the worst configuration measured, at less than half of rank 192
and less than half of OP. M0 predicted it: rebasis.core.linear sets a
proportional default precisely because a fixed rank of 64 was measured at d = 384
collapsing to ARR 0.458 against 0.834 for centred Procrustes. The paper reports
LA at r = 64 as its second-best adapter.
6. What actually separates the two protocols¶
Three protocols over the same 48 runs, varying one thing at a time. Retention is bridged ÷ full reindex, which under a kNN ground truth is exactly the ARR a published result reports.
| queries | ground truth | recall@10 | nDCG@10 | |
|---|---|---|---|---|
t0-knn@10 |
held-out documents | the new model's top-10 | 0.408 | 0.472 |
t0-knn@1 |
held-out documents | its nearest neighbour | 0.684 | 0.488 |
t0-knn-real-queries@10 |
real judged queries | the new model's top-10 | 0.417 | 0.485 |
t1-judged |
real judged queries | human judgement | 0.754 | 0.717 |
The third row exists only to make the walk attributable — between the first and the last, two things change at once, and a difference across both cannot be assigned to either.
| step | worth |
|---|---|
| the metric, at the published protocol (recall → nDCG) | +0.064 |
| the query distribution (documents → real queries) | +0.009 |
| the ground truth (a model's neighbours → human judgement) | +0.337 |
| the metric, at rebasis' protocol | −0.037 |
The query-proxy assumption is worth nine thousandths. The ground-truth definition is worth a third. That is M0's conclusion — "the document-as-query proxy is sound; the problem is the ground-truth definition" — reproducing on different corpora, a different ladder and a different harness, and it is the one part of the original premise that survived: the two protocols really do measure different things, and the difference really is almost entirely in what counts as a right answer.
It just does not run in the direction the premise assumed. rebasis' T1 retention is higher than the published protocol gives, not lower. 0.754 against 0.408.
7. The decision, which is the part a user holds¶
ADR 9 says bridge_advantage = ARR ×
upgrade_gain decides whether to bridge, and that nothing changes at T0 because
there is no break-even to consult. This is what that costs.
The paper's baselines are Oracle, Misaligned, Full Re-index and Dual Index
(section 4). "Old query into the old index" is never run. So the paper reports
no number from which upgrade_gain could be derived, and bridge_advantage
cannot be computed for any of its configurations. That is not a gap in
reporting that a reader could fill in: under a ground truth defined by f_new, the
old model's own retrieval score measures drift rather than quality, and quality
needs judgements the protocol does not have. The break-even asked of the paper's
own numbers has no answer, and this document does not estimate one.
What that is worth, measured on the 48 runs where both are available:
| bridging beat keeping the old model | |
|---|---|
t0-knn@10 — the published protocol |
38 / 48 |
t0-knn@1 |
26 / 48 |
t1-judged — human judgement |
1 / 48 |
The two protocols disagree about what to do in 37 of the 48 runs. Under the
published protocol bridging looks worth doing four times in five; under human
judgement it is worth doing once in forty-eight. (At T0 that comparison is the
break-even rather than a test of it: with the oracle pinned at 1.0 by
construction, gain × retention reduces to bridged ÷ status quo, so the 48-of-48
agreement the summary prints there is arithmetic and not evidence.)
This is where bridge-band.md section 2's shape reappears. A
0.99-retention run can still be worse than doing nothing — unix, potion→bge-base
at gain 2.10, retention 0.47, product 0.99, measured −0.7% — and a protocol with
no status-quo configuration cannot see that case at all, whatever its retention
number says.
8. The index-graph cost, scoped precisely¶
index-health.md measures that rewriting stored vectors damages an HNSW graph, and that a non-orthogonal map damages it 6–15× more than an orthogonal one. That cost does not apply to Drift-Adapter, and it would be an overreach to say the paper's "MLP is best" fails to price it in. Drift-Adapter maps queries and leaves the f_old index untouched; that is its whole premise, and no vector is ever rewritten.
What is defensible is narrower. The paper's ranking of adapter families is
established for the query-side arrangement only. It does not transfer to the
document-side arrangement — rebasis migrate — which the paper itself proposes in
section 2.3 ("Lazy/Background Re-embedding") and exercises in section 5.6, where
5% of the corpus is refreshed hourly and an adapter harmonises the mixed state.
In that arrangement the choice between an orthogonal map and an MLP costs
something the paper does not measure, and index-health.md is what it costs.
9. What would change this conclusion¶
Stated plainly, because a reproduction that names nothing that could overturn it is not a measurement.
- The paper's own code or fitted adapters. Every adapter here is rebasis'. The families match and the hyperparameters match where the paper names them, but a reference implementation scoring 0.97 on a corpus where this one scores 0.50 would locate the disagreement immediately.
- A statement of k. If the ground truth is neither the top-10 set nor the nearest neighbour, reading it correctly may close the gap; both readings this document could construct are reported above.
- The corpus. Any construction that yields 1,000,000 items and 10,000 test queries from AG-News' 120,000/7,600 split. Duplication was the one candidate here and it is eliminated; another would need testing on its own terms.
- A tighter ceiling. The bound in section 4 is an estimate, not a proof. A genuine upper bound on top-10 membership — or a demonstration that a query point exists which the centroid misses — would settle whether the published numbers are unreachable or merely unreached.
10. What this does not establish¶
- Not that the paper's adapters do not work. The mechanism reproduces everywhere. Under the published protocol bridging retains 0.472 of a full reindex against the misaligned baseline's 0.024 — twenty times as much, over 48 runs — and on AG-News, where the misaligned query cannot enter the index without padding, the ratio is four orders of magnitude. What did not reproduce is a set of numbers.
- Nothing about the image results. LAION and CLIP were not run. The text results are what this compares against.
- Nothing about latency, memory or cost. The paper's <10 µs and >100× figures are not contested and were not measured here; ADR 11 is this project's own account of that budget. Its downtime and recompute tables are labelled estimates in the paper itself.
- One seed. Every reproduction figure is at seed 0. The paper reports five seeds at standard deviations of 0.001–0.005, which cannot span 0.50 to 0.97, so this is a limit on precision rather than on the conclusion.
- The comparison is not like for like on scale. 110,000 documents against a stated 1,000,000 on AG-News, and 14,400 against 1,000,000 on Emotion. A larger collection makes recall@10 harder, so this difference works against the number measured here rather than for it — but it is a difference.
- Every other difference, including the ones that favour the paper. The
reproduction searches exactly where the paper searches an HNSW index at
ef_search = 50, which can only lower the paper's own figures. rebasis'
residual_mlpisxW + b + W₂·GELU(xW₁+b₁) + b₂, the paper's identity-residual MLP generalised — necessary at 768→384, where the paper'sx +term is not defined at all.low_rank_affinedefaults to rank 192 rather than 64, and the paper's own rank was run separately and scored worse. Query proxies are drawn from the same split as the database and removed from it, rather than from a separate test split, because the stated 10,000 test queries exceed that split. The two Emotion rows use different proxy counts (1,600 at rep = 1, 1,000 at ×62), so that pair is not a controlled comparison of replication alone. - How the paper searches a 384-dimensional index with 768-dimensional
queries. It is never stated. All three parameterizations in section 3 are
square maps and appendix A.1 computes every memory figure at d = 768, so
zero-padding the old space is the only reading consistent with the paper's own
text — and zero-padding preserves every inner product, so it is a relabelling
rather than a transform. Measured under it, the misaligned baseline is 0.0001.
That row is a convention of this harness, not an option anyone has. Where
the new model is the wider one, the padding falls on the indexed documents, and
widening an index means rewriting it — the one thing the tool exists to avoid.
IdentityAdapteris not a precedent either: it pads only when the new model is narrower and truncates in the other direction. Readnaive swap paddedas what the number would be if the configuration existed.
Reproducing¶
# The three protocols over sixteen corpora, one ladder — the 48-run comparison
uv run --extra sentence-transformers --with ir-datasets --with ranx \
--with model2vec --with datasets python tools/bridge_band.py \
--corpus heldout --corpus beir --ladder default \
--protocol t1-judged,t0-knn-real-queries,t0-knn \
--k 10,100,200 --cascade 100,200 --t0-truth-k 10 \
--out reports/band/protocol.jsonl
# The nearest-neighbour reading of the same protocol
uv run ... python tools/bridge_band.py --corpus heldout --corpus beir \
--ladder default --protocol t0-knn --t0-truth-k 1 \
--k 10,100,200 --cascade 100,200 --out reports/band/protocol.jsonl
# The paper's own corpora and model pair, at its own fit budget
uv run ... python tools/bridge_band.py \
--corpus drift-adapter --ladder drift-adapter --protocol t0-knn \
--t0-truth-k 10 --k 10,100 --cascade '' --fit-pairs 16000 \
--t0-queries 10000 --out reports/band/paper-corpora.jsonl
# The duplication reading, at each corpus' implied factor
uv run ... python tools/bridge_band.py --corpus hfdocs:dair-ai/emotion:split \
--ladder drift-adapter --protocol t0-knn --t0-truth-k 10 --replicate 62 \
--k 10,100 --cascade '' --fit-pairs 16000 --t0-queries 1000 \
--out reports/band/paper-corpora.jsonl
uv run python tools/bridge_band_report.py reports/band/protocol.jsonl --view protocol
Per-query scores are written beside the rows, one file per run under
reports/band/per-query/, carrying the query ids next to the arrays so two runs
cannot be lined up by position. Every row names its protocol and its
ground-truth depth, because those are two different measurements and a file holds
both.