Ettin リランカーファミリーの紹介

256

17.6M

384

32.8M

512

68.6M

768

150.9M

1024

401.6M

1792

1.00B

これら 6 つのモデルは、Ettin エンコーダーと同様に Apache 2.0 ライセンスの下でリリースされています。

結果

MTEB(eng, v2) リトリーバル

公開された各モデルを、MTEB の 2 段階再ランク付けフローを用いて、MTEB(eng, v2) Retrieval ベンチマーク（10 タスク、上位 100 件を再ランク付け）の全タスクに適用して評価を行いました。各再ランクャーには、速度と品質のスペクトラム全体をカバーする 6 つの埋め込みモデルをペアリングしました：

埋め込みモデル

アクティブパラメータ数

検索のみ NDCG@10

0.3495

23M

0.4292

33M

0.5149

137M

0.5226

308M

0.5463

596M

0.5980

各チャートにおける破線の検索のみラインが、競争相手となる主要な数値です。このラインより下回る結果は、平均的に再ランクャーがパイプラインの性能を低下させていることを意味します：

結果の全表（クリックして展開）

6 つの埋め込みモデルの組み合わせ全体における NDCG@10 の平均値を降順にソートした結果です。当社の 6 つのモデルは太字で、教師モデルである mixedbread-ai/mxbai-rerank-large-v2 は下線付きで表示されています。

Reranker

パラメータ数

MTEB(eng, v2) 検索 NDCG@10

4.02B

0.6367

1.54B

0.6115

1.00B

0.6114

401M

0.6091

151M

0.5994

596M

0.5940

494M

0.5920

68.6M

0.5915

2.44B

0.5856

150M

0.5843

32.8M

0.5779

150M

0.5656

17.6M

0.5576

568M

0.5526

4.02B

0.5300

560M

0.5098

22.7M

0.5082

33.4M

0.5066

435M

0.5063

19.2M

0.4979

70.8M

0.4968

278M

0.4890

184M

0.4865

† max_seq_length=8192 にキャップされています（4B Qwen3 ベースのrerankerは、ネイティブコンテキストでは単一の H100 80GB に収まりません。ネイティブコンテキストでの評価結果はおそらくさらに高くなります）。

NanoBEIR の完全な結果表（クリックして展開）

NanoBEIR は、BEIR の高速な 13 データセットからなるサブセットで、各データセットあたり最大 5000 ドキュメントに対して 50 クエリを使用します。NanoBEIR は、トレーニング時に metric_for_best_model に設定された指標（Evaluation を参照）であり、私が実験を導くために使用したものです。

Reranker

パラメータ数

NanoBEIR 平均 NDCG@10

1.54B

0.7318

1.00B

0.7237

2.44B

0.7197

401M

0.7193

494M

0.7162

151M

0.7086

150M

0.7017

568M

0.6971

68.6M

0.6915

150M

0.6909

32.8M

0.6825

17.6M

0.6746

435M

0.6488

560M

0.6379

33.4M

0.6369

22.7M

0.6312

19.2M

0.6298

184M

0.6231

70.8M

0.6136

278M

0.6027

私が公開する最も小さなモデルである17Mは、パラメータ数が約半分という条件下で、MTEBにおいて33Mのms-marco-MiniLM-L12-v2をNDCG@10で+0.051 (0.5576 vs 0.5066) 上回り、NanoBEIRでは+0.038 (0.6746 vs 0.6369) の差をつけています。また、32MはMTEBにおいて568MのBAAI/bge-reranker-v2-m3を+0.025 (0.5779 vs 0.5526) 上回っており、これはパラメータ数で17倍もの差があることを意味します。もし従来のMiniLM rerankers のいずれかを「検索後に再ランク付けする」スタックのデフォルトとして使用している場合、私たちの17M (または32M) に差し替えることは低リスクな置き換えとなり、両方のベンチマークで明確な品質向上が期待できます。

表を上位へ見ていくと、150MはMTEBにおいて600M未満の範囲で私がテストした中で最も強力なrerankerであり、直近のQwen/Qwen3-Reranker-0.6B (596M) を+0.005 (0.5994 vs 0.5940) の差でわずかに上回り、すべてのBAAI bge-reranker バリアントを0.03から0.05の差で下回っています。また、68Mにも言及する価値があります：0.5915というスコアはQwen3-Reranker-0.6B (0.5940) とほぼ同じ位置にありながら、パラメータ数はその約9分の1です。

公開された範囲の最上位にある1Bモデルは、教師モデルに非常に近い性能を示します。MTEBでは1.54Bのmxbai-rerank-large-v2と0.0001 (0.6114 vs 0.6115) の差しかなく、NanoBEIRでも0.008の差に収まっています。これは、自分自身より54%大きいモデルから知識蒸留（distillation）を行ったにもかかわらず達成された結果です。この蒸留プロセスは教師モデルとの格差を実質的に解消しており、今回の公開に向けて私が期待していた通りでした。

比較において全体最強の再ランク付けモデルは、MTEB で 0.6367 を記録し、当社の 1B モデルより +0.025 高い Qwen/Qwen3-Reranker-4B です。現在のレシピからこの差を埋めるには、おそらくより強力な教師モデルからの知識蒸留（distilling）が必要となるでしょう（当社の教師モデル自体も Qwen3-Reranker-4B より下位に位置しています）。ほとんどの「検索後に再ランク付け」のワークロードにおいては、パラメータ数が 1/4 の当社の 1B モデル（速度参照）の方が、はるかに実用的な選択肢となります。

速度

品質の数値は再ランク付けモデルにとって重要な要素の半分だけです。もう半分の重要度は、検索結果をユーザーに表示するまでの時間枠内で、そのレイテンシが予算内に収まるかどうかにかかっています。私が測定した内容について順を追って説明します。

私は、単一の NVIDIA H100 80GB グラフィックボード上で、公開されている 6 つのモデルすべてを、13 のパブリック再ランク付けモデル（約 1B パラメータまでの強力なベースライン）と比較してベンチマークしました。クエリとドキュメントは、sentence-transformers/natural-questions データセットの自然なドキュメント長分布から取得したものです：NQ の回答の多くは短く、一部は長くなります。ドキュメントは、古いモデルに不公平な優位性を与えないよう、max_length=512 で切り捨てられています。各モデルは、そのアーキテクチャがサポートする最良のアテンション実装を使用します。BERT、XLM-RoBERTa、ModernBERT、Qwen2 については Flash Attention 2 を使用し（Flash Attention 2）、それ以外では SDPA（SDPA）を使用します。DeBERTa-v2 の場合は eager モードを使用します（これは現在、transformers ライブラリにおいて FA2 も SDPA もサポートされていないためです）。

各モデルに対して、バッチサイズ検索はバッチサイズ 8 から開始し、GPU のメモリが尽きるまで倍増していきます。各バッチサイズでは 3 回の計測パスを実行し、その中央値（メディアン）のスループットを記録するため、単一の不運な実行結果が数値を大きく歪めることはありません。報告されるスループットは、勝利したバッチサイズにおけるものです。

表 1. 秒間あたりのペア数（すべて bfloat16）。当社の 6 つの再ランク化器（rerankers）は太字で示されています。

Model | Params | Attn | pairs / second

---|---|---|---

cross-encoder/ettin-reranker-17m-v1 | 17M | FA2 | 7517

cross-encoder/ettin-reranker-32m-v1 | 32M | FA2 | 6602

cross-encoder/ettin-reranker-68m-v1 | 68M | FA2 | 4913

cross-encoder/ms-marco-MiniLM-L4-v2 | 19M | FA2 | 4029

cross-encoder/ms-marco-MiniLM-L6-v2 | 22M | FA2 | 3817

cross-encoder/ms-marco-MiniLM-L12-v2 | 33M | FA2 | 3311

cross-encoder/ettin-reranker-150m-v1 | 150M | FA2 | 3237

BAAI/bge-reranker-base | 278M | FA2 | 2858

mixedbread-ai/mxbai-rerank-xsmall-v1 | 70M | eager | 2636

184M

eager

1953

400M

FA2

1738

560M

FA2

1659

568M

FA2

1569

150M

FA2

1418

150M

FA2

1404

FA2

928

435M

eager

867

494M

FA2

809

1.5B

FA2

387

私たちの17Mモデルは、比較対象全体の中で最も高速なリランカーであり、1秒間に7517ペアを処理します。これはms-marco-MiniLM-L6-v2（3817）のスループットのほぼ2倍に相当し、より小型のms-marco-MiniLM-L4-v2（4029）よりもさらに高速です。前述のMTEB表でお分かりいただいた通り、私たちの17MモデルはすべてのMiniLMバリアントよりも精度が高いことも確認されています。現在MiniLMクロスエンコーダーを実行中の方にとって、私たちの17Mモデルへ切り替えるのは単なる一行の変更で済み、レイテンシと検索品質の両方を向上させることができます。

私たちの150Mモデルはさらに興味深い比較対象となります。なぜなら、パラメータ数が正確に150Mである直接競合するアーキテクチャを持つモデルが2つ存在するからです：Alibaba-NLP/gte-reranker-modernbert-base と ibm-granite/granite-embedding-reranker-english-r2 です。これら両モデルは、同じModernBERT-baseバックボーン（基盤アーキテクチャ）に基づいて構築されています。私たちの150Mモデルは1秒間に3237ペアで動作するのに対し、この2つの競合モデルはそれぞれ1418と1404ペアで、速度差が2.3倍に達します。

150M モデルのすべてが Flash Attention 2 を使用していますが、そのうちの 2 つは AutoModelForSequenceClassification を通してロードされ、入力にパディングが保持されたままになります。つまり、アテンション自体は FA2 カーネルを実行しますが、モデルの残りの部分は、何にも貢献しないパディングトークンに対して依然として密な計算を行っています。一方、当社のモジュラー Transformer モジュール（上記の Architecture Details を参照）では、パディングのない入力をモデル全体にわたって伝播させるため、各層は実在するトークンのみに計算リソースを費やします。これが FA2 の一部のみを享受することと、その恩恵をすべて受け取ることの違いです。

表の下部にある 1B モデルは、秒間 928 ペアという処理速度を達成しており、これは MTEB スコアが 0.0001 の差以内で一致しつつも、1.54B の教師モデルである mxbai-rerank-large-v2（秒間 387 ペア）と比較して 2.4 倍高速です。この教師モデルは Qwen2 ベースであり、ペアごとにプロンプトテンプレートのオーバーヘッドが存在しますが、蒸留された学生モデルはその教師のキャリブレーションと判断力を継承しつつ、実行時のすべての負荷を排除しています。これは正直に言って、今回のリリース全体で私にとって最も満足感を得られる単一の数値です。

残念なことに、DeBERTa-v2 ベースの mxbai-rerank-{xsmall,base,large}-v1 シリーズは、DeBERTa-v2 が現在 transformers において Flash Attention 2 も SDPA（Sparse Dynamic Programming Attention）もサポートしていないため、表の他のモデルよりも大幅に低速になります。70M の mxbai-rerank-xsmall-v1 は秒間 2636 ペアで動作しますが、これはパラメータ数がほぼ同じである当社の 68M モデルのスループットの約半分です。これらのモデル自体は全く問題ありませんが、最新の注意カーネルを利用できないというだけです。

Same benchmark on a consumer GPU (RTX 3090, 24 GB)

データセンター向け GPU ではなく、コンシューマー向けカードでセルフホスティングする場合の、同じスループットスイープを RTX 3090 で実施した結果です。ベンチマーク設定は Table 1 と同一：bfloat16（ブーフローティン）、各モデルで最もサポートが充実しているアテンション機構、および収容可能な最大のバッチサイズにおける 3 回の実行の中央値スループット。

Model

Params

Best attn

pairs / second

17M

FA2（Flash Attention 2）

9008

19M

FA2

5071

32M

FA2（Flash Attention 2）

4497

22M

FA2

4234

33M

FA2

2847

68M

FA2（Flash Attention 2）

1916

70M

eager（イーガーモード）

1677

278M

FA2

1329

150M

FA2（Flash Attention 2）

982

184M

eager

772

150M

FA2

598

150M

FA2

586

560M

FA2

448

568M

FA2

436

400M

FA2

429

435M

eager

266

494M

FA2

221

FA2

189

1.5B

FA2

17M モデルは依然として表の中で最速であり、1 秒あたり 9008 ペアという速度は H100 の数値よりも高く、これは極小サイズでは純粋な計算リソースがボトルネックではなく、H100 の追加性能がそのまま転換されないことを示唆しています。表の中央部では再編が行われ、MiniLM リランカーが当社の 32M および 68M モデルを抜いて上位に位置し、1B モデルは mxbai-rerank-base-v2（1 秒あたり 189 ペア対 221 ペア）の下位に後退しました。一方、当社の 150M モデルは 2 つの 150M ベースの ModernBERT 系競合モデルに対して依然として堅固なリードを維持しており、教師モデル置換のストーリーも有効で、当社の 1B モデルは 1.5B の mxbai-rerank-large-v2（1 秒あたり 189 ペア対 69 ペア）と比較してスループットが 2.7 倍となっています。

同じベンチマークを CPU（Intel Core i7-13700K）で実行した場合

モデル | パラメータ数 | 最適アテンション | 秒間ペア数

cross-encoder/ettin-reranker-17m-v1 | 17M | SDPA | 267.4

cross-encoder/ms-marco-MiniLM-L4-v2 | 19M | SDPA | 206.2

cross-encoder/ms-marco-MiniLM-L6-v2 | 22M | SDPA | 143.9

cross-encoder/ettin-reranker-32m-v1 | 32M | SDPA | 92.5

cross-encoder/ms-marco-MiniLM-L12-v2 | 33M | SDPA | 75.9

mixedbread-ai/mxbai-rerank-xsmall-v1 | 70M | eager | 38.9

68M

SDPA

31.2

278M

SDPA

19.2

150M

SDPA

14.7

150M

SDPA

14.5

150M

SDPA

14.0

184M

eager

13.4

560M

SDPA

6.2

568M

SDPA

6.0

400M

SDPA

5.2

435M

eager

4.3

494M

SDPA

3.5

SDPA

2.1

CPU では bf16、Flash Attention 2、アンパディングの利点を活用できないため、レイテンシに関する話は少し単純になります：パラメータ数が多いほどモデルは遅くなります。17M モデルは ms-marco-MiniLM-L6-v2 よりも大幅に高速です（1 秒あたりペア数で 267.4 vs 143.9）

原文を表示

Back to Articles

TL;DR

Today I'm releasing six new Sentence Transformers CrossEncoder rerankers, state-of-the-art at their respective sizes, built on top of the Ettin ModernBERT encoders, together with the data and full training recipe that produced them:

cross-encoder/ettin-reranker-17m-v1

cross-encoder/ettin-reranker-32m-v1

cross-encoder/ettin-reranker-68m-v1

cross-encoder/ettin-reranker-150m-v1

cross-encoder/ettin-reranker-400m-v1

cross-encoder/ettin-reranker-1b-v1

The models were trained with a distillation recipe: pointwise MSE on mixedbread-ai/mxbai-rerank-large-v2 scores over cross-encoder/ettin-reranker-v1-data, which is a subset of lightonai/embeddings-pre-training mixed with a reranked subset of lightonai/embeddings-fine-tuning.

*Our six rerankers paired with google/embeddinggemma-300m on MTEB(eng, v2) Retrieval. See Results for five more embedder pairings.*

If you're new to rerankers and want the "why" first, jump to What is a reranker, and why pair one with an embedder?. If you just want to plug a model in, jump to Usage. If you want to train your own, jump to Training.

I bootstrapped the training recipe below with the new train-sentence-transformers Agent Skill shipped in Sentence Transformers v5.5.0. Install it with hf skills add train-sentence-transformers [--global] [--claude] and ask your AI coding agent (Claude Code, Codex, Cursor, Gemini CLI, ...) to fine-tune a SentenceTransformer, CrossEncoder, or SparseEncoder model on your data.

What is a reranker, and why pair one with an embedder?

Usage

End-to-end retrieve-then-rerank pipeline

Architecture Details

Results

MTEB(eng, v2) Retrieval

Speed

Training

Distillation recipe

Dataset

Training Arguments

Evaluation

Overall Training Script

Conclusion

Acknowledgements

What is a reranker, and why pair one with an embedder?

A reranker (a.k.a. pointwise cross-encoder) is a neural model that takes a (query, document) pair and outputs a single relevance score. Unlike an embedding model, which encodes the query and document separately and computes their similarity from the two embedding vectors, a reranker lets the two texts attend to each other through every transformer layer. That joint encoding is more accurate but also more expensive: the model has to be run once per (query, document) pair rather than once per text.

Because cross-encoders are too expensive to run over a full corpus, the common production pattern is retrieve-then-rerank: a fast embedding model retrieves the top-K candidates (cheap), then a cross-encoder re-orders just those K with high accuracy. The total cost stays bounded while the final ranking is much closer to what an exhaustive cross-encoder pass would produce.

Throughout this blogpost I'll use "reranker" and "cross-encoder" interchangeably.

Usage

The released models are normal Sentence Transformers CrossEncoder models, so you can use them with just 3 lines of code:

code

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ettin-reranker-32m-v1")
scores = model.predict([
    ("Where was Apple founded?", "Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne."),
    ("Where was Apple founded?", "The Fuji apple is an apple cultivar developed in the late 1930s and brought to market in 1962."),
])
print(scores)
# [11.393298  2.968891]   <- larger means more relevant

For a query and a list of candidates, you can also use rank to get back sorted indices and scores:

code

ranked = model.rank(
    query="Which planet is known as the Red Planet?",
    documents=[
        "Venus is often called Earth's twin because of its similar size and proximity.",
        "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
        "Jupiter, the largest planet in our solar system, has a prominent red spot.",
        "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
    ],
    top_k=4,
    return_documents=True,
)
for r in ranked:
    print(f"({r['score']:.2f}): {r['text']}")
# (10.82): Mars, known for its reddish appearance, is often referred to as the Red Planet.
# (9.86): Saturn, famous for its rings, is sometimes mistaken for the Red Planet.
# (8.55): Jupiter, the largest planet in our solar system, has a prominent red spot.
# (6.21): Venus is often called Earth's twin because of its similar size and proximity.

You can swap cross-encoder/ettin-reranker-32m-v1 for any other size to trade quality for speed. All six accept up to 8K tokens of context (useful for long-document reranking) thanks to ModernBERT's long-context pre-training.

It is recommended to install kernels and set model_kwargs={"dtype": "bfloat16", "attn_implementation": "flash_attention_2"} for the highest throughput. See the Speed section below for more details, but in general you can expect a 1.7x-8.3x speedup over default loading depending on model size and sequence length.

code

from sentence_transformers import CrossEncoder

model = CrossEncoder(
    "cross-encoder/ettin-reranker-32m-v1",
    model_kwargs={"dtype": "bfloat16", "attn_implementation": "flash_attention_2"},
)

End-to-end retrieve-then-rerank pipeline

A complete example with a fast embedder for retrieval and the reranker for the final ordering:

code

from sentence_transformers import SentenceTransformer, CrossEncoder

# Fast retrieval with a static embedder (sub-millisecond on CPU per query)
embedder = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1")
reranker = CrossEncoder("cross-encoder/ettin-reranker-68m-v1")

corpus = [
    "Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne.",
    "The Fuji apple is an apple cultivar developed in the late 1930s.",
    "Steve Jobs introduced the iPhone in 2007 at Macworld.",
    "Macintosh computers were sold by Apple from 1984 onward.",
    # ... thousands or millions more in production
]
query = "Where was Apple founded?"

# Step 1: encode + retrieve top-100
query_emb = embedder.encode_query(query, convert_to_tensor=True)
corpus_emb = embedder.encode_document(corpus, convert_to_tensor=True)
scores = embedder.similarity(query_emb, corpus_emb)[0]
top_k_idx = scores.topk(min(100, len(corpus))).indices.tolist()

# Step 2: rerank
top_k_docs = [corpus[i] for i in top_k_idx]
ranked = reranker.rank(query, top_k_docs, top_k=5, return_documents=True)
for r in ranked:
    print(f"({r['score']:.2f}): {r['text']}")
# (11.63): Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne.
# (4.71): Steve Jobs introduced the iPhone in 2007 at Macworld.
# (1.96): The Fuji apple is an apple cultivar developed in the late 1930s.
# (1.49): Macintosh computers were sold by Apple from 1984 onward.

This is the same shape used by most modern search systems. The retriever decides what enters the funnel, the reranker decides what wins.

Architecture Details

All six rerankers share the same architecture and differ only in their backbone size. The backbone is one of the six Ettin encoders from Johns Hopkins University's Ettin suite. These are ModernBERT-style models with unpadded attention, RoPE positional encodings, GeGLU, and 2T tokens of open-license pre-training, supporting up to 8192 tokens of context.

On top of each encoder, the reranker uses a 4-module classification head that mirrors ModernBertForSequenceClassification but is built from Sentence Transformers' modular components. The underlying Transformer is a plain AutoModel rather than AutoModelForSequenceClassification, which lets us use sequence unpadding for variable-length inputs for Flash Attention 2. At medium-document sequence lengths this is a 1.7x-8.3x speedup over fp32+SDPA depending on model size (see Speed for the full benchmark):

code

1. Transformer(FA2)
2. Pooling(cls)
3. Dense(H, H, bias=False, GELU)
4. LayerNorm(H)
5. Dense(H, 1, scores)

In my ablations, CLS pooling outperformed mean pooling. That was a little surprising. ModernBERT uses global attention only every third layer and the other two-thirds use local-window attention that cannot reach CLS from distant positions. Empirically, those few global layers carry enough signal to make CLS the better pooling choice.

Model

Backbone

Hidden size

Layers

Params (head incl.)

256

17.6M

384

32.8M

512

68.6M

768

150.9M

1024

401.6M

1792

1.00B

All six models are released under the Apache 2.0 license, matching the Ettin encoders.

Results

MTEB(eng, v2) Retrieval

I ran each released model through the full MTEB(eng, v2) Retrieval benchmark (10 tasks, top-100 reranked) using MTEB's two-stage reranking flow, pairing each reranker with six embedding models that span the speed/quality spectrum:

Embedding Model

Active params

Retriever-only NDCG@10

0.3495

23M

0.4292

33M

0.5149

137M

0.5226

308M

0.5463

596M

0.5980

The dashed retriever-only line in each chart below is the headline number to beat. Anything below it means the reranker actively hurts the pipeline on average:

Full table of results (click to expand)

Mean NDCG@10 over the 6 embedder pairings, sorted descending. Our six models are in bold, and the teacher mixedbread-ai/mxbai-rerank-large-v2 is underlined.

Reranker

Params

MTEB(eng, v2) Retrieval NDCG@10

4.02B

0.6367

1.54B

0.6115

1.00B

0.6114

401M

0.6091

151M

0.5994

596M

0.5940

494M

0.5920

68.6M

0.5915

2.44B

0.5856

150M

0.5843

32.8M

0.5779

150M

0.5656

17.6M

0.5576

568M

0.5526

4.02B

0.5300

560M

0.5098

22.7M

0.5082

33.4M

0.5066

435M

0.5063

19.2M

0.4979

70.8M

0.4968

278M

0.4890

184M

0.4865

† Capped to max_seq_length=8192 (the 4B Qwen3-based rerankers don't fit on a single H100 80GB at native context). Native-context evaluation is likely higher.

Full table of NanoBEIR results (click to expand)

NanoBEIR is a fast 13-dataset subset of BEIR that uses 50 queries per dataset against up to 5000 documents each. NanoBEIR is what metric_for_best_model was set to during training (see Evaluation), and what I used to guide the experimentation.

Reranker

Params

NanoBEIR mean NDCG@10

1.54B

0.7318

1.00B

0.7237

2.44B

0.7197

401M

0.7193

494M

0.7162

151M

0.7086

150M

0.7017

568M

0.6971

68.6M

0.6915

150M

0.6909

32.8M

0.6825

17.6M

0.6746

435M

0.6488

560M

0.6379

33.4M

0.6369

22.7M

0.6312

19.2M

0.6298

184M

0.6231

70.8M

0.6136

278M

0.6027

The smallest model I'm releasing, our 17M, beats the 33M ms-marco-MiniLM-L12-v2 by +0.051 NDCG@10 (0.5576 vs 0.5066) on MTEB and +0.038 (0.6746 vs 0.6369) on NanoBEIR at roughly half the parameter count. The 32M beats the 568M BAAI/bge-reranker-v2-m3 by +0.025 (0.5779 vs 0.5526) on MTEB, a 17x parameter gap. If you've been using one of the legacy MiniLM rerankers as the default in your retrieve-then-rerank stack, swapping in our 17M (or 32M) is a low-risk drop-in replacement, with a noticeable quality bump on both benchmarks.

Moving up the table, our 150M is the strongest reranker I tested in the under-600M range on MTEB, edging out the recent Qwen/Qwen3-Reranker-0.6B (596M) by +0.005 (0.5994 vs 0.5940) and beating every BAAI bge-reranker variant by 0.03 to 0.05. The 68M is also worth a mention: at 0.5915 it lands almost exactly on Qwen3-Reranker-0.6B (0.5940) while using a ninth of the parameters.

At the top of the released range, our 1B model closely tracks its teacher. It comes within 0.0001 of the 1.54B mxbai-rerank-large-v2 on MTEB (0.6114 vs 0.6115) and within 0.008 on NanoBEIR, despite distilling from a model 54% larger than itself. The distillation effectively closes the gap to the teacher, which is what I was hoping to see going into this release.

The overall strongest reranker in the comparison is Qwen/Qwen3-Reranker-4B at 0.6367 MTEB, +0.025 above our 1B model. Closing that gap from the current recipe would likely require distilling from a stronger teacher (our teacher itself sits below Qwen3-Reranker-4B). For most retrieve-then-rerank workloads, our 1B at a quarter of the parameters (see Speed) is a much more practical pick.

Speed

Quality numbers are only half of what matters for a reranker. The other half is whether its latency fits inside the budget you have between retrieval and showing results to the user. Let me walk through what I measured.

I benchmarked all six released models against thirteen public rerankers (strong baselines up to about 1B parameters) on a single NVIDIA H100 80GB. The queries and documents come from sentence-transformers/natural-questions at its natural document-length distribution: most NQ answers are short, some are long. Documents are truncated at max_length=512 to avoid giving the older models an unfair advantage. Each model uses its best supported attention implementation: Flash Attention 2 wherever the architecture supports it (BERT, XLM-RoBERTa, ModernBERT, Qwen2), SDPA where it doesn't, and eager for DeBERTa-v2 (which currently has neither FA2 nor SDPA support in transformers).

For every model an auto-batch search starts at batch size 8 and doubles until the GPU runs out of memory. At each batch size I run three timed passes and keep the median throughput, so a single unlucky run doesn't drag the number around. The reported throughput is at whichever batch size won.

Table 1. Throughput in pairs per second, all in bfloat16. Our six rerankers are in bold.

Model

Params

Attn

pairs / second

17M

FA2

7517

32M

FA2

6602

68M

FA2

4913

19M

FA2

4029

22M

FA2

3817

33M

FA2

3311

150M

FA2

3237

278M

FA2

2858

70M

eager

2636

184M

eager

1953

400M

FA2

1738

560M

FA2

1659

568M

FA2

1569

150M

FA2

1418

150M

FA2

1404

FA2

928

435M

eager

867

494M

FA2

809

1.5B

FA2

387

Our 17M is the fastest reranker in the whole comparison, at 7517 pairs per second. That's almost twice the throughput of ms-marco-MiniLM-L6-v2 (3817) and faster even than the smaller ms-marco-MiniLM-L4-v2 (4029). And as you saw in the MTEB table earlier, our 17M is also more accurate than every MiniLM variant. If you're currently running a MiniLM cross-encoder, swapping to our 17M is a one-line change that improves both your latency and search quality.

Our 150M is an even more interesting comparison, because there are two direct architectural peers at exactly 150M parameters: Alibaba-NLP/gte-reranker-modernbert-base and ibm-granite/granite-embedding-reranker-english-r2. Both are built on the same ModernBERT-base backbone. Our 150M runs at 3237 pairs per second, while the two peers come in at 1418 and 1404 respectively, for a 2.3x speed gap.

All three 150M models use Flash Attention 2, but the two peers load through AutoModelForSequenceClassification, which keeps the inputs padded. So attention itself runs the FA2 kernel, but the rest of the model is still doing dense compute on padding tokens that don't contribute anything. Our modular Transformer module (see Architecture Details above) propagates unpadded inputs all the way through the model, so every layer only spends compute on real tokens. That's the difference between getting some of FA2's benefit and getting all of it.

At the bottom of the table, our 1B model hits 928 pairs per second, which is 2.4x faster than the 1.54B teacher mxbai-rerank-large-v2 (387 pairs per second) while matching its MTEB score within 0.0001. The teacher is Qwen2-based with a prompt-template overhead per pair, so the distilled student inherits the teacher's calibration and judgement but skips all the runtime baggage. This is honestly the most satisfying single number in the whole release for me.

One unfortunate note: the DeBERTa-v2-based mxbai-rerank-{xsmall,base,large}-v1 series ends up much slower than the rest of the table because DeBERTa-v2 currently supports neither Flash Attention 2 nor SDPA in transformers. The 70M mxbai-rerank-xsmall-v1 runs at 2636 pairs per second, about half the throughput of our 68M at almost the same parameter count. The models themselves are perfectly fine, they just don't get to use modern attention kernels.

Same benchmark on a consumer GPU (RTX 3090, 24 GB)

If you're self-hosting on a consumer card rather than a datacenter GPU, here's the same throughput sweep on an RTX 3090. Same benchmark setup as Table 1: bfloat16, best-supported attention per model, three-trial median throughput at the largest batch that fits.

Model

Params

Best attn

pairs / second

17M

FA2

9008

19M

FA2

5071

32M

FA2

4497

22M

FA2

4234

33M

FA2

2847

68M

FA2

1916

70M

eager

1677

278M

FA2

1329

150M

FA2

982

184M

eager

772

150M

FA2

598

150M

FA2

586

560M

FA2

448

568M

FA2

436

400M

FA2

429

435M

eager

266

494M

FA2

221

FA2

189

1.5B

FA2

Our 17M is still the fastest model in the table at 9008 pairs per second, actually higher than its H100 number, which suggests that at tiny sizes raw compute isn't the bottleneck and the H100's extra muscle doesn't translate. The middle of the table reshuffles a bit, with the MiniLM rerankers overtaking our 32M and 68M, and the 1B slipping behind mxbai-rerank-base-v2 (189 vs 221 pairs per second). Our 150M model still holds a solid lead over the two 150M ModernBERT-based peers, and the teacher-replacement story still holds, with our 1B at 2.7x the throughput of the 1.5B mxbai-rerank-large-v2 (189 vs 69 pairs per second).

Same benchmark on CPU (Intel Core i7-13700K)

Model

Params

Best attn

pairs / second

17M

SDPA

267.4

19M

SDPA

206.2

22M

SDPA

143.9

32M

SDPA

92.5

33M

SDPA

75.9

70M

eager

38.9

68M

SDPA

31.2

278M

SDPA

19.2

150M

SDPA

14.7

150M

SDPA

14.5

150M

SDPA

14.0

184M

eager

13.4

560M

SDPA

6.2

568M

SDPA

6.0

400M

SDPA

5.2

435M

eager

4.3

494M

SDPA

3.5

AWS Machine Learning Blog重要度42026年7月15日 01:33

SDPA

2.1

On CPU, we can't take advantage of bf16, Flash Attention 2, or unpadding, so the latency story is a bit simpler: the higher the parameter count, the slower the model. The 17M model is considerably faster than ms-marco-MiniLM-L6-v2 (267.4 vs 143.9 pairs per sec

この記事をシェア

Amazon Bedrock を活用した Flo Health の医療コンテンツレビューの拡張（第 2 部）

AI News重要度42026年7月15日 00:54

AWS と Bluesight が病院の 340B コンプライアンス向け AI を構築

Vercel Blog2026年7月14日 09:00

Vercel Blob、プライベートストレージでの一貫性ある読み取りをサポート

今日のまとめ

AI日報で今日の重要ニュースをまとめ読み

ニュース一覧に戻る元記事を読む

Hugging Face Blog·2026年5月19日 09:00·約21分

Ettin リランカーファミリーの紹介

#RAG #CrossEncoder #ModernBERT #知識蒸留 #Sentence Transformers

TL;DR

AI深層分析2026年7月5日 18:08

重要/ 5段階

深度40%

キーポイント

多サイズモデルのリリース

17M から 1B パラメータまでの 6 つの異なるサイズの CrossEncoder リランカーが、それぞれの規模で最良の性能を達成するよう設計されて公開された。

ModernBERT エンコーダーの採用

大規模モデルからの知識蒸留

完全なトレーニングレシピの公開

柔軟なサイズ選択とコンテキスト長

品質と速度のトレードオフに応じて6種類のモデルサイズから選べ、ModernBERTベースにより最大8Kトークンの文脈を処理可能。

高速化のための推奨設定

最高スループットを得るにはkernelsのインストールとbfloat16、flash_attention_2の設定が推奨され、最大8.3倍の速度向上が見込める。

高速なエンドツーエンドパイプラインの構築

静的埋め込みモデルによるサブミリ秒単位の高速検索と、Ettin Rerankerを用いた最終順位付けを組み合わせた完全な例が示されています。

重要な引用

Today I'm releasing six new Sentence Transformers CrossEncoder rerankers, state-of-the-art at their respective sizes

The models were trained with a distillation recipe: pointwise MSE on mixedbread-ai/mxbai-rerank-large-v2 scores

All six accept up to 8K tokens of context (useful for long-document reranking) thanks to ModernBERT's long-context pre-training.

It is recommended to install kernels and set model_kwargs={"dtype": "bfloat16", "attn_implementation": "flash_attention_2"} for the highest throughput.

Fast retrieval with a static embedder (sub-millisecond on CPU per query)

Step 1: encode + retrieve top-100

影響分析・編集コメントを表示

影響分析

編集コメント

記事一覧に戻る

TL;DR（要約）

cross-encoder/ettin-reranker-17m-v1

cross-encoder/ettin-reranker-32m-v1

cross-encoder/ettin-reranker-68m-v1

cross-encoder/ettin-reranker-150m-v1

cross-encoder/ettin-reranker-400m-v1

cross-encoder/ettin-reranker-1b-v1

*Our six rerankers paired with google/embeddinggemma-300m on MTEB(eng, v2) Retrieval. See Results for five more embedder pairings.*

What is a reranker, and why pair one with an embedder?

Usage

エンドツーエンドの retrieve-then-rerank パイプライン

Architecture Details

Results

MTEB(eng, v2) Retrieval

Speed

Training

蒸留レシピ

Dataset

Training Arguments

Evaluation

Overall Training Script

Conclusion

Acknowledgements

What is a reranker, and why pair one with an embedder?

本ブログ記事全体を通じて、「リランカー」と「クロスエンコーダー」は互換的に使用されます。

使用方法

公開されているモデルは通常の Sentence Transformers CrossEncoder モデルですので、3 行のコードだけで利用可能です：

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ettin-reranker-32m-v1")

scores = model.predict([

("Where was Apple founded?", "Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne."),

("Where was Apple founded?", "The Fuji apple is an apple cultivar developed in the late 1930s and brought to market in 1962."),

])

print(scores)

[11.393298 2.968891] <- larger means more relevant

クエリと候補リストに対しては、rank を使用してソートされたインデックスとスコアを取得することもできます:

ranked = model.rank(

query="Which planet is known as the Red Planet?",

documents=[

"Venus is often called Earth's twin because of its similar size and proximity.",

"Mars, known for its reddish appearance, is often referred to as the Red Planet.",

"Jupiter, the largest planet in our solar system, has a prominent red spot.",

"Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",

top_k=4,

return_documents=True,

)

for r in ranked:

print(f"({r['score']:.2f}): {r['text']}")

(10.82): Mars, known for its reddish appearance, is often referred to as the Red Planet.

(9.86): Saturn, famous for its rings, is sometimes mistaken for the Red Planet.

(8.55): Jupiter, the largest planet in our solar system, has a prominent red spot.

(6.21): Venus is often called Earth's twin because of its similar size and proximity.

from sentence_transformers import CrossEncoder

model = CrossEncoder(

"cross-encoder/ettin-reranker-32m-v1",

model_kwargs={"dtype": "bfloat16", "attn_implementation": "flash_attention_2"},

)

検索後再ランク付けの完全エンドツーエンドパイプライン

検索には高速な埋め込みモデルを、最終的な順序付けには再ランク付け器を使用する完全な例を示します：

from sentence_transformers import SentenceTransformer, CrossEncoder

静的埋め込みモデルによる高速検索（CPU 上でクエリあたりサブミリ秒）

embedder = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1")

reranker = CrossEncoder("cross-encoder/ettin-reranker-68m-v1")

corpus = [

"Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne.",

"The Fuji apple is an apple cultivar developed in the late 1930s.",

"Steve Jobs introduced the iPhone in 2007 at Macworld.",

"Macintosh computers were sold by Apple from 1984 onward.",

# ... thousands or millions more in production

]

query = "Where was Apple founded?"

Step 1: encode + retrieve top-100

query_emb = embedder.encode_query(query, convert_to_tensor=True)

corpus_emb = embedder.encode_document(corpus, convert_to_tensor=True)

scores = embedder.similarity(query_emb, corpus_emb)[0]

top_k_idx = scores.topk(min(100, len(corpus))).indices.tolist()

Step 2: rerank

top_k_docs = [corpus[i] for i in top_k_idx]

ranked = reranker.rank(query, top_k_docs, top_k=5, return_documents=True)

for r in ranked:

print(f"({r['score']:.2f}): {r['text']}")

(11.63): Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne.

(4.71): Steve Jobs introduced the iPhone in 2007 at Macworld.

(1.96): The Fuji apple is an apple cultivar developed in the late 1930s.

(1.49): Macintosh computers were sold by Apple from 1984 onward.

アーキテクチャの詳細

Transformer(FA2)
Pooling(cls)
Dense(H, H, bias=False, GELU)
LayerNorm(H)
Dense(H, 1, scores)

モデル

バックボーン

隠れ層サイズ

層数

パラメータ（ヘッド含む）

256

17.6M

384

32.8M

512

68.6M

768

150.9M

1024

401.6M

1792

1.00B

これら 6 つのモデルは、Ettin エンコーダーと同様に Apache 2.0 ライセンスの下でリリースされています。

結果

MTEB(eng, v2) リトリーバル

埋め込みモデル

アクティブパラメータ数

検索のみ NDCG@10

0.3495

23M

0.4292

33M

0.5149

137M

0.5226

308M

0.5463

596M

0.5980

結果の全表（クリックして展開）

Reranker

パラメータ数

MTEB(eng, v2) 検索 NDCG@10

4.02B

0.6367

1.54B

0.6115

1.00B

0.6114

401M

0.6091

151M

0.5994

596M

0.5940

494M

0.5920

68.6M

0.5915

2.44B

0.5856

150M

0.5843

32.8M

0.5779

150M

0.5656

17.6M

0.5576

568M

0.5526

4.02B

0.5300

560M

0.5098

22.7M

0.5082

33.4M

0.5066

435M

0.5063

19.2M

0.4979

70.8M

0.4968

278M

0.4890

184M

0.4865

NanoBEIR の完全な結果表（クリックして展開）

Reranker

パラメータ数

NanoBEIR 平均 NDCG@10

1.54B

0.7318

1.00B

0.7237

2.44B

0.7197

401M

0.7193

494M

0.7162

151M

0.7086

150M

0.7017

568M

0.6971

68.6M

0.6915

150M

0.6909

32.8M

0.6825

17.6M

0.6746

435M

0.6488

560M

0.6379

33.4M

0.6369

22.7M

0.6312

19.2M

0.6298

184M

0.6231

70.8M

0.6136

278M

0.6027

速度

表 1. 秒間あたりのペア数（すべて bfloat16）。当社の 6 つの再ランク化器（rerankers）は太字で示されています。

Model | Params | Attn | pairs / second

---|---|---|---

cross-encoder/ettin-reranker-17m-v1 | 17M | FA2 | 7517

cross-encoder/ettin-reranker-32m-v1 | 32M | FA2 | 6602

cross-encoder/ettin-reranker-68m-v1 | 68M | FA2 | 4913

cross-encoder/ms-marco-MiniLM-L4-v2 | 19M | FA2 | 4029

cross-encoder/ms-marco-MiniLM-L6-v2 | 22M | FA2 | 3817

cross-encoder/ms-marco-MiniLM-L12-v2 | 33M | FA2 | 3311

cross-encoder/ettin-reranker-150m-v1 | 150M | FA2 | 3237

BAAI/bge-reranker-base | 278M | FA2 | 2858

mixedbread-ai/mxbai-rerank-xsmall-v1 | 70M | eager | 2636

184M

eager

1953

400M

FA2

1738

560M

FA2

1659

568M

FA2

1569

150M

FA2

1418

150M

FA2

1404

FA2

928

435M

eager

867

494M

FA2

809

1.5B

FA2

387

Same benchmark on a consumer GPU (RTX 3090, 24 GB)

Model

Params

Best attn

pairs / second

17M

FA2（Flash Attention 2）

9008

19M

FA2

5071

32M

FA2（Flash Attention 2）

4497

22M

FA2

4234

33M

FA2

2847

68M

FA2（Flash Attention 2）

1916

70M

eager（イーガーモード）

1677

278M

FA2

1329

150M

FA2（Flash Attention 2）

982

184M

eager

772

150M

FA2

598

150M

FA2

586

560M

FA2

448

568M

FA2

436

400M

FA2

429

435M

eager

266

494M

FA2

221

FA2

189

1.5B

FA2

同じベンチマークを CPU（Intel Core i7-13700K）で実行した場合

モデル | パラメータ数 | 最適アテンション | 秒間ペア数

cross-encoder/ettin-reranker-17m-v1 | 17M | SDPA | 267.4

cross-encoder/ms-marco-MiniLM-L4-v2 | 19M | SDPA | 206.2

cross-encoder/ms-marco-MiniLM-L6-v2 | 22M | SDPA | 143.9

cross-encoder/ettin-reranker-32m-v1 | 32M | SDPA | 92.5

cross-encoder/ms-marco-MiniLM-L12-v2 | 33M | SDPA | 75.9

mixedbread-ai/mxbai-rerank-xsmall-v1 | 70M | eager | 38.9

68M

SDPA

31.2

278M

SDPA

19.2

150M

SDPA

14.7

150M

SDPA

14.5

150M

SDPA

14.0

184M

eager

13.4

560M

SDPA

6.2

568M

SDPA

6.0

400M

SDPA

5.2

435M

eager

4.3

494M

SDPA

3.5

SDPA

2.1

原文を表示

Back to Articles

TL;DR

cross-encoder/ettin-reranker-17m-v1

cross-encoder/ettin-reranker-32m-v1

cross-encoder/ettin-reranker-68m-v1

cross-encoder/ettin-reranker-150m-v1

cross-encoder/ettin-reranker-400m-v1

cross-encoder/ettin-reranker-1b-v1

*Our six rerankers paired with google/embeddinggemma-300m on MTEB(eng, v2) Retrieval. See Results for five more embedder pairings.*

I bootstrapped the training recipe below with the new train-sentence-transformers Agent Skill shipped in Sentence Transformers v5.5.0. Install it with hf skills add train-sentence-transformers [--global] [--claude] and ask your AI coding agent (Claude Code, Codex, Cursor, Gemini CLI, ...) to fine-tune a SentenceTransformer, CrossEncoder, or SparseEncoder model on your data.

What is a reranker, and why pair one with an embedder?

Usage

End-to-end retrieve-then-rerank pipeline

Architecture Details

Results

MTEB(eng, v2) Retrieval

Speed

Training

Distillation recipe

Dataset

Training Arguments

Evaluation

Overall Training Script

Conclusion

Acknowledgements

What is a reranker, and why pair one with an embedder?

Throughout this blogpost I'll use "reranker" and "cross-encoder" interchangeably.

Usage

The released models are normal Sentence Transformers CrossEncoder models, so you can use them with just 3 lines of code:

code

from sentence_transformers import CrossEncoder

model = CrossEncoder("cross-encoder/ettin-reranker-32m-v1")
scores = model.predict([
    ("Where was Apple founded?", "Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne."),
    ("Where was Apple founded?", "The Fuji apple is an apple cultivar developed in the late 1930s and brought to market in 1962."),
])
print(scores)
# [11.393298  2.968891]   <- larger means more relevant

For a query and a list of candidates, you can also use rank to get back sorted indices and scores:

code

ranked = model.rank(
    query="Which planet is known as the Red Planet?",
    documents=[
        "Venus is often called Earth's twin because of its similar size and proximity.",
        "Mars, known for its reddish appearance, is often referred to as the Red Planet.",
        "Jupiter, the largest planet in our solar system, has a prominent red spot.",
        "Saturn, famous for its rings, is sometimes mistaken for the Red Planet.",
    ],
    top_k=4,
    return_documents=True,
)
for r in ranked:
    print(f"({r['score']:.2f}): {r['text']}")
# (10.82): Mars, known for its reddish appearance, is often referred to as the Red Planet.
# (9.86): Saturn, famous for its rings, is sometimes mistaken for the Red Planet.
# (8.55): Jupiter, the largest planet in our solar system, has a prominent red spot.
# (6.21): Venus is often called Earth's twin because of its similar size and proximity.

code

from sentence_transformers import CrossEncoder

model = CrossEncoder(
    "cross-encoder/ettin-reranker-32m-v1",
    model_kwargs={"dtype": "bfloat16", "attn_implementation": "flash_attention_2"},
)

End-to-end retrieve-then-rerank pipeline

A complete example with a fast embedder for retrieval and the reranker for the final ordering:

code

from sentence_transformers import SentenceTransformer, CrossEncoder

# Fast retrieval with a static embedder (sub-millisecond on CPU per query)
embedder = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1")
reranker = CrossEncoder("cross-encoder/ettin-reranker-68m-v1")

corpus = [
    "Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne.",
    "The Fuji apple is an apple cultivar developed in the late 1930s.",
    "Steve Jobs introduced the iPhone in 2007 at Macworld.",
    "Macintosh computers were sold by Apple from 1984 onward.",
    # ... thousands or millions more in production
]
query = "Where was Apple founded?"

# Step 1: encode + retrieve top-100
query_emb = embedder.encode_query(query, convert_to_tensor=True)
corpus_emb = embedder.encode_document(corpus, convert_to_tensor=True)
scores = embedder.similarity(query_emb, corpus_emb)[0]
top_k_idx = scores.topk(min(100, len(corpus))).indices.tolist()

# Step 2: rerank
top_k_docs = [corpus[i] for i in top_k_idx]
ranked = reranker.rank(query, top_k_docs, top_k=5, return_documents=True)
for r in ranked:
    print(f"({r['score']:.2f}): {r['text']}")
# (11.63): Apple Inc. was founded in Cupertino, California in 1976 by Steve Jobs, Steve Wozniak, and Ronald Wayne.
# (4.71): Steve Jobs introduced the iPhone in 2007 at Macworld.
# (1.96): The Fuji apple is an apple cultivar developed in the late 1930s.
# (1.49): Macintosh computers were sold by Apple from 1984 onward.

This is the same shape used by most modern search systems. The retriever decides what enters the funnel, the reranker decides what wins.

Architecture Details

code

1. Transformer(FA2)
2. Pooling(cls)
3. Dense(H, H, bias=False, GELU)
4. LayerNorm(H)
5. Dense(H, 1, scores)

Model

Backbone

Hidden size

Layers

Params (head incl.)

256

17.6M

384

32.8M

512

68.6M

768

150.9M

1024

401.6M

1792

1.00B

All six models are released under the Apache 2.0 license, matching the Ettin encoders.

Results

MTEB(eng, v2) Retrieval

Embedding Model

Active params

Retriever-only NDCG@10

0.3495

23M

0.4292

33M

0.5149

137M

0.5226

308M

0.5463

596M

0.5980

The dashed retriever-only line in each chart below is the headline number to beat. Anything below it means the reranker actively hurts the pipeline on average:

Full table of results (click to expand)

Mean NDCG@10 over the 6 embedder pairings, sorted descending. Our six models are in bold, and the teacher mixedbread-ai/mxbai-rerank-large-v2 is underlined.

Reranker

Params

MTEB(eng, v2) Retrieval NDCG@10

4.02B

0.6367

1.54B

0.6115

1.00B

0.6114

401M

0.6091

151M

0.5994

596M

0.5940

494M

0.5920

68.6M

0.5915

2.44B

0.5856

150M

0.5843

32.8M

0.5779

150M

0.5656

17.6M

0.5576

568M

0.5526

4.02B

0.5300

560M

0.5098

22.7M

0.5082

33.4M

0.5066

435M

0.5063

19.2M

0.4979

70.8M

0.4968

278M

0.4890

184M

0.4865

† Capped to max_seq_length=8192 (the 4B Qwen3-based rerankers don't fit on a single H100 80GB at native context). Native-context evaluation is likely higher.

Full table of NanoBEIR results (click to expand)

Reranker

Params

NanoBEIR mean NDCG@10

1.54B

0.7318

1.00B

0.7237