Liquid AI、CPU で高速な 8K コンテキスト対応バイディレクショナルエンコーダーを公開

Liquid AI has released two open-weight bidirectional encoders, LFM2.5-Encoder-230M and LFM2.5-Encoder-350M. Both are masked language models built on the LFM2 hybrid backbone. Both carry an 8,192-token context.

Encoders sit underneath classifiers, intent routers, safety filters, and PII detectors. Those jobs run continuously, usually without a GPU, and increasingly on longer inputs. BERT established the class. ModernBERT pushed its accuracy, speed, and context. Liquid AI’s argument is that the LFM2 architecture continues that line, because its cost grows more slowly as inputs get longer.

How a decoder became an encoder

The encoders are not trained from scratch. They are initialized from the LFM2.5-230M and LFM2.5-350M decoder backbones, then converted with three changes:

First, the causal attention mask is replaced with a bidirectional one, so every token attends to both sides.

Second, the LFM2 short convolutions are made non-causal using symmetric center padding, so each token’s convolution mixes in neighbours on both sides.

Third, the model is trained with a masked language modeling objective at a 30% mask rate. That is denser than BERT’s 15%, following evidence Liquid AI cites that a higher mask rate helps at this scale.

Training runs in two stages:

Stage one establishes general language competence with a short-context MLM objective on a large web corpus at 1,024 tokens.

Stage two extends context to 8,192 tokens on the full data mix, strengthening factual, legal, and multilingual competence.

Architecturally, the backbone interleaves gated short-convolution blocks with grouped-query attention, the same design described in the LFM2 technical report. Both checkpoints use a hidden size of 1024 and a 65,536-token vocabulary, and support 15 languages. The license is the LFM Open License v1.0.

The embed below breaks down the conversion, the full ranking table, the per-task scores, and the CPU latency figures.

The benchmark results

Liquid AI evaluated 14 models on 17 tasks pulled from GLUE, SuperGLUE, and multilingual classification. Every model is fully fine-tuned per task, and the reported score is that fine-tuned model’s result.

LFM2.5-Encoder-350M posts a 17-task mean of 81.02 (±1.00), ranking fourth. The three models ahead of it are all larger. They are XLM-R XL at 3.5B (83.06), ModernBERT-large at 395M (81.68), and XLM-R large at 560M (81.34). The top model is nearly 10 times its size.

LFM2.5-Encoder-230M posts 79.29 (±1.02), ranking sixth. It beats ModernBERT-base at 78.19 and every EuroBERT model in the table, including EuroBERT-610M (75.87) and EuroBERT-2.1B (72.19). Both new encoders also score above Liquid AI’s own retrieval siblings, LFM2.5-ColBERT-350M (76.18) and LFM2.5-Embedding-350M (75.68). That gap is the stated reason Liquid AI built a general-purpose encoder instead of reusing the retrievers.

The methodology is the more interesting part, and it is open-sourced under Apache-2.0. Every model is loaded with fp32 master weights and bf16 autocast, so the table compares models rather than number formats. Every model uses the same AdamW recipe, taken from the EuroBERT card. Learning rate is selected per model and task across 10 rates and 3 seeds. Scores are then reported as the mean over 5 fresh seeds that never touched selection. The transformers version is pinned to 4.56.2 so dependency drift is not an uncontrolled variable.

Use Cases and Deployment Environments

The release names three settings. Edge and embedded devices come first. A car’s onboard compute or an industrial controller has no spare GPU, and cannot afford a cloud round trip. Regulated and on-premise systems in finance, healthcare, and legal, where documents are long, sensitive, and cannot leave in-house infrastructure. And high-volume cost-sensitive pipelines, where a small encoder acts as a cheap first pass in front of a larger model.

Liquid AI also puts a useful number on the context window: 8,192 tokens is roughly 13 to 15 pages. One forward pass covers a full contract or a complete patient record.

To show what a fine-tuned encoder looks like, the research team shipped five demos. Each runs in a CPU-only Hugging Face Space. They cover zero-shot prompt routing, zero-shot policy linting, and spell checking. A PII detector handles 40 information types across 16 languages. A bonus masked-diffusion demo runs the encoder as a chatbot that generates by iteratively unmasking.

Getting them running

Both encoders load through transformers. The body is exposed as Lfm2BidirectionalModel and masked-LM loading uses Lfm2BidirectionalForMaskedLM. Both are wired through auto_map, so trust_remote_code=True is required on every load call.

A base encoder produces general-purpose representations, not task outputs, so fine-tuning is mandatory. Liquid AI’s fine-tuning tutorial walks through long legal documents at an 8k context configuration. The model selection guidance is straightforward: 350M when accuracy matters most, 230M for tighter hardware or higher throughput.

Key Takeaways

Liquid AI released LFM2.5-Encoder-230M and LFM2.5-Encoder-350M, open-weight bidirectional encoders with 8,192-token context.

Both convert LFM2.5 decoders into encoders via bidirectional masking, non-causal short convolutions, and 30% MLM.

LFM2.5-Encoder-350M ranks 4th of 14 models at 81.02; the three above it are all larger.

At 8K tokens on CPU, the 230M runs one forward pass in ~28s versus over 90s for ModernBERT-base.

Check out the Liquid AI blog post, the Hugging Face write-up, and the model cards for LFM2.5-Encoder-230M and LFM2.5-Encoder-350M.

The post Liquid AI Releases LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: Bidirectional Encoders That Stay Fast at 8K Context on CPU appeared first on MarkTechPost.