dmis-lab Releases UltraBERT, 150M and 380M Encoders With 32K Context, Plus an Embedding Model It Says Tops MTEB Under 200M Parameters

dmis-lab's Oct 14 UltraBERT post lists Apache 2.0 encoders at 150M and 380M with 32K context; benchmark claims, including MTEB, are the authors' own.

Verified pipeline

Sources: 4 Publisher: signed Contributor: signed Hash: eff530adb9 View

Editor's Note · October 15, 2026

Clarification:

The article states that the sources read contain no independent evaluation of the models. The blog post's comment thread includes an informal fine-tuning test by a Hugging Face user, Tom Aarsen, who writes that "a UltraBERT-150M finetuned as a dense embedding model outperforms e.g. ModernBERT-large (~400M) and Ettin-400M finetuned in that same way." It is a single comment on a 32k-triplet MS MARCO run, not a formal evaluation, and the article's other benchmark figures remain the authors' own.

Overview

The Data Mining and Information Systems Lab (dmis-lab) published UltraBERT in a Hugging Face community post dated October 14, 2026. The post describes “a family of English encoders (150M and 380M) with a native 32K context, plus UltraBERT-Embed-150M, a ready-to-use embedding model built on top of it,” and says everything is Apache 2.0. The accompanying arXiv paper, submitted on 10 October 2026 by Jungwoo Park, Taewhoo Lee and Jaewoo Kang, is titled “UltraBERT: Beyond the Limit of BERT Pre-Training.” All performance figures below are the authors’ own.

What Is in the Release

The UltraBERT-380M model card lists a 150M encoder with a hidden size of 768 and a 380M encoder with a hidden size of 1,024, both with a 32K context. The UltraBERT-Embed-150M card says that model maps text to a 768-dimensional vector, accepts inputs up to 32K tokens, follows task instructions on the query side and embeds code as well as text. Both cards carry an apache-2.0 license field.

Parameter counts differ between pages from the same authors. The model cards list 150M, while the blog post’s benchmark tables list 152M for the base encoder and 151M for the embedding model.

For use, the 380M card says the model needs transformers >= 5 and trust_remote_code=True. The embedding card adds sentence-transformers >= 6.0.

How the Authors Say It Was Built

According to the blog post, ablations across all runs added up to more than 70 trillion training tokens. It reports that encoders stop improving after 1-2T tokens even as loss keeps falling, and that low masking ratios favor classification while high ratios favor retrieval. UltraBERT mixes 5% and 50% masked sequences at a 4:1 ratio in each batch, with a separate language-model head for each ratio and a shared encoder body. Training ran on 2T tokens, the last 100B at 32K context. The 5% head learns from a 1.3B teacher with a KL loss, while the 50% head uses standard masked language modeling.

The paper abstract states the same three findings and adds: “To our knowledge, UltraBERT is the first encoder to achieve state of the art on both classification and retrieval tasks, reaching 91.0 on GLUE and 47.7 on BEIR.”

Reported Results

The blog post reports the top score on 9 of 10 benchmarks at both sizes. Its tables give the 150M encoder 89.5 on GLUE and 45.2 on BEIR, and the 380M encoder 91.0 and 47.7. For comparison, the same table lists Ettin Encoder (149M) at 88.9 on GLUE and 42.4 on BEIR. The post says its retrieval scores come from fine-tuning each backbone on MS MARCO, and MLDR for long context, under the same sweep. It also reports 99% passkey accuracy up to 32K tokens.

For the embedding model, the post says it was trained in two contrastive stages, using about 500M weakly supervised text and code pairs and then 5.7M curated examples, with public data only and no distillation from a larger embedding model. The model card reports 67.24 on MTEB(eng, v2) and says that, as of October 11, 2026, this is the state of the art among dense embedding models under 200M parameters. The next entries in the card’s table are Tarka-Embedding-150M-V1 at 66.39 and gte-modernbert-base at 66.26, so by The Machine Herald’s arithmetic the lead over the runner-up is 0.85 points.

The same table shows the lead is not uniform across task groups. In its classification column the card lists 81.74 for UltraBERT-Embed-150M, against 86.28 for Tarka-Embedding-150M-V1 and 87.13 for F2LLM-v2-160M.

What We Don’t Know

The sources read for this article contain no independent evaluation of the models, so every benchmark claim, including the “first encoder” claim, rests on the authors. The embedding card’s link to the paper was empty when read, although the blog post links the arXiv entry. The post is labeled a community article.