Masked language modeling (MLM)

Pretraining objective that hides random input tokens and trains a model to reconstruct them from bidirectional context; central to BERT-style encoder models.

Masked language modeling (MLM) is a language model training objective used mainly with encoder-only Transformers. A fraction of input tokens is replaced with a mask symbol (or a random token); the network must predict the original tokens using both left and right context. That bidirectionality is what makes MLM the standard pretraining task for BERT-style models in natural language processing.

MLM differs from the left-to-right next-token objective used by GPT-style decoders, which scale to modern LLMs but cannot read future tokens during pretraining. Hybrid and later architectures sometimes combine both ideas. See the BERT article and timeline entry.