vasiliy.ch
← Блог
Ссылка 20 ноября 2026

Positional Embeddings in LLMs

Читать на: Русский English Srpski

Ссылка · magazine.sebastianraschka.com Understanding Positional Embeddings in Modern LLMs ↗

How do LLMs know which token comes earlier and which comes later? The tokens themselves don't give any hint.

In GPT‑2, the vector of a token's position was added to the token's vector. In the original transformer, positions were encoded using sine and cosine. Both methods tell the model where a word is located, but modern LLMs more often use RoPE (Rotary Position Embedding). By rotating the vectors of the tokens the model compares with each other, it takes the distance between the tokens into account. The meaning is preserved even when the positions of the words change.

A masterful longread by Sebastian Raschka on how this works.