What is Rotary Position Embedding (RoPE)?
LLM Engineer Interview Questions: Transformer Architecture, Self-Attention, and Modern LLM Foundations
Audio flashcard · 0:20Nortren·
What is Rotary Position Embedding (RoPE)?
0:20
Rotary Position Embedding, or RoPE, encodes token positions by rotating query and key vectors in pairs of dimensions by an angle proportional to the position. Unlike absolute positional encodings, RoPE naturally captures relative position and extrapolates better to longer sequences than seen in training. It is now standard in models like Llama, Mistral, and Qwen.
arxiv.org