What are query, key, and value vectors in attention?
LLM Engineer Interview Questions: Transformer Architecture, Self-Attention, and Modern LLM Foundations
Audio flashcard · 0:18Nortren·
What are query, key, and value vectors in attention?
0:18
Query, key, and value are three projections of each token's embedding produced by separate linear layers. The query represents what the current token is looking for. The key represents what other tokens offer. The value is the actual content that gets passed forward. Attention scores are computed as the dot product of queries and keys, then applied to the values.
arxiv.org