What is multi-head attention?

Multi-head attention runs several attention mechanisms in parallel, each with its own learned projections. Each head can focus on different relationships, such as syntactic structure, semantic similarity, or coreference. The outputs of all heads are concatenated and projected back to the original dimension. This gives the model richer representational capacity than a single attention layer.