Transformer Attention: What are Query, Key, and Value?

This is a series where I answer the questions my tutoring students really ask – the ones textbooks tend to skip. I teach machine learning, deep learning, statistics, and math to learners of all levels.

The question. “Attention is softmax(QK^T)V. Fine. But why three matrices? Why does every token need a query, a key, AND a value?”

The shortest honest answer: attention is a dictionary lookup that works without exact matches. Here is the whole idea in one picture, adapted from our teaching notebook.

The Python dictionary analogy

A Python dict stores key-value pairs and answers queries – but only exact ones. Ask for ‘feline’ when the key is ‘cat’ and you get nothing:

d = {'kitty': 10.0, 'cat': 9.0, 'dog': -7.5}

d['cat']     -&gt; 9.0
d['feline']  -&gt; None     # close, but not exact
d['potato']  -&gt; None

Attention replaces exact match with similarity match. The query is compared against every key, the similarities pass through a softmax so they sum to one, and the answer is a weighted blend of ALL the values:

query 'cat':    0.2*10.0 + 0.8*9.0 + 0.0*(-7.5) = 9.2
query 'feline': 0.3*10.0 + 0.7*9.0 + 0.0*(-7.5) = 9.3   # no exact key, still useful
query 'potato': 0.3*10.0 + 0.3*9.0 + 0.4*(-7.5) = 2.7   # nothing matches, mostly 'dog'

‘feline’ never appears as a key, yet it retrieves a sensible answer because it is similar to ‘kitty’ and ‘cat’. That is the entire trick: a dictionary that gives partial credit.

The Python dictionary analogy for query key and value in attention

Mapping the analogy onto the transformer

In a real attention layer, the database is not given – it is built from the sentence itself. Every token produces all three artifacts through learned projections:

query_i = x_i @ W_query    # what this token is looking for
key_i   = x_i @ W_key      # what this token advertises about itself
value_i = x_i @ W_value    # what this token shares if selected

Then the lookup runs exactly like the soft dictionary:

weights = softmax(query_i . key_j  for all j)   # similarity -&gt; distribution
output_i = sum_j weights_j * value_j            # blended retrieval

Why split one embedding into three roles? Because the same word plays three different parts. The word “it” in “the animal didn’t cross the street because it was too tired” needs to look for something tired (query), while every candidate noun advertises its own properties (key), and the retrieved content (value) is what actually flows forward into the next layer. Three separate learned matrices let the model specialize each role instead of forcing one vector to do all three jobs.

One constraint worth noticing: the softmax weights always sum to one, so the output is a weighted average and never explodes – the price is that attention cannot “look up two things at full strength” the way you might hold two exact dict results at once.

One thing to try

In the soft-lookup arithmetic above, change potato’s weights from (0.3, 0.3, 0.4) to (0.0, 0.0, 1.0). The answer becomes -7.5, a pure ‘dog’ retrieval. Now set them to (¹⁄₃, ¹⁄₃, ¹⁄₃) and you get 3.8, the plain average – this is what attention degenerates to when nothing is similar to anything. Play with the weights until the behavior feels obvious, because those weights are the only thing the network ever learns to control.

The analogy and arithmetic in this article are adapted from the teaching notebooks I use with my students.

References

Vaswani, A., et al. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems 30 (NeurIPS 2017).
Raschka, S. (2024). Build a Large Language Model (From Scratch). Manning.

I tutor machine learning, deep learning, statistics, and math. If attention – or anything else – is not clicking for you, you can find me on Wyzant.