Transformer Attention: How does ROPE work?

Published

This is a series where I answer the questions my tutoring students really ask – the ones textbooks tend to skip. I teach machine learning, deep learning, statistics, and math to learners of all levels.

The question. A student who had just finished the attention posts came back with a sharper one: “Attention computes every pair of tokens at once, and the softmax does not care about order. So how does the model know that ‘Your journey starts’ and ‘starts journey Your’ are different sentences? Everyone says RoPE, but what is actually rotating?”

The answer is beautifully concrete: the query and key vectors themselves are rotated in their plane, by an angle proportional to the token’s position. Let us do one rotation by hand.

The setup: one pair of numbers per token

RoPE groups the dimensions of a query or key vector into pairs and treats each pair as a little 2-D vector. Each pair gets rotated by an angle that grows with the token’s position m. We stay with our usual six tokens – “Your journey starts with one step” – so “Your” sits at m = 0, “journey” at m = 1, and “step” at m = 5.

The angle for position m and pair k is:

angle(m, k) = m * theta^(-2k/d)

With the smallest interesting case, d = 2, there is only one pair (k = 0) and the frequency is theta^0 = 1, so the angle is simply m radians. “journey” rotates by exactly 1 radian.

RoPE rotation steps for the token journey

The arithmetic: rotating “journey” by one radian

Suppose the key-side pair for “journey” before rotation is (1.2445, 1.3189). A 2-D rotation by angle alpha uses the matrix every graphics programmer knows:

[ x1' ]   [ cos a  -sin a ] [ x1 ]
[ x2' ] = [ sin a   cos a ] [ x2 ]

cos(1) = 0.5403,  sin(1) = 0.8415

x1' = 1.2445 * 0.5403 - 1.3189 * 0.8415 =  0.6724 - 1.1099 = -0.4374
x2' = 1.2445 * 0.8415 + 1.3189 * 0.5403 =  1.0473 + 0.7126 =  1.7598

The vector keeps its length but swings from angle 0.814 rad to 1.814 rad – exactly one radian of rotation, encoding exactly one step of position. “starts” would swing 2 radians, “step” 5 radians. Position is now baked into the direction of the vector, and nothing was added to it.

Why rotation and not addition: the relative-position gift

Here is the payoff that made RoPE the industry default. When a query at position i meets a key at position j, both rotated, their dot product depends only on the distance i - j, never on the absolute positions. You can see it with the simplest possible pair, q = k = (1, 0), where the rotated score reduces to cos(i - j):

score(i, j) = cos(i - j)

positions (2,0):  cos(2) = -0.416
positions (4,2):  cos(2) = -0.416
positions (5,3):  cos(2) = -0.416

Three completely different locations in the sentence, one identical score, because all that matters is that the tokens are two apart. That is exactly the bias you want for language: “starts with” means the same thing whether it appears at the top of the page or the bottom. Adding a position vector cannot give you this; rotating can, because rotating both vectors by the same extra amount never changes the angle between them.

Real models use more than one pair. With head dimension 4 you get two pairs, and the second pair rotates at a slower frequency, theta^(-1⁄2). Fast pairs track fine-grained nearby positions, slow pairs stay coherent across long distances – the same idea as the hands of a clock.

DeepSeek’s twist: decoupled RoPE

DeepSeek-V2 runs into a conflict: its MLA attention compresses keys and values into a tiny latent vector to save cache memory, but RoPE needs to rotate keys after projection, which would force the model to cache full-size keys again. The escape is to decouple. Every query and key is split into two channels:

  • a token channel, reconstructed from the shared latent, carrying content but no position, and
  • a RoPE channel, a small dedicated vector (2 dimensions in our toy, 64 in the real model) that carries position and nothing else.

The two channels are concatenated only inside the score computation. The RoPE key is a single vector shared by all heads, so the cache grows by only a handful of numbers per token instead of a full key per head. Position information survives; the cache savings survive too.

Gemma’s twist: dual RoPE

Gemma faces a different problem. Its local sliding-window layers only ever compare tokens a few positions apart, while its occasional global layers compare tokens across the entire 128K context. One rotation speed cannot serve both: too fast and distant positions alias (the angles wrap around and scramble), too slow and nearby positions blur together.

So each layer type gets its own base frequency theta. Here is the picture, with toy bases 10 and 10000 standing in for Gemma 3’s real 10000 and 1000000:

Dual RoPE with fast theta for local layers and slow theta for the global layer

With head dimension 4, the frequencies work out to:

local  theta = 10:     inv_freq = (1.000, 0.316)   fast dial, sharp nearby
global theta = 10000:  inv_freq = (1.000, 0.010)   slow dial, stable far away

Watch the second pair at position m = 5: the local layer rotates it by 5 * 0.316 = 1.581 radians, clearly distinguishable from its neighbors, while the global layer rotates it by only 5 * 0.010 = 0.050 radians, so a key from thousands of tokens ago still points in nearly the same direction as a recent one and the angles never wrap into nonsense. The dials in the figure show this at a glance: the local hands fan out, the global hand barely moves. Note that RoPE is applied to q and k only – v is never rotated, because position should decide who attends, not what gets copied.

One thing to try

Take the pair (1, 0) for both q and k, as in the score table above, and rotate them by hand at positions i = 3 and j = 1. You should get cos(2) = -0.416 again. Now add the same arbitrary extra rotation – say 0.7 radians – to both vectors and recompute the dot product. It does not move. That small exercise is the whole reason RoPE encodes relative position, and it is one line of trigonometry: cos(A)cos(B) + sin(A)sin(B) = cos(A - B).

The arithmetic and diagrams in this article are adapted from the teaching notebooks I use with my students, where every rotation is computed by hand before any library is allowed.

References

  1. Su, J., et al. (2021). “RoFormer: Enhanced Transformer with Rotary Position Embedding.” arXiv:2104.09864.
  2. DeepSeek-AI (2024). “DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.” arXiv:2405.04434.
  3. Gemma Team, Google DeepMind (2025). “Gemma 3 Technical Report.” arXiv:2503.19786.
  4. Vaswani, A., et al. (2017). “Attention Is All You Need.” Advances in Neural Information Processing Systems 30 (NeurIPS 2017).

I tutor machine learning, deep learning, statistics, and math. If attention – or anything else – is not clicking for you, you can find me on Wyzant.