The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups

This paper proposes Lie-Algebra Attention, where the token itself is a group element g_i rather than a vector with an attached feature payload. That sounds abstract, but the practical idea is simple: compare tokens through their relative transform g_i^{-1}g_j, map it with log(), and score proximity with a closed-form algebra norm instead of a learned attention kernel.

For developers, the interesting part is that this works directly on matrix Lie groups, including non-compact affine groups with scale and shear. The author argues that many existing equivariant methods rely on irreps, spherical harmonics, or surjective-exp constructions that cannot cover those groups. Here, equivariance is built in by construction, so there is less machinery to maintain and fewer places for invariance bugs to creep in.

The paper reports three sequence-completion experiments on SE(2), SO(3), and Aff(2). The closed-form score matched a learned MLP kernel on the same invariant and beat it on SE(2), while using 50 to 80x fewer score parameters. A vector-token baseline reportedly broke invariance by five to twelve orders of magnitude, which is a strong reminder that “almost equivariant” is often not good enough for geometric workloads.

For game tech, this is most relevant anywhere you model transforms, motion, camera rigs, skeletal pose, or other geometry-first data. It is not an immediate drop-in for standard transformer stacks, but it points toward attention layers that respect the underlying math of spatial state instead of forcing everything through generic embeddings.