Higher Order Attention 

15-08-2026

I investigated whether higher-order interactions in attention can improve small language models trained under severe data constraints. In a Transformer, the attention score is a scaled dot product of the query and key vectors. Viewing this as a first order representation, I added a second order term to the attention score. The term consists of the scaled dot product of UQ = qP_q * qR_q and UK = kP_k * kR_k.

The paper has been submitted to the BabyLM 2026 competition and workshop.

The HuggingFace model can be found here.

#artificial-intelligence #llm