Higher Order Attention
1-10-2026
Can Higher Order Attention Help Small Language Models?
Independent AI Research | PyTorch · Transformers · Experimental Design · BabyLM 2026
Peer reviewed and accepted to the BabyLM 2026 Workshop at EMNLP 2026. Not published in the proceedings.
Overview
Can small language models benefit from richer attention mechanisms than conventional query key dot products?
I investigated this question by designing and implementing Force Attention, a low rank quadratic correction to standard Transformer attention. The modification introduces higher order interactions between learned token features while adding relatively few parameters to a GPT style architecture.
The research was conducted for the BabyLM 2026 Strict Small Challenge, using a constrained 10 million word training corpus.
Research and implementation
I developed a custom attention mechanism in PyTorch and integrated it into a 12 layer causal Transformer, replacing a single middle attention layer.
To evaluate its effectiveness, I designed controlled experiments comparing Force Attention with standard GPT baselines across five model widths and six paired random seeds per width.
The experiments examined three questions:
Model quality: Does higher order attention improve validation loss compared with an otherwise matched Transformer? Learning efficiency: Does the modification improve learning dynamics under limited training data? Computational efficiency: What are the additional parameter and runtime costs, and can these be reduced?
I also investigated a Flash Attention compatible implementation to reduce the computational overhead of the new mechanism.
Key findings
The results show small, model width dependent improvements rather than a universal advantage over standard attention.
- 6 head models: The largest mean validation loss improvement, outperforming the matched GPT baseline in five of six paired runs.
- 7 head models: The most consistent improvement, outperforming the baseline in all six paired runs.
- Computational efficiency: A preliminary Flash compatible implementation reduced mean training time overhead from approximately 9% to 2.1%, while maintaining comparable validation performance.
- Official BabyLM evaluation: The released 8 head model achieved an NLP Average of 50.14 and an Overall Average of 39.04.
The findings suggest that higher order attention can introduce a useful inductive bias in certain capacity constrained settings, although the effects are modest and do not consistently translate into downstream improvements.
Research outcomes
This project produced a peer reviewed accepted research manuscript, a trained language model released on Hugging Face, a custom PyTorch attention implementation, and a series of controlled experiments examining architecture, learning dynamics and implementation efficiency.
The work strengthened my interest in the intersection of model architecture, experimental methodology and efficient AI systems.
Resources
#transformers #LLM #PyTorch #BabyLM_2026