Our foundational result (manuscript https://arxiv.org/abs/2405.15943, blog post https://www.alignmentforum.org/posts/gTZ2SxesbHckJ3CkF/transformers-represent-belief-state-geometry-in-their) showed that transformers trained on next-token prediction spontaneously organize their activations into geometries predicted by Bayesian belief updating over hidden states of a world model. Even when trained on simple token sequences from hidden Markov models, complex fractals emerge in the residual stream, structures far removed from the surface statistics of the training data.
We think of this work as providing the first steps into an understanding of what fundamentally we are training AI systems to do, and what representations we are implicitly training them to have.
Since then, we've pushed in several directions. In Constrained Belief Updating Explains Transformer Representations https://openreview.net/pdf?id=f6Hl60FBFU, we asked how attention implements belief updating when Bayesian inference is fundamentally recurrent. We found that attention parallelizes recurrence by decomposing belief updates spectrally across heads, and we were able to make verified predictions about embeddings, OV vectors, attention patterns, and residual stream geometry at different layers.
We've also developed a theory of in-context learning https://arxiv.org/abs/2505.18373 grounded in training data structure. When training data mixes multiple sources, models must infer not just what hidden state the generator is in, but which source is active. This hierarchical belief updating necessarily produces power-law loss scaling with context length and explains why induction heads emerge.
We've been asking what the most general computational framework for understanding neural network representations might be. Our initial work implied activations should lie in simplices, but we've now shown that networks discover quantum and post-quantum belief geometries https://arxiv.org/abs/2507.07432 when these are the minimal way to model their training data. This offers a new foundation for thinking about features, superposition, and what representations neural networks use on their own terms.
Most recently, we’ve shown that transformers naturally decompose https://arxiv.org/pdf/2602.02385 their world model into interpretable parts. These factored belief representations provide an exponential-dimensional advantage, and suggest that we can understand and surgically intervene upon low-dimensional subspaces of large models.
For a comprehensive overview of where we are and where we're headed, see our July 2025 progress report https://www.alignmentforum.org/posts/fhkurwqhjZopx8DKK/simplex-progress-report-july-2025 on the Alignment Forum. You can also watch Paul and Adam discuss the research program at the FAR Seminar https://www.youtube.com/... or read this recent interview https://asterainstitute.substack.com/p/understanding-intelligence-simplex from August 2025.