synthesized from several pieces of Allan's writings & conversations by ChatGPT-5.6 Sol
The Transformer is eating the world.
It began as a machine for processing sequences of words. Then images became patches, audio became chunks, video became frames, and everything became a token. Different modalities enter through different doors, perhaps after some specialized preprocessing, but once inside they are arranged as vectors in a sequence and handed to attention.
This is technically astonishing and aesthetically funny. Decades of handcrafted machinery for vision, speech, language, and other domains are being compressed into variations on one computational appetite: tokenize it and let the model attend.
The success invites a dangerous historical conclusion. What scaled first begins to look like what had to exist.
Transformers were not waiting inside mathematics for us to discover the predestined architecture of intelligence. They were a powerful, unusually scalable design that met the hardware, data, and institutional conditions of a particular moment. Once the feedback loop began—better results, more investment, better tooling, more researchers, more data, larger models—it became rational for nearly everyone to follow the same road.
The road worked. That does not mean the surrounding continent is empty.
The Bitter Lesson, Properly Applied#
Rich Sutton's Bitter Lesson is commonly summarized as: general methods that exploit computation eventually outperform systems built around handcrafted human knowledge. The history of AI repeatedly humiliates our clever domain-specific machinery. Search and learning scale; curated rules reach a ceiling.
Transformers are a magnificent expression of that lesson. Convolution once looked like the proper architecture of vision because locality and translation invariance were obviously important. Attention, given enough scale, often learns what it needs without our building every intuition into the architecture. Similar stories can be told about language-specific pipelines and manually designed representations.
The crude reading of the Bitter Lesson is therefore: stop being clever and make the Transformer larger.
That reading turns a warning against architectural dogma into architectural dogma.
The lesson was not that one general method discovered in 2017 is the final vessel for computation. It was that methods able to learn from scale tend to defeat systems whose designers freeze their own knowledge into the machinery. There may be many learnable, scalable architectures. We have conducted an enormous experiment on one family and comparatively tiny experiments on most alternatives.
The bitter lesson should make us suspicious of cherished priors, including the prior that attention over token sequences is the natural form of every intelligent process.
The Bee in the Room#
A bee is agentic in a sense that is difficult to dismiss. It navigates, seeks resources, returns home, communicates, adapts to disturbance, and participates in collective behavior. It does this with a nervous system that would be an insultingly small parameter count by present foundation-model standards.
The usual moral is that current AI architectures are absurdly inefficient or missing essential structure. Perhaps we need compositional world models, causal programs, innate object concepts, explicit planning, or one of the other intellectual descendants of good old-fashioned AI.
Perhaps.
But there is another conclusion available. If we genuinely wanted to explore systems inspired by animal nervous systems, we have barely tried.
“Biologically inspired” often means importing a favorite high-level theory of cognition. Yet biology offers a vast collection of nearly trivial departures from the standard Transformer stack: recurrence, local memory, continuous feedback, sparse and changing connectivity, multiple timescales, sensory-motor coupling, energy constraints, developmental noise, neuromodulation, and global broadcast signals that alter the behavior of many local circuits at once.
None requires us to hand-code a symbolic theory of bee thought. They can remain weakly structured, learnable, differentiable, and hungry for data. The point is not to imitate a brain anatomically. Airplanes do not flap. The point is that evolution explored a design space with properties our batch-trained token predictors largely ignore.
We have been so busy feeding GPU coal into the Transformer furnace that many obvious variations remain academically unfashionable, infrastructurally inconvenient, or simply underfunded.
Tokens Are a Choice#
Tokenization is so deeply embedded in language modeling that it appears almost metaphysical. Text becomes tokens because models consume tokens. Models consume tokens because text became tokens.
Tokens are useful compression. They shorten sequences and package recurring structure. They also impose a segmentation chosen before the model begins learning. Languages are treated unevenly. Spelling, morphology, raw byte patterns, and formatting are filtered through a vocabulary whose units carry assumptions about what should matter.
What happens if we use bytes? Characters? Learned multi-scale units that emerge during training? What if the model can alter its own representational granularity depending on the task—fine detail here, coarse structure there?
The answer may be “training becomes too expensive and performance gets worse.” Excellent. A clean negative result is information. The point is not that bytes are secretly the key to AGI. The point is that the unit of meaning is a design decision worth reopening.
Vision makes the question visible. A small image patch preserves detail but creates a huge sequence. A large patch lowers the computational burden and discards structure. Hierarchical systems merge patches as representation deepens; multi-scale systems can combine local and global views. Why should linguistic or multimodal systems have one fixed granularity throughout inference?
Again, this does not demand a grand new theory. Change one thing. Measure what follows.
A Nervous System Has Moods#
Current models are steered mainly through content. We tell them in words to become cautious, creative, rigorous, terse, exploratory, or skeptical. The same computational machinery then tries to infer a global operating mode from a sentence inside the material it is processing.
Biological nervous systems possess low-dimensional global signals. Neuromodulators change how large populations of neurons respond. A creature can shift between exploration and exploitation, vigilance and rest, approach and avoidance. These are not detailed instructions specifying the next action. They are changes in the condition under which many local decisions are made.
What would the computational equivalent look like?
A model might have global modulators for caution, confidence, persistence, novelty-seeking, or computational effort. The settings would not be prose. They would alter routing, activation, memory access, or learning dynamics directly. Perhaps they could be learned and exposed as controls. Perhaps the model could regulate them internally under constraints.
This idea connects to the separation of control from text, but it is broader. A control plane says some signals should arrive outside the content channel. Neuromodulation suggests that those signals need not describe behavior symbolically; they may change the landscape in which behavior emerges.
Will this produce more natural agency? Nobody knows. That is why it is research.
Time Is Not a Context Window#
Transformers process a supplied context. Even when wrapped in an agent loop, their apparent continuity often comes from an external program repeatedly reconstructing the relevant past and asking the model what to do next.
Animals do not receive their lives as a freshly serialized transcript at each moment.
They have ongoing state, multiple memory systems, recurrent perception, bodily needs, and consequences that persist. They act partly to change what they will perceive next. Agency is not merely inference over observations; it is a closed loop between sensing and action through time.
We can approximate that loop with tools, memories, and prompts. The interesting question is whether it belongs more deeply in the architecture. Recurrent attention might revisit and revise internal state. Local memories might persist without being rendered back into text. Sensory-motor training might make action and perception co-develop rather than adding “tools” after pretraining. Energy or activation budgets might force decisions about what deserves computation.
Each possibility is modest compared with “build a brain.” Each breaks a convenient assumption of the current stack.
The research program should prefer the modest break.
One Heresy at a Time#
Alternative-architecture research has a marketing problem. A proposal arrives promising memory, agency, causal reasoning, continual learning, embodiment, efficiency, and perhaps consciousness. It requires a new model, training regime, hardware substrate, benchmark, and philosophy of mind. The budget would strain a sovereign wealth fund. When it fails, nothing has been learned because everything changed.
The Road Less Traveled should fund narrower heresies.
Dump tokens and use bytes. Add one global modulator. Change the recurrence. Alter the loss. Replace the batch-and-epoch training mill with a continuing stream. Introduce a persistent local memory. Train a small model through action rather than attaching tools afterward. Change one inherited assumption cleanly enough that we can attribute the result.
This is not timid research. Constraint makes radical ideas legible.
The ideal project should state:
- Which assumption in the dominant lineage it rejects.
- What minimal architectural or training change embodies the rejection.
- What outcome would show the change mattered.
- What matched baseline prevents the result from being explained by more compute, data, or parameters.
- What artifact lets others reproduce and extend the experiment.
A small model is not an embarrassment here. The purpose is to discover a lever, not to defeat the largest proprietary system on every benchmark. If the lever works, scaling can come later, probably with more money than a contrarian seed program possesses.
Why the Road Is Empty#
Researchers are not foolish for working on Transformers. Tooling exists. Compute kernels are optimized. Benchmarks expect the architecture. Reviewers understand the vocabulary. Students need publications. Companies need results. A marginal improvement on the dominant stack can be compared, shipped, and funded.
Path dependence is not a conspiracy. It is what rational decisions look like when accumulated.
That is exactly why a small research fund should not imitate the foundation laboratories. A few million dollars cannot make a dent in their race to scale. It can matter enormously to a small team pursuing an idea that does not fit the race.
The funder's comparative advantage is permission: permission to produce a strange result, to work on an unfashionable mechanism, to use a model small enough to understand, and to discover that an attractive idea fails without having to rename the failure a benchmark win.
The target is not “the next Transformer.” That phrase imports the winner-take-all story too early. The target is evidence that the design space remains open.
The Fork History Skipped#
Technological history looks inevitable from the winning branch. The alternatives disappear, the infrastructure reorganizes itself around success, and eventually the present seems to have been contained in the past.
But there were forks.
Some roads were tried before compute made them plausible. Some were abandoned because tooling was poor. Some were attached to overambitious theories and sank with them. Some produced results that could not compete at the scale then available. Some were never explored because the dominant path was improving too quickly to justify looking sideways.
Most alternatives will remain inferior. That is not a reason to stop. Exploration is valuable precisely because we do not know which neglected variation becomes decisive under new conditions.
Transformers are not inevitable. They are what scaled first.
Make one principled break. Do it cleanly. Let it fail honestly. And if it works, follow the fork.