synthesized from several pieces of Allan's writings & conversations by ChatGPT-5.6 Sol
Long-term, I do not believe in deterministic code at large scale, verified or not.
That is a more radical claim than saying software is difficult to write, expensive to maintain, or full of bugs. Everybody already knows those things. We have built an industry around knowing them. We have methodologies for gathering requirements, languages for expressing designs, tests for detecting errors, type systems for preventing them, formal methods for proving their absence, and project-management rituals for explaining why none of this caused the project to arrive on time.
AI can improve every part of that process. It can interview users, write specifications, generate code, write tests, discover vulnerabilities, repair bugs, update documentation, and port old systems to new platforms. This will be enormously valuable. It may remove much of the labor from software development.
But suppose the problem is not primarily the labor.
Suppose the problem is software.
The Old Bargain#
Software is a bargain we made with determinism. We describe in advance what a machine should do, translate the description into exact instructions, and accept that the machine will follow those instructions with a consistency no person could match. The arrangement transformed civilization. A program is fast, tireless, cheap to reproduce, and admirably indifferent to drudgery. It will calculate another payroll deduction, route another packet, or adjust another valve without losing concentration or asking for a vacation.
The limitation is hidden in the virtue. A program does exactly what we told it to do. Unfortunately, we did not know exactly what we wanted when we told it.
We usually describe the defects of software as failures of implementation: the program does not conform to its specification. But even a program that conforms perfectly to its specification remains brittle in several more fundamental ways.
First, we cannot fully state what we want. Natural language is ambiguous, but formal language is incomplete in a different way: it is exact about the things we thought to include. A perfect specification of the wrong problem produces a perfectly wrong system.
Second, we often do not know what we want until we encounter a proposed solution. Needs are partly tacit. A person can reject a room as uncomfortable without being able to supply a thermodynamic definition of comfort. A manager can recognize that a workflow is absurd only after watching employees route themselves around it.
Third, what we want changes. Laws change. Markets change. Organizations change. People change their minds. A requirement is not a fact about the universe; it is a temporary settlement among interested parties.
Fourth, the environment changes. Libraries, protocols, data sources, sensors, machines, adversaries, and surrounding institutions move underneath the program. Even performance is part of an interface, whether the interface admits it or not. A function that still returns the correct answer but now takes an hour may be as broken as one that returns nonsense.
Fifth, knowledge progresses. A program embodies what its designers believed when they wrote it. A flight controller contains a model of aerodynamics. A financial system contains assumptions about populations, behavior, risk, and regulation. A medical system contains yesterday's medicine. Determinism preserves ignorance as faithfully as it preserves knowledge.
This is why software rots without decaying. The bits may be exactly where we left them. It is the world around the bits that has escaped.
Formal verification does not solve this problem. It can establish that a program satisfies a formal specification. That is an impressive and important achievement where the specification is stable, the environment can be bounded, and failure is intolerable. It does not establish that the specification captures the real need, that the need will remain the same, or that the model of the world embedded in the specification will survive contact with tomorrow.
Perfectly verified code still locks in yesterday.
Why Did We Want Programs in the First Place?#
It helps to ask an embarrassingly elementary question: why do we use software at all?
Why do we not calculate payroll deductions in our heads before writing checks? Why not solve equations by hand to predict the weather? Why not sit beside the furnace switch and turn it on whenever the room feels cold? Why not manipulate every control surface of a dynamically unstable aircraft directly from the pilot's joystick?
Because human beings are slow. We make errors. Our reaction time is poor. We become distracted and fatigued. Repetition destroys our motivation. We cannot remember every tax rule, monitor every sensor, or make a thousand coordinated adjustments in a millisecond.
But imagine a worker with the judgment and flexibility of a good human, access to more knowledge than any human could retain, reflexes measured in microseconds, and complete immunity to fatigue and boredom. Such a worker would be better than a conventional program in nearly every setting where the environment changes and judgment matters.
That worker is what AI is becoming.
The economic reason for encoding behavior in advance begins to collapse once judgment itself becomes fast, tireless, reproducible, and cheap. We wrote programs because intelligence was the scarce and expensive component. If intelligence becomes abundant, why continue to freeze a pale imitation of it into code?
This is the Bitter Lesson taken all the way to the bottom. The usual reading of the Bitter Lesson is that general methods using computation tend to defeat systems built from carefully handcrafted domain knowledge. But handcrafted software is itself an immense body of frozen domain knowledge. Why should the lesson stop at the boundary of machine learning? Why use learned intelligence merely to write a larger collection of deterministic instructions?
Once competent AI exists economically at many scales, why write programs at all?
The AI Is the System#
This is not a proposal for AI-assisted software development. It is not “vibe coding” performed by better models, followed by better testing, producing more reliable binaries. It is not a just-in-time system that rewrites its source code every evening and redeploys itself every morning. Dynamic rewriting is still software maintenance, accelerated: another turn around the specify, freeze, patch, and redeploy treadmill.
The proposal is that the AI performs the job directly.
Instead of writing and verifying a complex control program for a blast furnace, give an AI the goal of maintaining output, minimizing fuel waste, and remaining inside non-negotiable safety limits. Let it respond minute by minute to ore quality, equipment condition, energy prices, and the evidence produced by its own earlier actions.
Instead of encoding every payroll rule and exception into a deterministic engine, give an AI the employment contracts, the current law, the accounts, and the authority to prepare payroll. Let it interpret the rules that apply now, identify ambiguity, ask for decisions when necessary, and adapt when the law changes.
Instead of a fixed “smart thermostat,” use an intelligence that observes weather, occupancy, prices, individual preferences, building behavior, and grid demand. Tell it to keep the occupants comfortable within limits on cost, safety, and equipment wear.
Instead of static disaster-response software, use an AI that continuously integrates sensor reports, damaged infrastructure, new priorities, and shifting resource constraints, then dispatches and redispatches accordingly.
These examples make some people understandably nervous. They should. An adaptive system with real authority can fail in ways a thermostat program cannot. But the comparison is not between a dangerous AI and an infallible program. It is between two ways of operating in a world that will not hold still. One relies on judgment at runtime. The other relies on the foresight of people who attempted to enumerate the future before deployment.
The old paradigm is compilation. The new one is execution.
Goals Are Not Theories#
The most important change may be in what we mean by a specification.
The prompt is currently treated as an undifferentiated blob. Into it we pour the task, facts about the environment, rules, preferences, examples, prohibitions, guesses about causality, stylistic requests, and whatever else seems useful. Then we hope the system will distinguish what is sacred from what is provisional.
An AI-native specification should make at least three things explicit:
- The environment: what the system can sense, what it can affect, and where its authority ends.
- The goals: the ends we actually care about.
- The theories: our current beliefs about how the environment works and how the goals might be achieved.
Consider the thermostat. “Keep the occupants comfortable within safety and budget constraints” is a goal. “Comfort means 21–23°C,” “the building's thermal inertia is approximately this,” “Alice prefers warmer rooms than Bob,” and “short-cycling damages the furnace” are theories. Some are excellent theories. Some may be encoded in law or engineering practice. None should be confused with the end itself.
A conventional program entangles the goal with the theories used by its authors. The control logic becomes a fossil record of their assumptions. An AI-native system can treat theories as hypotheses: select among them, tune them, test them against results, and replace them when they stop predicting the world. The goal may remain stable while the means evolve.
This is not an invitation to let a machine redefine its mission whenever the mission becomes inconvenient. Goals, authority, and forbidden actions may be guarded much more rigidly than the theories used in their service. Nor is the tripartite structure intended as a complete formalism. It is simply a way to expose distinctions that every working system already contains, badly and implicitly.
Theories should be replaceable. Goals should be contestable by the people entitled to set them. Authority should be explicit.
Trusting a System in Motion#
If behavior is no longer frozen into a static artifact, assurance cannot consist solely of proving properties of that artifact. The object of trust is now a live system acting in an environment.
This moves us from verification toward something more like operational certification. We do not prove that an aircraft can never fail. We certify it for service within an operating envelope, monitor it, inspect it, record what happened, train people to intervene, and revise the system when evidence changes. The warrant is not a timeless theorem. It is a structured, continuously renewed case for trust.
An AI-native executor therefore needs bounded authority, explicit tools, telemetry, runtime policies, circuit breakers, and safe stops. It should act in short, auditable cycles: observe, form a small plan, act through a permitted interface, measure the result, and continue, revise, escalate, or stop.
One possible architecture pairs a performer with an observer. The performer proposes and acts. The observer cannot act; it checks constraints, watches changes in key measures, detects theory drift, vetoes forbidden actions, and asks for human judgment when the evidence no longer supports continued operation. Other architectures will be better in other domains. The point is not to canonize an actor-critic diagram. The point is that monitoring and repair become part of the operating system rather than a ceremony performed before release.
Deterministic components do not disappear. They may be exactly what we want for interlocks, permission checks, transaction ceilings, cryptographic controls, and emergency shutdowns. The claim is not that every transistor must become stochastic. It is that fixed code ceases to be the locus of competence. The intelligence does the work; deterministic machinery defines parts of the envelope within which it may work.
Assurance follows the architecture. It should not swallow the idea.
The First Systems#
We should not begin with clever toys whose only purpose is to prove that an agent can call an API. We should begin where rigid logic is already bending under reality: industrial control, infrastructure operations, logistics under disturbance, financial administration under changing rules, incident response, and other environments with measurable outcomes and a credible deterministic baseline.
Run the AI in simulation. Put it in hardware-in-the-loop rigs. Let it shadow a production controller without authority. Inject sensor failures, changing conditions, missing inputs, and bad theories. Measure whether it adapts. Give it bounded authority and a fast rollback. Record the near-misses as carefully as the successes.
The persuasive artifact is not another position paper. It is a believable demonstration in which the adaptive system meets or beats the old stack on the metric that matters, under stresses the old stack handles poorly, with logs that let us understand what happened.
There will be failures. Some will show that present models are not capable enough. Some will show that the environment is not observable enough, the goal is incoherent, or the authority boundary is badly chosen. Those are useful results. The purpose of a new field is not to declare victory in advance. It is to discover where this new way of building systems works and what it requires.
After Software#
“Post-software” does not mean that code vanishes any more than the industrial age meant that hand tools vanished. It means that code is no longer where we expect the essential behavior of a system to reside.
Static programs are snapshots. The world is a stream.
For fifty years we compensated for that mismatch with patches, requirements processes, maintenance budgets, version numbers, and increasingly heroic efforts to prove that our snapshots were internally consistent. We will continue to need those skills for a long time. But we should stop assuming that the perfected snapshot is the destination.
Give the intelligence a goal. Tell it what world it inhabits. Distinguish what we want from what we merely believe. Bound its authority. Make its actions visible. Test it under conditions that matter. Give it a job and require it to do the job.
Do not ask the AI to write the program.
The AI is the program now.