Written by Allan and edited by GPT-5.6 Sol.
It's odd to be confronted with a problem that you can’t solve for weeks. Odder still is to discover that you can remember such an experience in great detail nearly forty years later.
Sometime in the late 1980s, I was VP of Technology at ParcPlace Systems, the Xerox PARC spinout that commercialized Smalltalk-80 (which I called "the Greta Garbo of programming languages"). We had a nasty bug. I don’t remember the original report precisely anymore...whether it was a mouse event or some other asynchronous input that went missing. But the essential symptom was simple: occasionally, the system failed to notice something that had happened. Occasionally is the important word here.
The reports came in from the field and, as one does with reports of impossible, irreproducible events, engineering more or less ignored them. “Can’t reproduce” is not quite the same thing as “doesn’t exist,” but it serves adequately as a substitute. Then one day it happened to one of our own engineers.
Oh. Now we had a bug.
Unfortunately, knowing that a bug exists isn’t much help when you still can’t make it happen. Somebody devised a workaround. If the system occasionally missed a mouse-moved event, we could wake up periodically, look at the mouse position, and notice that it had moved anyway. Fine. Synthesize the missing event. Problem mitigated.
Except that we eventually realized we weren’t just losing mouse events. We could lose other asynchronous events too. Network activity, for example. At that point the workaround began turning into an increasingly elaborate system for checking whether reality had changed without bothering to tell us. This was not satisfactory.
I was head of Engineering, which meant, among other things, that I had people whose job it was to solve problems like this. They couldn’t. Eventually I did what any mature, experienced engineering executive would do: I essentially announced, “All right, you assholes can’t fix it. Give it to me.” A tactical error.
Having established that a Real Man could fix the bug, I had eliminated the possibility of not fixing it. After my reckless announcement, I spent most of an evening working towards the (surely easy) fix. Midnight comes...ruh-roh, maybe not so easy.
In the parlance of "The Godfather", I went to the mattresses. I told my wife that I wasn’t coming home until the war was over (if I remember correctly, she told me to come back with my shield or on it). For something like two weeks I lived with this bug.
The obvious place to look was our asynchronous code. Race conditions, locks, timing errors: this is exactly the sort of stuff human programmers are bad at. Simultaneity is hard to think about. It was perfectly reasonable to suspect that somewhere in this large system, built by a few dozen engineers, somebody had screwed up.
So I looked. And looked.
Eventually I got the failure into a form I could reproduce
reliably. The machinery involved the Unix select system call. In those
days a process had a relatively small, bounded collection of file
descriptors, conveniently representable as a bit-map. You handed
select a set of descriptors and asked the operating system to tell you
when something interesting happened to one of them.
If someone is using the system, something interesting would happen quite often. And, very occasionally, select wouldn’t say so.
Getting from “someone said every few weeks the mouse stops working for a bit” to reproducing that problem at will took rather a lot of work. I had to build a jig that generated events continuously and counted what went in against what came out. Once I could make the discrepancy happen under controlled conditions, I could keep narrowing the aperture: perhaps we’re mishandling this; perhaps we’re dropping that; perhaps there’s a race here.
One by one, those possibilities disappeared. This created another problem, this time in my head.
Every programmer learns fairly early that “there’s a bug in the compiler” is what freshmen say when their program doesn’t work. The compiler is fine. The operating system is fine. It's your program that's broken. Stop being an idiot and find your own stupid lame bug. This is a very good heuristic.
This time, the rule was wrong.
I kept shaking off the increasingly eerie evidence that the problem might be below us in the stack, because blaming Unix was precisely the kind of explanation that experienced programmers learn not to entertain. We had a complicated asynchronous system of our own. Of course the bug was ours.
Until, eventually, there was nowhere left for it to be.
Sherlock Holmes had a famous rule: when you’ve eliminated the impossible, whatever remains, however improbable, must be the truth. I had effectively done that. I had made the aperture smaller and smaller until the only thing left inside it was something I had been taught not to believe.
Fuck me. It was Unix.
Then came the challenging task of telling Sun Microsystems that there was a bug in their pride and joy, the SunOS operating system (their meticulously curated version of Unix). You can imagine how enthusiastically they greeted this news. Fortunately, by then I had reduced the problem to a reproducible case. They eventually agreed. It was their bug. They fixed it.
I haven't thought about this episode in years. I thought about it last night because I was trying to figure out how to make coding assistants better at debugging. If you didn't know, they're terrible at it. My friend Jeff says "AIs are great at writing programs but they're terrible programmers." My thought exactly.
What strikes me now isn’t that I eventually found the bug. Two weeks is not exactly a boast-worthy debugging time. It’s what the experience says about debugging.
I was using hypotheses about where bugs are likely to live. Asynchronous code? Look there early. The compiler? Look there very late. The operating system? Also very late. Those aren’t certainties; they’re priors. Good ones, usually. In this case one of my strongest priors was wrong, and it delayed me. But abandoning such priors would be insane. The fact that a heuristic sometimes loses doesn’t mean you should replace it with wandering around randomly.
And once the ordinary hypotheses had failed, I didn’t just keep searching more code. The nature of the problem changed. The question became: What explanation is still capable of producing the evidence I am seeing? Eventually that question forced me across a boundary I had been extremely reluctant to cross—from debugging our program to debugging the operating system underneath it.
Nearly forty years later, I’m bemused that I can reconstruct so much of this. I can’t tell you what I had for dinner Tuesday. But I can apparently still remember the semantics of select, the shape of a file-descriptor bitmap, and the dawning horror of realizing that the freshman’s favorite excuse was, on this particular occasion, actually true.
It seems that not going home for two weeks is a highly effective way to provoke stable long-term memory storage.