LLMs have no theory of the program.

Peter Naur talks about programming as theory-building. You develop intuition about a system over time, leading to an unspoken understanding of what the thing does. A program is rarely ever completed; the small changes, bugs, and adjustments that pile up of the months and years demand maintenance or to be wound down. (I keep coming back to The Maintenance Race.)

What characterizes intellectual activity, over and beyond activity that is merely intelligent, is the person’s building and having a theory, where theory is understood as the knowledge a person must have in order not only to do certain things intelligently but also to explain them, to answer queries about them, to argue about them, and so forth. A person who has a theory is prepared to enter into such activities; while building the theory the person is trying to get it.

You see this lack of a coherent theory frequently whenever you correct the model and it agrees with you and criticizes its earlier actions.

It can know which abstractions are present, but not which are the most appropriate; another way of saying its solutions are often inelegant.

But what about when humans program? Most code is sloppy, barely working, and riddled with vulnerabilities. Flaws are flaws, right? Is an uncaring programmer worse than a sycophantic agent?