> It’s crystal clear that AI can be a complexity factory.
> I’ve been thinking a lot lately about what software would look like if we made keeping software understandable to humans a first-class design goal in the age of AI.
I agree, and I've been thinking similarly. But I don't think there's anything "new" about what understandable and well-factored code should look like. It's the same principles as ever.
A lot of agent-written code looks like what you'd get if you gave an enthusiastic human slightly too many stimulants and asked them to take the shortest path to reach the goal. Plausibly this is just a result of the LLMs not being "smart enough" to do any better, but I think there's also an incentives problem. How do you reward human-understandability in benchmarks and unsupervised training?
I think the worst part is that to the bullshit artists that usually float to the top of any org chart, AI generated docs, and implementation plans, and code is superficially plausible. It's hard to find any particular thing that is wrong with all of it - just a general vibe of verbosity. I am observing entire engineering departments drowning in AI generated RFCs and TDDs and thousand plus word Jira tickets and PR reviews and feedback to the PRs. All just Claude talking to itself via various meat-based secretaries.
Meanwhile, as far as leadership is concerned, they don't really understand what is being delivered they just know there's a lot of it. So they're happy - for now.
I think a lot of shops are going to have to go through a couple years of Github-style "why the fuck is our service always down?" before they put 2 and 2 together.
There are deterministically computable metrics for cognitive complexity and readability[0].
They’re not perfect by any means — and I suspect they’re already included in the RL process for coding evals, and have been for some time. I do think we’ll see ongoing improvement in this area though.
I don't think these comparisons between AI generated code and an idealised version of the code are productive. We have to compare with code that exists in the real world, so AI against human, and not only that, good AI generated code with good human written code, as well as AI slop with human slop.
I think if we did that we'd see that on average the code written by humans are less legible, parts of the intention will live forever in the mind of the developer at time of writing, and more prone to complexity build-up over time, simply because there wasn't time and incentive to go back and refactor code that works, apparently.
Now with AI sure you produce a lot more complexity, more than the human prompter could write by himself, but complexity can be managed with the same workflow that created it, by analysing and removing code paths, changing code architecture, replacing reimplementation with consolidated libs etc...
It's a matter of knowing how to use the tool and not creating a false sense of nostalgia where we feel like we had it better in the old days, which is not true at all.
> How do you reward human-understandability in benchmarks and unsupervised training?
And to answer this, there is no replacement for humans immersed in their world yet, so we need developers with with good understanding of the domain and that are able to write good descriptive prose in order to steer agents into producing acceptable code.
Set acceptable review standards, outline appropriate frameworking, document approaches, test standards, documentation standards and overall just set good examples in both context and the codebase.
If your codebase is slop it's because you approved it.
I’ve settled on a practice of having the agent write a lesson plan with quizzes for me, for any PR or feature of sufficient size to warrant it. It works pretty well and forces me into a back and forth with the model where I otherwise might be tempted to let it do whatever it wants.
> We’ve never needed to treat understanding our own code as a separate cost from writing it, because writing it largely forced us to understand it.
Once a system gets to a certain scale, understanding it has always been a problem; it's just exacerbated in the age of LLMs. On many occasions, I've even seen folks write code they didn't understand, in the hopes that it would solve issues we were experiencing. Sometimes this code shipped. AI has just laid it bare.
I've been working for many months on exactly this problem: a tool to codify and accelerate understanding. Help with prod incidents. Architect better. Get to systematic understanding faster to save businesses valuable time (read: money). This is only getting worse in the AI age, but it has always been a problem for software at scale, and I'm thoroughly enjoying solving problems that I and my teams have had for many years.
we will go full circle "what software would look like if we made keeping software understandable to humans" maybe we invent a pseudo programming language so we get deterministic results ;)
I honestly have a vision of a database of .md files that fully describes an application in human-readable, (ok, engineer-readable), text. Making a change is committing a change to the human-readable text, and then AI comes and figures out the rest and/or suggests edits to the human-readable text. (Or asks questions.)
if you spec fully describes a program, it is at least the length of the source code, and probably longer. It will also be just as difficult to read and understand
The vision is that AI owns the process end to end, from autonomous user interviews through spec generation all the way to implementation. We shouldn't need to tell AI what to do for a high level goal, it should figure it out on its own.
Interestingly enough, the question "do we want to support it" has been the main one (not "can we write it") in any sufficiently large and mature project for as long as I remember.
Now, becoming large is now a solved problem essentially, but that's still only maybe 10-20% of the software engineering work in the long run. I'm grateful I can delegate some grunt work to LLMs, but, essentially, the main reason why large projects slow down development has never been due to inability to write lots of code quickly.
Yes I'm more than okay. AI has enabled a scale of personal ambition I could only previously have dreamed of.
I've never been the kind of coder who could easily sit down and get their thoughts out from mind to written lines. I've seen that happen in a few gifted individuals, and AI might be frustrating them, because for them, coding was never the bottleneck, but for me, I would always get stuck in analysis paralysis, and just writing the first line of a project was a daunting prospect.
With AI I'm able to construct an entire ecosystem of programs I've always wanted.
The code isn't perfect, but I understand enough to fix architectural mistakes and to guide the AI to a good enough solution.
It's true I've always been better at breaking things than making things, but for someone who was "never really a coder" I've had a good career doing it all the same!
What was the point to you? What I took away from the article is that AI adds a bunch of complexity that is making it harder, not easier to understand.
And I'm saying that is true, but it's also enabling a kind of complexity and ambition in outcomes that might be a worthwhile trade-off.
In the same way we accept that cars are no longer something that a mechanic can understand and repair without a laptop, because we trade off the repairability and understanding for better mileage or safer handling.
Either I misunderstood the article, or people are misunderstanding my point, because I'm genuinely confused by being called a bot.
It's useful to you. I'd say take the win and let the haters quibble. I'm good enough at coding that I can use it as part of my day job, but I hate coding for the sake of coding. For me coding is a tool for solving problems, and now AI can sometimes help me solve problems I might not have had the motivation to persue. My scale is usually "a small application that helps me do X", not "I develop photoshop singlehandedly".
The balance sheets of the AI providers are pretty clear on this ownership thing: I shouldn't own it, NVidia should. Instead, I should pay at least once a very large amount to initially produce it. And then I should have a recurring usage-based monthly fee - for a license allowing me to say it's "my code". That's on top of hosting fees, etc.
I'm not OK with it, but I have no choice besides maybe quitting my job.
I've also tried to understand the code that AI writes, but it's often insane and untangling it would slow me down so much that it would nullify the gains in speed that AI brings.
To me, either companies realize they're spending a lot more money to ship crappier code and AI becomes a niche, or we'll just stop looking at code. I don't think there's any other option because I don't see AI getting better at it. It seems to actually be getting worse and more annoying to use.
My company recently had to cut back our usage to effectively nothing and I get to code by hand again. I'm going to appreciate it while I can. The AI just wasn't providing enough value for us
> We’ve never needed to treat understanding our own code as a separate cost from writing it, because writing it largely forced us to understand it.
Completely agree with it.
> I’ve been thinking a lot lately about what software would look like if we made keeping software understandable to humans a first-class design goal in the age of AI.
It is already understandable if you are producing only required and necessary features. Understandability can be managed by scope control and not getting carried away with being able to produce more and more features.
I believe engineering teams will be working on trying to optimize for more engineering accountability in coming years, as the reality is catching up.
My tiny two pet projects I keep developing with LLMs taught me how useful TDD is - I'm not a software engineer so I've been never formally introduced to these, but now I learn the best practices.
So in consequence I have less bugs in this code than what I could achieve myself, and as a second outcome of these is a solid workspace templates that make it faster and easier to get another small thing up to speed, the way that's good and "me". And the very short leash, sandbox and hooks make sure I learn a lot in the process too.
It's yours if during the generation process you correctly used the tool. e.g. You designed the architecture, you reviewed the generated code, you assisted all the development, you decided the type of tests and used your brain to guide the generation.
It's not your code if you just said "implement this" and then you shipped it. It's going to backfire - and if you still think that you own that code... well you're wrong. You're a code custodian rather than a code owner.
I don't believe that AI is the 'higher level of abstraction[1]' that it is often claimed to be. However I wonder if it's possible to work with a codebase when other developers are treating AI as a higher level of abstraction.
Analogously, people can write in assembly, but can you write in assembly when the codebase is frequently altered by someone with an optimizing compiler?
[1] - Do higher levels of abstractions even exist? By my way of thinking that's called not an abstraction. C isn't a higher level of abstraction; it's a different abstraction implemented by assembly. There are more powerful abstractions and more ergonomic abstractions, but that doesn't make them 'higher level'.
You wouldn't be the first person to eventually conclude that C is just a very fancy macro assembler.
But AI generated code is much like every other generated code: it is more voluminous and rather less elegant than what you'd write as a person. And here too there is a link with assembly: early compilers routinely put out absolutely terrible assembly by assembly programmers' standards of the day. And then they got better. To the point that today it takes some pretty rare niches that it pays off to break out the assembler and roll your own.
I expect AI generated code to develop in the same way, and the way to get them to improve is to hold generated code to high standards. The problem is the perverse incentive: generated code of low quality will eventually result in many more tokens burned than a one-shot piece of perfect and elegant code. Here you can draw a link with medicine...
> Analogously, people can write in assembly, but can you write in assembly when the codebase is frequently altered by someone with an optimizing compiler?
Yes. There's an interface boundary in the form of function prototypes and ABI rules. To the optimizing compiler, your assembly code is undistinguishable from separately compiled high-level code, and to your assembly code, the compiler output is undistinguishable from separately assembled low-level code.
One problem with LLMs, is that they don't respect these boundaries like an optimizing compiler would.
> can you write in assembly when the codebase is frequently altered by someone with an optimizing compiler
This itself is quite normal (e.g. we have .S files and .c files and link them together), but I suppose a more accurate analogy would be that someone is dumping their optimising compiler output directly into your .S files.
Maybe what we need is better ways to partition the slop-code and the not-slop code.
Sounds like you just don't like the phrase 'higher level'. To most people this is synonymous to 'powerful abstractions' and 'more ergonomic abstractions' in the sense that they allow us to think "at a higher level" - map instead of loop, objects instead of memory addresses and size, modules instead of gotos.
I find it incredibly difficult to contribute to code files that were initiated by LLMs. Need to dig deeper to understand why but I just don't like it. I end up prompting my way out of any issues I see, sometimes it works out sometimes it doesn't.
> I’ve been thinking a lot lately about what software would look like if we made keeping software understandable to humans a first-class design goal in the age of AI.
Why? Humans can't keep up. We can't keep up with writing the code, we can't keep up with debugging, and I think we're approaching a 'claude code' moment where we won't be able to keep up with system architecture.
This is why I don't use AI to write code directly but instead will use it to generate code snippets or ideas or critique, or even just use it as documentation I can talk to.
By doing this I can still get the benefits of the technology including - and this is crucial - as a learning aid to improve my own skills, while not entirely outsourcing my own thinking to the machine.
This is exactly how AI should be used when coding. "Agentic" AI produces garbage code. The entire codebase should never be accessible to the context (even if parts at a time, if the agent chooses the parts). And the entire codebase should never be write accessible. Only limited scope writes, human read, and approved.
> It’s crystal clear that AI can be a complexity factory.
> keeping software understandable to humans a first-class design goal in the age of AI.
I'm not sure this is going to be a worthwhile goal for much longer. Rarely is anyone caring about the complexity and comprehensibility of the ASM that GCC or LLVM generates from higher-level C code. I think the code-generation abstraction is just moving even higher now into "prompts". The architecture and engineering process of developing software systems is the important human component. The source code itself is not super valuable. What remains valuable is the input, direction, scrutiny, and "battle testing" of the solution.
"The system should email the customer when their subscription renews."
When? Which timezone? What should we email them? Which email address (the customer changed their email 10 minutes ago, the address at time of the charge?)
I can look at a page of code and answer those questions pretty quickly.
You could write all of those in plain English, but then there are more tiny flaws in the ambiguous language. Maybe we could standardize the language in some way. To reduce ambiguity we start using keywords. When we want to refer to the same thing twice we create a variable or a type. Two things must interleave, let's create concurrency primitives. To check that the prompt is correct, we can write a test for it.
And maybe we DO end up with a different format of prompting that supplies this, but at some point we are still giving directions to computers. If those directions are lossy, the system runs differently every time (which I've noticed is bad for building good software).
A big difference with libraries is that (hopefully) a whole lot of other people have used that same library the same way you are going to use it for the same purpose you are using it for. All the significant bugs hopefully were found by them and the library was fixed.
That usually makes it safe to use even if you have no idea how it does things inside.
With AI code there is a very good chance you are the first person to ever use that code.
I've inherited plenty of code bases. AI generated code is a special kind of foobar. With a code base inherited from a human you can at least be assured at least one human has understood each portion for a brief period. With agentic slop all bets are off.
Maybe I'm a bad programmer or have worked at places with terrible dev culture, but honestly I've made changes to code I didn't really understand. The changes I made seemed to accomplish the task I was aiming at, but the downstream effects were sometimes impossible for me to grok. Before vibe coding took over, using AI to help me get a handle on what a bunch of code was doing was actually one of the things that sold me on using AI.
I keep getting downvoted for saying this but I still review all of my AI generated code. I feel like I’m sufficiently sped up.
Of course you can go faster if you don’t read the code at all, that’s always been true. But it seems like the ability to emit large quantities of text is already a huge leap in productivity. I don’t think these tools are good enough yet to offload all or our thinking into them.
IMO human-verifying will always be a must, no matter how good the LLMs get. If I'm shipping code that I don't understand and it ends up having major problems, I'm still the one who's responsible for it, not the LLM.
You could ship code without understanding it, it just needs to be treated as throwaway, unreliable and potentially malicious. This is OK for some tasks, e.g. best-effort scraping of a frequently updated website that's actively hostile to automation
Where it really shines for me is when the best solution is explicit and obvious code, but really tedious to write. Now I'll choose to have AI write it instead of me, where before I might have tried to write a bad abstraction. Things like generating a TypedDict for a JSON A document I'm returning as a convenience from a postgres query.
I also review all of my AI-generated code. I feel that it’s disrespectful to my teammates to send unverified slop for review.
I’ve noticed that while my coding skills have very noticeably atrophied (to the point I am consciously scheduling time to manually code, just for practice), my ability to read and comprehend code quickly has actually improved quite dramatically.
I notice style inconsistencies and semantic errors far more quickly and consistently when reviewing both human- and LLM-generated code, as compared to a year ago. Cognitive tradeoff hypothesis and all that…
Absolutely! I haven't had this much fun since I was coding MySpace layout glitter graphic websites in 2006 with PHP, MySQL, and CodeIgniter.
I'm getting older, I have lots of ideas, and now I have tools that can help bring them together faster instead of spending two weeks obsessing about how I want to set up Ubuntu and have nothing to show for it. Helps replace the all-nighters that I used to be able to do.
I still think having decades of experience in everything from domains, DNS, FreeBSD/Linux servers, databases, all sorts of programming languages, and full stack dev helps get more out of these tools compared to somebody that has no idea what they're doing. Especially around the deployment side right now. People keep saying that's changing though.
Before the age of slop we are living in, there were brilliant people who designed, implemented, tested and debugged some very brilliant and enormous codebase that the average HN user would take years to understand at a bird's eye view.
So no, having access to the new fancy model does not make the average person a genius in the same way that having access to Stephen King's typewriter does not make anyone a writer of success.
> I’ve been thinking a lot lately about what software would look like if we made keeping software understandable to humans a first-class design goal in the age of AI.
I agree, and I've been thinking similarly. But I don't think there's anything "new" about what understandable and well-factored code should look like. It's the same principles as ever.
A lot of agent-written code looks like what you'd get if you gave an enthusiastic human slightly too many stimulants and asked them to take the shortest path to reach the goal. Plausibly this is just a result of the LLMs not being "smart enough" to do any better, but I think there's also an incentives problem. How do you reward human-understandability in benchmarks and unsupervised training?
Meanwhile, as far as leadership is concerned, they don't really understand what is being delivered they just know there's a lot of it. So they're happy - for now.
I think a lot of shops are going to have to go through a couple years of Github-style "why the fuck is our service always down?" before they put 2 and 2 together.
They’re not perfect by any means — and I suspect they’re already included in the RL process for coding evals, and have been for some time. I do think we’ll see ongoing improvement in this area though.
[0] (pdf warning) https://www.sonarsource.com/docs/CognitiveComplexity.pdf
I think if we did that we'd see that on average the code written by humans are less legible, parts of the intention will live forever in the mind of the developer at time of writing, and more prone to complexity build-up over time, simply because there wasn't time and incentive to go back and refactor code that works, apparently.
Now with AI sure you produce a lot more complexity, more than the human prompter could write by himself, but complexity can be managed with the same workflow that created it, by analysing and removing code paths, changing code architecture, replacing reimplementation with consolidated libs etc...
It's a matter of knowing how to use the tool and not creating a false sense of nostalgia where we feel like we had it better in the old days, which is not true at all.
> How do you reward human-understandability in benchmarks and unsupervised training?
And to answer this, there is no replacement for humans immersed in their world yet, so we need developers with with good understanding of the domain and that are able to write good descriptive prose in order to steer agents into producing acceptable code.
Set acceptable review standards, outline appropriate frameworking, document approaches, test standards, documentation standards and overall just set good examples in both context and the codebase.
If your codebase is slop it's because you approved it.
I published a skill and short article on the technique I settled on: https://condour.github.io/lesson-plan-quiz/
No, I specifically don’t. My company owns it, and that bastard will lay me off without a second thought.
Once a system gets to a certain scale, understanding it has always been a problem; it's just exacerbated in the age of LLMs. On many occasions, I've even seen folks write code they didn't understand, in the hopes that it would solve issues we were experiencing. Sometimes this code shipped. AI has just laid it bare.
I've been working for many months on exactly this problem: a tool to codify and accelerate understanding. Help with prod incidents. Architect better. Get to systematic understanding faster to save businesses valuable time (read: money). This is only getting worse in the AI age, but it has always been a problem for software at scale, and I'm thoroughly enjoying solving problems that I and my teams have had for many years.
The vision is that AI owns the process end to end, from autonomous user interviews through spec generation all the way to implementation. We shouldn't need to tell AI what to do for a high level goal, it should figure it out on its own.
Now, becoming large is now a solved problem essentially, but that's still only maybe 10-20% of the software engineering work in the long run. I'm grateful I can delegate some grunt work to LLMs, but, essentially, the main reason why large projects slow down development has never been due to inability to write lots of code quickly.
I've never been the kind of coder who could easily sit down and get their thoughts out from mind to written lines. I've seen that happen in a few gifted individuals, and AI might be frustrating them, because for them, coding was never the bottleneck, but for me, I would always get stuck in analysis paralysis, and just writing the first line of a project was a daunting prospect.
With AI I'm able to construct an entire ecosystem of programs I've always wanted.
The code isn't perfect, but I understand enough to fix architectural mistakes and to guide the AI to a good enough solution.
Looks like you were never really a coder, to be honest.
Nth iteration of pointless AI navel gazing?
And I'm saying that is true, but it's also enabling a kind of complexity and ambition in outcomes that might be a worthwhile trade-off.
In the same way we accept that cars are no longer something that a mechanic can understand and repair without a laptop, because we trade off the repairability and understanding for better mileage or safer handling.
Either I misunderstood the article, or people are misunderstanding my point, because I'm genuinely confused by being called a bot.
I've also tried to understand the code that AI writes, but it's often insane and untangling it would slow me down so much that it would nullify the gains in speed that AI brings.
To me, either companies realize they're spending a lot more money to ship crappier code and AI becomes a niche, or we'll just stop looking at code. I don't think there's any other option because I don't see AI getting better at it. It seems to actually be getting worse and more annoying to use.
Completely agree with it.
> I’ve been thinking a lot lately about what software would look like if we made keeping software understandable to humans a first-class design goal in the age of AI.
It is already understandable if you are producing only required and necessary features. Understandability can be managed by scope control and not getting carried away with being able to produce more and more features.
I believe engineering teams will be working on trying to optimize for more engineering accountability in coming years, as the reality is catching up.
So in consequence I have less bugs in this code than what I could achieve myself, and as a second outcome of these is a solid workspace templates that make it faster and easier to get another small thing up to speed, the way that's good and "me". And the very short leash, sandbox and hooks make sure I learn a lot in the process too.
It's not your code if you just said "implement this" and then you shipped it. It's going to backfire - and if you still think that you own that code... well you're wrong. You're a code custodian rather than a code owner.
Analogously, people can write in assembly, but can you write in assembly when the codebase is frequently altered by someone with an optimizing compiler?
[1] - Do higher levels of abstractions even exist? By my way of thinking that's called not an abstraction. C isn't a higher level of abstraction; it's a different abstraction implemented by assembly. There are more powerful abstractions and more ergonomic abstractions, but that doesn't make them 'higher level'.
But AI generated code is much like every other generated code: it is more voluminous and rather less elegant than what you'd write as a person. And here too there is a link with assembly: early compilers routinely put out absolutely terrible assembly by assembly programmers' standards of the day. And then they got better. To the point that today it takes some pretty rare niches that it pays off to break out the assembler and roll your own.
I expect AI generated code to develop in the same way, and the way to get them to improve is to hold generated code to high standards. The problem is the perverse incentive: generated code of low quality will eventually result in many more tokens burned than a one-shot piece of perfect and elegant code. Here you can draw a link with medicine...
Yes. There's an interface boundary in the form of function prototypes and ABI rules. To the optimizing compiler, your assembly code is undistinguishable from separately compiled high-level code, and to your assembly code, the compiler output is undistinguishable from separately assembled low-level code.
One problem with LLMs, is that they don't respect these boundaries like an optimizing compiler would.
This itself is quite normal (e.g. we have .S files and .c files and link them together), but I suppose a more accurate analogy would be that someone is dumping their optimising compiler output directly into your .S files.
Maybe what we need is better ways to partition the slop-code and the not-slop code.
Why? Humans can't keep up. We can't keep up with writing the code, we can't keep up with debugging, and I think we're approaching a 'claude code' moment where we won't be able to keep up with system architecture.
By doing this I can still get the benefits of the technology including - and this is crucial - as a learning aid to improve my own skills, while not entirely outsourcing my own thinking to the machine.
> keeping software understandable to humans a first-class design goal in the age of AI.
I'm not sure this is going to be a worthwhile goal for much longer. Rarely is anyone caring about the complexity and comprehensibility of the ASM that GCC or LLVM generates from higher-level C code. I think the code-generation abstraction is just moving even higher now into "prompts". The architecture and engineering process of developing software systems is the important human component. The source code itself is not super valuable. What remains valuable is the input, direction, scrutiny, and "battle testing" of the solution.
"The system should email the customer when their subscription renews."
When? Which timezone? What should we email them? Which email address (the customer changed their email 10 minutes ago, the address at time of the charge?)
I can look at a page of code and answer those questions pretty quickly.
You could write all of those in plain English, but then there are more tiny flaws in the ambiguous language. Maybe we could standardize the language in some way. To reduce ambiguity we start using keywords. When we want to refer to the same thing twice we create a variable or a type. Two things must interleave, let's create concurrency primitives. To check that the prompt is correct, we can write a test for it.
And maybe we DO end up with a different format of prompting that supplies this, but at some point we are still giving directions to computers. If those directions are lossy, the system runs differently every time (which I've noticed is bad for building good software).
For the AI Act, it is the case for images and videos, don't know about code.
Also relevant: including libraries written by others in your own software.
As always it comes down to finding a healthy balance.
That usually makes it safe to use even if you have no idea how it does things inside.
With AI code there is a very good chance you are the first person to ever use that code.
Of course you can go faster if you don’t read the code at all, that’s always been true. But it seems like the ability to emit large quantities of text is already a huge leap in productivity. I don’t think these tools are good enough yet to offload all or our thinking into them.
I’ve noticed that while my coding skills have very noticeably atrophied (to the point I am consciously scheduling time to manually code, just for practice), my ability to read and comprehend code quickly has actually improved quite dramatically.
I notice style inconsistencies and semantic errors far more quickly and consistently when reviewing both human- and LLM-generated code, as compared to a year ago. Cognitive tradeoff hypothesis and all that…
I'm getting older, I have lots of ideas, and now I have tools that can help bring them together faster instead of spending two weeks obsessing about how I want to set up Ubuntu and have nothing to show for it. Helps replace the all-nighters that I used to be able to do.
I still think having decades of experience in everything from domains, DNS, FreeBSD/Linux servers, databases, all sorts of programming languages, and full stack dev helps get more out of these tools compared to somebody that has no idea what they're doing. Especially around the deployment side right now. People keep saying that's changing though.
So no, having access to the new fancy model does not make the average person a genius in the same way that having access to Stephen King's typewriter does not make anyone a writer of success.