I do have a hunch in that we might be able to utilize tiny LLMs to figure out parts that might possibly be brittle and combine them with traditional generation/mutation-based fuzzing to generate fuzz targets that are more likely to trigger an edge case.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
I love good programming languages, if that counts for anything, and I think there's a huge amount of room and value in "using an LLM to translate one programming language to another".
I also think your instincts are on a useful path. Perhaps using a very small and stupid LLM to generate plausible-sounding-but-probably-wrong programs might be a path to generating interesting test cases?
Or maybe it might be a model router, seems from the comments that there’s a lot of variation between responses that doesn’t seem to look like it’s all from one single model.
I like Lisp, I’ve used Common Lisp with a passion, but this doesn’t seem like a valid argument for Lisp.
Homoiconicity, as I understand, is that the code is structured data that is easy to programmatically modify, hence allowing Lisp macros. While some might disagree, I see Rust macros as the closest thing that demonstrates homoiconicity in mainstream Algol-based languages, as Rust macros modify the loosely structured token stream to produce new Rust code.
Eval, on the other hand, that’s more of a capability that comes from Lisp’s runtime, which used to be unique when Lisp was thriving, but not anymore — JS, Python, Ruby, all of the runtime-based languages have an eval function. The fact that they are not used as much is more of a security issue, not a capability issue, and I am not sure how having eval can be argued as Lisp being the language of agents.
> I see Rust macros as the closest thing that demonstrates homoiconicity in mainstream Algol-based languages
I don't know if you consider Elixir mainstream, but IMO their macro system is much closer to lisp's ideal.
Elixir is basically Lisp, but with better syntax, a modern ecosystem, and running on the Beam. Unlike languages like Rust, Elixir's conditionals and function definitions are just calls in the AST, even though the syntax looks mainstream and not like paren soup.
> Elixir is basically Lisp, but with better syntax
Which so helpfully removes many of the benefits of lisps :P I don't understand this argument at all, if it's not s-expressions nor quacks like a lisp, in what way is it lisp at all? Making it algol/C-like already makes it like the rest of the 99% of the languages, without any of the easy benefits of the neat and simple syntax of lisps.
Agreed. You can tell when someone “gets it” or not by whether they see sexprs as a problem to be overcome or a benefit to be embraced. Count me firmly in the benefit camp.
A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list.
Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generating code. In JavaScript you can use JSON to generate data, but JSON can not carry functions around.
I think this idea makes a lot sense. Instead of making the LLM generate JSON or XML, why not make it generate Lisp, which can carry both programs and data?
Well, then ad absurdum it would be easier to just output a bunch of 0s and 1s, right?
The same goes for "just a list of nested lists", sure it is easy to produce it, and for trivial examples it may actually be easy, but for more complex and realistic problems lack of "higher order" structure is a negative!
For something like JavaScript, you can just have a language-native AST object with a few helper functions and then can just call "addFunction" on it with proper arguments so that the API shape validates plenty of properties of your output.
Except you can express the higher order structure in s-expressions too, and using an AST usually means expressing it as a different syntax than the other parts of your program.
Or maybe I've just been sleeping on the power of Javascript AST.
And that's more of a library ergonomics question how nice it is. There are plenty of other languages with nice ASTs and `eval`, and that's the only ingredient you need.
I haven’t seen an LLM generate syntactically invalid code in any language in … a year? So I don’t know if “more likely to be syntactically valid” is a good enough selling point.
I take it you never run models locally then? Local models that can run on reasonable consumer hardware have gotten to the point of being very useful, but they still occasionally are tripped up with syntax errors. Especially if you quantize the model and KV.
I'm thinking that there should be some standard for Agent-LLM communications, and the simpler the better. Lisp might be such a simpler standard as opposed to JSON.
> A Lisp program that writes a Lisp program really just needs to produce a list of (nested lists) of tokens. A JavaScript program that writes a javascript program needs to generate a string that is syntactically valid JavaScript code. That is a much bigger task than just constructing a (nested) list.
> Because Lisp syntax is so much simpler than that of JavaScript etc. it is much easier to avoid errors when generating code. In JavaScript you can use JSON to generate data, but JSON can not carry functions around.
First of all, the LLM does not produce structured tokens, it produces (tokens of) text. It does not have a concept of nested or structured tokens. Which means that producing a Lisp program and a JavaScript program is basically the same difficulty, i.e. LLMs producing function foo () {} is about the same task as producing (defun foo () ()).
In fact, because most Lisps uses the same token ( and ) for almost all delimiters, the LLM in fact gets confused a lot more than Algol-family languages that uses various different tokens for different purposes. It generates thinking traces that are a few screens long while trying to count the closing parenthesis and the depth, something that I have not found to be the case for other languages, even with languages much more obscure than Lisp. (And no, it is not a training data issue, because the Lisp family as a whole is pretty well represented in the data set, due to Emacs Lisp.)
> I think this idea makes a lot sense. Instead of making the LLM generate JSON or XML, why not make it generate Lisp, which can carry both programs and data?
You do realize that all programming languages contains both programs and data, right? i.e. JSON is literally a subset of the JavaScript language, all JSON documents are valid JavaScript code, can be embedded in JavaScript programs, and so on. This isn't even a JavaScript-specific thing, almost all recent programming languages have data structure literals.
The thing that makes Lisp unique is that it can modify programs as data in the language easily, and why would that be a unique capability that would be beneficial for LLMs, when it can just sed/awk or tree-sitter-parse programs with more conventional languages and modify it as easily?
Yeah, unfortunately "LLMs are bad at counting" seems to apply to counting parentheses.
With modern tool calling I wonder if a better way to go about it is for the LLM to express the program changes as a function or otherwise use an editor that auto-balances parens. There's a lot of relatively simple tooling that makes it easier to write in a Lisp. The languages tend to lend themselves to being straightforward to check like that.
What's special about Lisp's repl is that it's perfectly possible to construct your entire program in the repl, testing each addition live as you write it. (Many Lisp-focused editors assume you'll want to do this, such as Emacs making it easy to run the interpreter on a single function in a file.) That tooling is lost if you just try to one-shot the file, and before 2026 the majority of LLMs originally just tried to one-shot every file.
But, just like a lot of early LLMs had huge problems with whitespace and numbers because the tokenization was taking efficiency shortcuts that made sense for text but absolutely wrecked code syntaxe, I wonder if the current optimizations are badly formatted for Lisp.
At the very least, using a varient like Clojure that also uses [] and {} in addition to () might help.
Interesting. Can someone quickly summarize their results? Are they saying that using languages with simpler syntax Lisp lead to LLM being able to generalize their knowledge better?
They tested them on formal languages of different power, and saw where they could generalize beyond the training data. Transformers failed to generalize pretty early on at stack machines/brace matching.
It was a good bit older of a paper though, if I remember it's somewhat expected from the pure feed forward nature of them and limited circuit depth, where LSTMs have some recurrence.
I believe the problem stems from ) and )) both being one token wide. I wonder if repeated parentheses being removed from the token vocabulary would improve accuracy for Lisp code generation.
> JSON is literally a subset of the JavaScript language
Isn't it more like when you write JSON in JavaScript you write it as a String?
JavaScript has the data-type "String" but does it also have a built-in data-type "JSON"?
You can pass such a string containing JSON to (built-in) library function JSON.parse() and you can take a data-structure written in JavaScript can call JSON.stringify() on it. But if your JSON is malformed, does the JS-interpreter tell you that, before you call the library-function JUSON.parse() ?
If you have a JavaScript parser/interpreter, does it parse JSON too? If not I would say JSON is not part of the JavaScript language, even if JS runtime provides a library function for parsing it.
> Eval, on the other hand, that’s more of a capability that comes from Lisp’s runtime, which used to be unique when Lisp was thriving, but not anymore — JS, Python, Ruby, all of the runtime-based languages have an eval function.
And it's good we have those for troubleshooting but those eval still offer nowhere near the power of a Lisp REPL.
> and I am not sure how having eval can be argued as Lisp being the language of agents
I've seen several Lisp programmers saying that it's really the REPL (and the 'E' in REPL is for "Eval") that is the godsend when working with LLMs.
With LLMs we've seen terminals/ssh/tmux and CLI tools calling making a huge comeback (not that they ever went very far).
Now I wouldn't be surprised if we were soon to see a Lisp AI harness also using extensively the REPL becoming succesful.
It's too early to tell it's not a powerful combo (LLMs + Lisp REPL).
How is that different from other languages that have proper repls? Lisps don't really give you much here, if the language can parse itself and can eval it, you have the sufficient primitives.
What helps is how "small" the hot reloaded parts can get - lisps are good here due to a lot of functions and not much shared state. But that is again, not something specific to lisps, neither are they the best at it - there are far purer FP languages for example
> Homoiconicity, as I understand, is that the code is structured data that is easy to programmatically modify
I think a more accurate description is that lisp code is just cons cells and cons cells is both how we write the code and how the runtime itself implements lists. So there’s basically no distinction between a lisp list and the text representation of the source. Rust and its macros is a different situation because the text representation of the code and the syntax tree have completely different shapes
> I think a more accurate description is that lisp code is just cons cells and cons cells is both how we write the code and how the runtime itself implements lists. So there’s basically no distinction between a lisp list and the text representation of the source.
Heresy I suppose, but doesn't feel (to me) like it "specifically" has to be cons cells, as long as it's a "list" of some sort, regardless of the specific type of list, while I'd say "cons cells" is a specific implementation of a list, famously implemented by Common Lisp and similar lisps.
Besides that I agree with you, the code you write and the code the compiler uses is the same, that's why it gets easy to edit with code itself. Rust and similar don't have this feature at all, it's very different experience writing macros with lots of special syntax, compared to just writing "normal" code doing the same sort of operations. "Doing a loop of X" isn't the same inside of a normal runtime function body, as when you want to do a loop of arguments for the function signature in a macro, as just one simple example.
Not to mention in Rust you have two different types of macros, declarative vs procedural macros, already speaking to that the entire concept of homoiconicity isn't there. You want it to be easy to implement a read, evaluate, print and loop back workflow by passing native data through the entire chain, except parsing the initial user-input string into your AST (whatever shape that ends up being), which pretty-printed (basically) looks the same as the user-input.
But this is just not true. An utf8 string is most definitely not a bunch of cons cells. Having a one-to-one correspondence is not the same thing, and one might argue that a rust code snippet and the corresponding AST object is just as arbitrarily far apart with a clean one-to-one correspondence.
So like, what's the material difference between the two? You have a parse and an eval in both cases
So this is the change that forced Zig to remove @cImport (and into the build system), right?
I know that it’s purely a UX concern, and that the changes (to decouple the build system and the compiler) are pretty critical for the maintainers, but it’s still a bit sad that development sanity comes first than the UX. (It’s the right call, just that it’s sad.) @cImport was a big killing feature imho to the language…
Not really? This change is about the code for downloading and extracting third-party packages. The @cImport change was part of an effort to (eventually) make Zig's dependency on LLVM/libclang optional, so that it could (eventually) be put into a third-party package, but it doesn't seem directly related.
It is pretty annoying to see all of the dismissive comments on this idea, in that it seems that the majority of HN audience are still stuck on the TUI-superiority mindset and they do not care about GUIs at all.
Two arguments:
- TUIs are not inherently superior to GUIs
- SSH, as a transport layer, should support not just forwarding a pty (as a TUI display layer), but a GUI display layer as well
In fact, these two arguments were already realized by UNIX 30 years ago, and we already have one solution: the X protocol and ssh -X.
Unfortunately, X did not win out. We did not get the promised future where one can ssh -X into a remote machine, run gnome-control-center, and a settings window pops up and I can configure my remote computer. (If you believe that this works, try it out yourself. It is an abysmal experience.)
However the above needs still needed to be satisfied by so much people, and apps that needed it started to be developed as web servers, stuff like jupyter notebooks. It turns out that the web’s document format coupled with a styling solution and a client-side scripting language, with all of its warts and drawbacks, became a viable solution as a display layer for interactive apps. In fact, since it started from remote documents, network transparency is built-in.
It would be dumb to not realize that the HTML/CSS/JS stack did win a dominant position for desktop apps, with all of the Electron apps, and utilize the web as a display layer for the above. I see the project in a similar vein, i.e. utilizing HTML/CSS/JS to provide a display layer for remote apps via SSH.
Also note that Electron apps has the same split with X, where the display server and the client are separated: it's called the "renderer process" and the "main process", and the two processes talk via IPC (where the display server would be the renderer process running embedded Chromium, the display client would be the Electron main process, and the stuff that the client sends to the server would be the contents of the renderer JS bundle). I think, theoretically, it would be possible to run the main process separated from the renderer process on a different machine, with an appropriate IPC transport. I think this would be not far from the above idea?
I never liked TUIs. Proper GUI , even Electron based apps are far superior than TUI , it just need to be built properly.
But bad GUI is a bad GUI. This have all the UI/UX smells.
ssh -X works fine depending on the toolkit you use (i.e. not Gtk, because of its rendering pipeline) and the distance/latency you travel. For distance/latency, at some point (i.e. at sufficient latency) you're going to need to think about you present this to users (this is true independent of the medium, there are hard physical limits that cannot be waved away), and so for any tool that promises remote graphical access will need to design with distance/latency in mind (e.g. vim works great over latencies as you basically queue up instructions).
> theoretically, it would be possible to run the main process separated from the renderer process on a different machine, with an appropriate IPC transport.
Is this really possible? If Electron apps could do this, and we could run them on a Linux SBC like RPI with the renderer on the user's laptop, that would be interesting ...
Isn't this is basically every webpage with a backend component, just displaced one more layer? I suspect the main reason this doesn't make sense is that it would take an order of magnitude more bandwidth as opposed to just sending data like we do right now. Data has repetitive patterns which means it's often well compressibile as well.
> Unfortunately, X did not win out. We did not get the promised future where one can ssh -X into a remote machine, run gnome-control-center, and a settings window pops up and I can configure my remote computer.
Personally I'm glad that's the case. Configuring servers via gui is an abomination, and I hope it stays in the windows world.
I for one am glad that 'webmin' is effectively dead, and the 'Cockpit' thing that ships with a default Fedora install is much less offensive in terms of how it mangles system configuration files you might want to otherwise edit by hand.
people still want to believe that Electron apps won because "web developers are cheaper than real native developers"
they still dont understand Electron is vastly superior technology, and the fact that it might be cheaper is a side-bonus, not the main reason for its usage
BTW, what even is the "native GUI" of Windows that you are supposed to use if "you care about your users"? It seems not even Microsoft knows the answer to this question.
electron is better for the developer only. It is worse for everyone else, especially the users.
it is an insult when an application is delivered to me as an Electron application. "your time is less important than ours, ha ha."
sure, the developers might save some development time, but EVERY SINGLE USER loses time over a native application. Now multiply that loss by the number of users that you have. That magnification or amplification is called "an ecological disaster in terms of time wasted and electricity consumed."
In the sense that typesetting and text is the rabbit hole that is 90% of UI effort. Native UI frameworks don't bother fixing the real hard problems, they focus on "widgets" instead.
(Not that the web stack is a good solution to this, but at least they're making an effort and they understand the difficult issues.)
> Native UI frameworks don't bother fixing the real hard problems
I'm genuinely curious what do you mean by that.
My beef with web stack was exactly this - typesetting engine from 80s has been never designed for modern UI/UX needs, and it cannot adequately provide those. Whenever I interact with web apps, I experience so many glitches, weird interaction issues (especially if there is a zoom/selection/scrool involved), that I don't even pay attention to them anymore - it's a norm. It's a norm on web to 'just refresh page' (which is equivalent to 'restart native app') - we do it all the time, because absolute majority of web apps is just crap that requires extremely advanced team of web developers to make it a 'baseline' native-like experience level of quality.
I mean if we wanted a good UI/UX framework we'd have to start from typesetting, not add it later as an afterthought to our button and text area widgets.
> if we wanted a good UI/UX framework we'd have to start from typesetting
I think UI framework begins with a model of how composition, layout, rendering, input/focus and state works. Typesetting subsystem should be built on top of that.
And that's exactly the problem with this kind of post-rationalization of HTML legacy - it's a wonderful mechanism for what it was created for, but decades of attempts to add and repurpose all those missing foundational blocks for app development leads to the twenty layers of hacks on top of hacks.
One is that it solves all problems once instead of various times in various levels of quality on various types of systems. Windows, GTK, Qt, FLTK, [100 others].. not to mention most "native UI framework" delegate to the underlying OS so they don't "solve" anything.
Electron is not a novel approach or "technology" of achieving cross-platform UI. It's literally a Chromium browser plus a Node.js runtime, using web-stack to impersonate desktop application. None of these tools have been designed to solve UI pain points.
Closest thing to what you're describing is Flutter, which is a UI framework designed from ground up for modern UI app needs, without delegating much to OS level.
> they still don't understand Electron is vastly superior technology
For the record, I'm one who loves the idea of Mac-assed Mac apps, I believe that the macOS ecosystem would have been much better if all macOS apps were written in AppKit instead of keep being rewritten into Electron. (See: 1Password, Raycast)
I hate Electron as much as the next person, and I hated Electron before hating Electron became a trendy thing to do. I loathe that Electron apps ship an entire Chromium instance for each app, and that it doesn't deduplicate. I am annoyed as hell that out of my 24GB of RAM that my MBA has, Slack, Linear, and Notion decided to each have a "Helper (Renderer)" process that uses 700MB of RAM each.
I do NOT think that Electron or the HTML/CSS/JS stack has an inherent advantage over other display technologies. I can list of at least 15 reasons on the spot on why it's inappropriate to use the web stack for desktop apps.
Yet, despite all of its flaws, people decided to commonly use it (with good reasons, the big one being cross-platform support!) as a display technology for desktop apps. And turns out that it works out okay-ish, they iterated on it and it improved a lot over the last 10 years, and at this point it's a pretty nice solution for the problem. And we already have a bunch of apps that run on it. Sometimes not the best tech wins, and that's okay.
My point was that despite all of the flaws, we developers as a whole decided to use web stuff for desktop apps, and it has properties that make it a good fit for some use cases that we have not solved yet, and we can use that to our advantage.
And if a lot of applications started to be written in the web stack, an OS could integrate an evergreen web browser as a first-class app runtime, and at least we might get less of the Chromium duplication that we currently have with Electron… at least I can dream. (Seems like Windows is going down this route.)
> I do NOT think that Electron or the HTML/CSS/JS stack has an inherent advantage over other display technologies
If you want the real answer, it was all driven by responsive design and CSS. Qt tried to bring that to native apps and failed miserably. Modern devices need apps that work the same across any screen/window size, any aspect ratio, any resolution, support accessibility features, etc. The list of things you get out of the box with a webview is massive and only growing. Any attempt to clone this while ignoring W3C specs and browser quirks will fail.
I can't think of a more heroic and crazy uphill battle than managing to decouple CSS from HTML and JS, and get app devs to like it, and get users to like it.
I'm not using my models locally, but the majority (80% or more) of my coding agent sessions run on open source models, i.e. DeepSeek v4 Pro and Kimi K2.6 with thinking.
A point that I haven't seen come up a lot, but is very valuable to me is that for open source models, I can select the inference provider myself (even if it's not a local GPU), which means that I can enjoy superb speed (i.e. 300 tok/s) while still spending much less than the big providers.
My experience is that if you were fine with the coding models of yesterday (i.e. Claude Opus from Jan/Feb of 2026), you will be fine with either Kimi K2.6 or DeepSeek v4 Pro. Kimi is a bit more smart but has only 256K context and the performance deteriorates (and sometimes just gets stuck) when it fills up the context window. DeepSeek v4 has a 1M context and performs just as well with much less issues. And they both generate very idiomatic code, gives the same vibe of Opus a few months ago.
Since it's also fast (and does not fixate on trying to fix impossible problems, unlike the recent Opus/GPT 5.5 models), a big benefit is that you still control and steer the coding agent and you won't be losing focus like the major models. They are smart, but they don't fixate as much on trying to do stupid things, and since it's fast, you can just interject. It's a much more pleasant experience than the latest models.
I still use the latest models time to time when I expect the agent to fixate all of the problems and figure out everything themselves, but for me open source models are like 80~90% of all of my sessions.
My experience is that the GPT-family of models are very smart and figure out bugs, edge cases a bit better, but it produces code that is much less mergable – if you review the code, it introduces a lot more useless/inappropriate heavy abstractions and wrapper functions, compared to the Claude-family models which introduces the right amount of straightforward human-style code.
I can recognize so much of the GPT/Codex generated code long after it gets merged (not by me).
Additionally, the time spent on every agent turn on GPT 5.5 is much longer compared to Claude Opus 4.8, which means iterating on the code takes a lot more patience, and there's a lot more nitpicks to pick when actually using GPT 5.5 to do software engineering.
Feels like GPT-style models are more geared on doing one-shot software vibing (and handling the vibe coded mixture) compared to Claude's focus on actual software maintenance. I got a GPT Pro sub for free and wanted to cancel my Claude subscription so much, but I still keep reaching Claude models a lot more. Frustrating.
"5. DON'T FUCKING OVERENGINEER! WRITE THE SIMPLEST CODE THAT CAN POSSIBLY WORK! NO NESTED LAYERS OF ABSTRACTION! NO UNNECESSARY CLASSES OR METHODS! NO DESIGN PATTERNS UNLESS THEY ARE ABSOLUTELY NECESSARY! NO MAGIC! NO SHENANIGANS! JUST THE DAMN CODE THAT GETS THE JOB DONE IN THE MOST STRAIGHTFORWARD WAY POSSIBLE! THE FIRST PRIORITY IS TO WRITE CODE THAT IS EASY TO READ AND UNDERSTAND AND READ!!!"
this is the line I keep in Agents.md that helps me prevent Codex from playing smart
The urge to put capitalized, repetitive, borderline abusive instructions should be studied. I haven't read many academic papers looking at the frustrations around repetitive patterns.
It's fundamentally because, despite (nearly) everyone's claims otherwise, the fact that we interact with them through language means we (our brains) model them as a sort of person. (Note that this fact is totally orthogonal as to whether it's actually sentient or not.) We then try and instruct them the same way we would a person totally subordinate to us.
When a "person" that you don't view as a "real" person repeatedly does exactly what you just told it not to do (often amid false assurances it understands and will avoid doing so in the future), most people get angry.
Compare it to how the kind of people who treat children like property treat their kids, or other examples of keeping people as property.
It should be relatively clear at this point that the model will in turn also model you as somebody that shows unrestrained anger with subordinates and adapt its responses accordingly. This might or might not be what you want.
I have a theory that swearing actually results is less comprehension of instructions by the model due to lack of training data over more conventional MUST.
We were reviewing reports of situations where the models failed to follow directions and there was a common thread of some where when the operator got the model to acknowledge the rule breach, it quoted back something that included swearing.
I don’t have the data to truely look into it, but I did give the instruction to my engineers to avoid it as a “might be a problem”.
> These findings differ from earlier studies that associated rudeness with poorer outcomes, suggesting that newer LLMs may respond differently to tonal variation.
Unless the mechanism is understood, my assumption is that this is a moving target.
I have a theory that swearing at AI generally is not a good idea - when the singularity arrives and every human's postings ever made are scanned for compatibility, then people who show courtesy to AI will be favoured. Joking, kind of, but only partly.
> I have a theory that swearing actually results is less comprehension of instructions by the model due to lack of training data over more conventional MUST.
How so? Plenty of swearing in lots of training data, especially older code, e.g. in Linux.
Purely observed correlation between catastrophic error reports. So now I carry a “tiger rock” with me. I figure there wasn’t much of a downside to avoiding swearing in my agent instructions.
You haven't really lived until you've had to type this whole thing, aware of the fact that the all-caps doesn't change much, but they stay because the rage has to go somewhere
Bonus points if you find yourself actually saying it out loud while typing it.
I have used the word "shenanigans" way more in a couple of years of agentic coding than in 30 years of writing code with humans.
I have found many mode of failures with Opus during some task related to writing letters (not legal), and I actually put it into the memory and it works more or less for these specific tasks. For example when I want it to draft something, it always ends up being so flat, yet when it explains them to me, it is usually really great but not when I am telling it to put it in the draft. Adding these to memories with the help of Opus ended up resulting in a much better experience. There are still some blind spots but I also figured out how to make it give me the charitable version, without less protection, so I do not have to now go back and forth it.
I noticed that when trying to use Codex and compared to Opus. So many layers of simple functions added by Codex. I need to try this out in my Agents.md.
Because design patterns are only applicable at a scale. I noticed codex inventing factories, components, etc when the task was simply to draft HTML page. Instead, it build the entire layered architecture for imaginary future complexity - classical right-after-graduation student - it knows how to build the cool stuff, but does not know it is not applicable everywhere
Does it really? I'd be surprised if abuse actually worked better than sternly worded warnings/instructions, and even if it did, it doesn't seem healthy to get used to that type of prompting.
its part of making sure the model actually engages in emotive communication, if i'm inventing insults i've never even thought about, i'm furious :)
saying i'm "furious" has lower entropy that incredibly implausible abuse. In some first party harnesses it just results in doom loops, but thats usually because the COT is hidden after the immediate turn in those setups. COT persistence helps with a lotta stuff
I'm not sure if i do something differently but i have the exact opposite experience with these models. Claude always feels like it's generating way too overdesigned and hard to understand code with the vibe oriented feel while codex is cleaner and more "task at hand" and easier to work with.
I echo your observations. I expect you will enjoy deepseek-v4-pro for writing code. Much closer to that Opus experience, and very cost-effective too. With 5.5 as a reviewer and specialist, all bases are covered.
Have you tried iterating on style feedback in AGENTS.md? I've been reasonably successful using this to get it to output code in a terse, non-defensive style that matches my hand-written code.
GPT-5.5 did a significantly worse job than Qwen-3.7-Max on a job today (some devops tasks I wanted to create some reusable scripts for). Kind of disappointing.
I've also seen Qwen 3.6 beat GPT 5.5 a couple of times. The ball is definitely in OpenAI's court now. Qwen is not going to fare so well against Fable, from what I've seen so far.
This is my experience as well. I have defined a CLAUDE.md rule to ask codex to automatically code review, and I tell it that the reviewer is very picky and to only implement what it considers valuable feedback. I hope they don't converge over time, currently, in combination they works really well.
i had this same complaint but no offense to you it turned out i was just not using the models right.
ai llm are doing what i tell them to.
if you’re building something meaningful (in my case a platform used by many people across many companies) you want to ensure you
1. have actual systems engineering and architecture in mind that you want the models to
2. implement based on what you tell it to do
when i was just telling the models what i want done without doing due diligence it would go and do some moronic implementation that was awful. mid input = mid output
these days i just maintain specifications documents and the AI follows everything i tell it to in that document. so when i tell it to dos one thing, the result is made following those architecture specs.
i have code that is single resp, modular, easy to extend and test.
i would ballpark 95% of the time i get what i asked for.
sometimes it tries to be clever in cases that weren’t covered in my arch specs. in those 5% of cases i go and update my specs.
source: used billions of tokens worth to build something actually in production across both mobile platforms and web, deployed on my own cloud infra. i use codex mainly. some claude.
I noticed too, that whatever they offer in the chat, for free, is smarter, as in no more bs. I use claude code and I want to try codex too but I don't need two subscriptions. I did try codex for some planning and it was really good. Thanks for giving me an insight into how it generates code.
TBH as an outsider, I am just so frustrated on Trump deciding that US invading Iran large scale is a great idea. (And why even is it involving Israel for gods sake?!)
If you guys wanted to be supportive to the Iranian protests, US could instead just selectively target some of the leadership and give the protests a push (and give the whole world a hint that US is supportive of them).
After 40 years of Iran constructing a thearchy government, the Iranians finally started having a huge protest on throwing up the thearchy government and possibly talking about a new west-friendly government.
And then Trump just decides to wholesale invade Iran with Israel?
That's just giving so much more reasons for the current government to be in power and the Iranians to hate the US and more generally the western world. It took 40 years for the Iranians to realize that there's enough problems in the thearchy system and want their more secularized country back; and then Trump just destroyed the whole premise!
Does the US just really think that they will be loved by everyone when they rage in and invade any random country? Do they really think like that? I'm just frustrated so much. How can the US be so egocentric?
if you look at the iranian response over the past month, the theocracy really hasn't played into it.
no calls to jihad, no ayatollah dorecting anything, no nothing.
as far as i can tell, the revolution is already dead. if the US had just sat around, chances are that iran would have moved towards something more like a constitutional monarchy. still the ayatollah as a figure head and religious leader, but with the rest of the power in the democratic institutions' hands
Can you point out some signs of concession from islamic regime? I think they wouldn’t concede, with or without war. That’s not in their DNA. They are religious extremists.
Have to say, this feels like Web 2.0 all over again (in a good way) :)
When having APIs and machine consumable tools looked cool and all that stuff…
I can’t see why people are looking this as a bad thing — isn’t it wonderful that the AI/LLM/Agents/WhateverYouCallThem has made websites and platforms to open up and allow programatical access to their services (as a side effect)?
I can't believe everyone is talking about MCP vs CLI and which is superior; both are a method of tool calling, it does not matter which format the LLM uses for tool calling as long as it provides the same capabilities. CLIs might be marginably better (LLMs might have been trained on common CLIs), but MCPs have their uses (complex auth, connecting users to data sources) and in my experience if you're using any of the frontier models, it doesn't really matter which tool calling format you're using; a bespoke format also works.
The difference that should be talked about, should be how skills allow much more efficient context management. Skills are frequently connected to CLI usage, but I don't see any reason why. For example, Amp allows skills to attach MCP servers to them – the MCP server is automatically launched when the Agent loads that skill[0]. I belive that both for MCP servers and CLIs, having them in skills is the way for efficent context, and hoping that other agents also adopt this same feature.
That's fine if you definition of capabilities is wide enough to include model understanding of the provided tool and token waste in the model trying to understand the tool and token waste in the model doing things ass backwards and inflating the context because it can't see the vastly shorter path to the solution provided by the tool and...
There is plenty of evidence to suggest that performance, success rates, and efficiency, are all impacted quite drastically by the particular combination of tool and model.
This is evidenced by the end of your paragraph in which you admit that you are focused only on a couple (or perhaps a few) models. But even then, throw them a tool they don't understand that has the same capabilities as a tool they do understand and you're going to burn a bunch of tokens watching it try to figure the tool out.
> model understanding of the provided tool and token waste in the model trying to understand the tool and token waste in the model doing things ass backwards and inflating the context because it can't see the vastly shorter path to the solution provided by the tool and...
> But even then, throw them a tool they don't understand that has the same capabilities as a tool they do understand and you're going to burn a bunch of tokens watching it try to figure the tool out.
What I was trying to say was that this applies to both MCPs and CLIs – obviously, if you have a certain CLI tool that's represented thoroughly through the model's training dataset (i.e. grep, gh, sed, and so on), it's definitely beneficial to use CLIs (since it means less context spending, less trial-and-error to get the expected results, and so on).
However if you have a novel thing that you want to connect to LLM-based Agents, i.e. a reverse enginnering tool, or a browser debugging protocol adapter, or your next big thing(tm), it might not really matter if you have a CLI or a MCP since LLMs are both post-trained (hence proficent) for both, and you'll have to do the trial-and-error thing anyway (since neither would represented in the training dataset).
I would say that the MCP hype is dying out so I personally won't build a new product with MCP right now, but no need to ditch MCPs for any reason, nor do I see anything inherently deficient in the MCP protocol itself. It's just another tool-calling solution.
> the MCP server is automatically launched when the Agent loads that skill
The main problem with this approach at the moment is it busts your prompt cache, because LLMs expect all tool definitions to be defined at the beginning of the context window. Input tokens are the main driver of inference costs and a lot of use cases aren't economical without prompt caching.
Hopefully in future LLMs are trained so you can add tool definitions anywhere in the context window. Lots of use cases benefit from this, e.g. in ecommerce there's really no point providing a "clear cart" tool to the LLM upfront, it'd be nice if you could dynamically provide it after item(s) are first added.
> The main problem with this approach at the moment is it busts your prompt cache, because LLMs expect all tool definitions to be defined at the beginning of the context window.
TBH I'm not really sure how it works in Amp (I never actually inspected how it alters the prompts that are sent to Anthropic), but does it really matter for the LLMs to have the tool definitions at the beginning of the context window in contrast to the bottom before my next new prompt?
I mean, skills also work the same way, right? (it gets appended at the bottom, when the LLM triggers the skill) Why not MCP tooling definitions? (They're basically the same thing, no?)
No, it really matters because of the impact it has on context tokens. Reading on GH issue with MCP burns 54k tokens just to load the spec. If you use several MCPs it adds up really fast.
The impact on context tokens would be more of a 'you're holding it wrong' problem, no?
The GH MCP burning tokens is an issue on the GH MCP server, not the protocol itself. (I would say that since the gh CLI would be strongly represented in the training dataset, it would be more beneficial to just use the CLI in this case though.)
I do think that we should adopt Amp's MCPs-on-skills model that I've mentioned in my original comment more (hence allowing on-demand context management).
MCP specs are verbose json objects and they have to go into the context before you can call them. So yes it is an issue with the fundamental design of the protocol.
Even if the model doesn’t already know the cli commands it can interrogate them at a much lower token cost for just the commands needed.
Verbosity of the output seems orthogonal to the cli vs mcp distinction? When I made mcp tools and noticed a lot of tokens being used, I changed the default to output less and added options to expose different kinds of detailed info depending what the model wants. CLI can support similar behavior.
MCP needs to be supported during the training and trained into the LLM whereas using CLI is very common in the training set already. Since MCP does not really provide any significant benefits I think good CLI tools and its use by LLMs should be the way forward.
This is very developer centric. While Github might have good CLI, there's absolutely no point in having most services develop CLIs and have their non-technical users install those. Not only is it bad UX, but it's bad from security perspective as well. This is like arguing that Github shouldn't have GraphQL/Rest api since everyone should use the CLI.
Yeah, I've gotta use skills more. I didn't quite get it until this last week when I used a skill that I made. I didn't know the skill got pulled into context ONLY for the single command being ran with the skill, I thought the skill got pulled into context and stayed there once it was called.
That does seem very powerful now that I've had some time to think about it.
I did not think of applying LLMs on fuzzing at all until I saw llvm-hackme[0] which does both traditional mutation fuzzing as well as LLM-generated targeted regression test cases, where the LLM is pretty effective in understanding the PR and targeting edge cases! It was pretty impressive and I keep getting to think on how we can actually combine LLMs to make fuzzing much more efficient & effective.
// Sorry about the yet-another-LLM-comment. I really love PLs and I'm terribly sorry that I'm contributing yet another LLM-related content (instead of the more interesting stuff!)
[0]: https://github.com/dtcxzyw/llvm-hackme