So, I was thinking about how modern LLMs got better at coding in the last 1 year, despite not having much good new training data.
The way it was done is largely via reinforcement learning with verifiable reward (and a little bit with human feedback).
Basically, you use an LLM to make slop, random token choice and everything, monkeys on typewriters style, and then if slop passes a test, you use said slop as the training data. You use the compiler, unit and integration tests, any static analysis tools, etc etc as verification.
For natural language, attempts to do anything like this work miserably, the whole slop centipede, model collapse, all that. But for coding this turned to be unexpectedly effective.
So that raises the question: can an LLM coding assistant be developed like a normal corporate project, with no YOLO exemptions from software legal compliance, without any special treatment like it’s a favorite child that grows up to be a techbro? Like normal fucking software?
I think, the answer is probably a yes on the purely technical level but still a hard no commercially.
On the technical level, an LLM could be pre-trained on compatibly licensed open source projects, RLVR’d in a license-compliant way.
The end result isn’t going to be good for plagiarism, though, and won’t use plagiarism to cover every gap in capabilities. I think it’s pretty obvious all the folks who are trying to clone Adobe or whatever would reject such a product.
Meanwhile the one “strength” of LLMs is this intuitive argument that if you have a good enough predictor of what your workers do, then you don’t need workers. That is the whole reason the “AI companies” are rolling in investor money and are able to operate at such large loss.
The other thing is that having done copyright infringement, OpenAI/Anthropic/etc etc have to portray it as necessary, and alternative as impossible.
Anyhow basically my thought on this is that I’ll continue not using any LLM coding tools in my personal projects until some acceptable tooling appears and proves itself actually useful.
edit: For a concrete example if someone makes a tool that can take the compiled assembly, run it through non-LLM translation to some much abridged Lean and prove that nothing depends on exact timing of threads (no race conditions) or identify race conditions, or complain it could do neither, would I use it? If the carbon footprint is not insanely greater than what I would do instead (which is either a: run stress tests overnight, or b: ship buggy software that users have to restart, which also has a large energy footprint), sure.
Hmm, I guess they could have written a lot of code lines, but its not like the code they write would be equivalent to million LOC of a coherent whole.
You can also do a lot with contractor code for RLVR, like have contractors make extremely comprehensive regression tests (simulate user clicking etc etc, compare screenshots) and then you can do a lot of training on replicating pieces of the app.
My understanding is that LRVR for mysterious reasons results in somewhat better generalization, probably because the training samples are always tailored to the model they are applied to.
I do actually have a decent GPU i bought 4 years ago in my PC, so maybe I’ll do a few simple experiments again… my preferred approach is to pick a simple but nonlinear problem e.g. binary multiplication of two large numbers, where you e.g. skew the training sample distribution or omit some number pairs and see if the resulting model can learn correct multiplication anyway.
If a reasonably small model doesn’t generalize multiply beyond seen numbers, then of course a large model isn’t gonna do the whole “soulstealing” thing that the techbros hope would allow jobstealing. And if it only generalizes beyond seen numbers if using RLVR, that bodes well for making various tools that could perhaps actually generate some value, which is very different from antiemployment shit techbros originally embarked on doing.
I’d have guessed it was better for a training set to have lots of different mostly-known-good solutions to common problems and good visibility of the solution space, rather than one linux-sized program, even with the same bug rate between the two. I’d imagine leetcode is monetizing their user generated code this way too. IDK.
For that particular experiment, I’d also point you towards some of the 90s literature on “genetic programming”, because you could reuse most of your scaffolding for it. No GPU req’d, ought to learn it fairly quickly.
I’m specifically interested in “neural networks” and transformer architecture and limitations of what it can and can’t learn, though. Specifically, if blinded to some number pairs, can it learn to multiply numbers it hasn’t seen?
Or with all number pairs but some pairs being more common than others.
The curious thing about learning, is that what can be learned in practice is only a minuscule fraction of what you could get if you could just guess the right weights somehow.