There is a benchmark out there that hints at this phenomenon: https://www.scbench.ai/ The idea is to track how well the model modifies code over multiple change requests. No surprise for readers here that the LLM will bitrot your code quickly, and the odds of it doing so is worse than a coin toss.