So the boosters are making a big deal about the dump of “proofs” OpenAI has made recently, so I thought I would collect some relevant links in one place…
Stuff on autoformalization:
- A short blog post breaking down some of the bugs found and potential for more
- preprint discussing the problems with natural language to lean in detail and pointing out some blatant inconsistencies in the recent Navier-stokes “proof”
- /r/betteroffline discussion of above preprint
- Some of our very good friends are aware of the problem! Of course, this is lesswrong, so there is a tinge of “this is why we need to be careful with using this sort of approach to make our AI God”.
- The soundness bug that led to the fallacious disproof of the Collatz Conjecture
And some other related links:
Let me know in the comments if you have any links I should to these lists!
Here is a video I came across today that seems relevant to the discussion: https://www.youtube.com/watch?v=KqiDqJMxg6o
He is a professor at Imperial College in London.
He says that a potential negative consequence of having AI do the math is that if people stop doing going into math, then there will not be enough people to understand and check the math proofs from AI and to push back against bad proofs.
A few years ago I was at the local Java meetup and the speaker was from Azul Systems. They contribute to OpenJDK, offer support contracts, and I think they used to make a special JDK for embedded systems. He said there is a big sign at one of their offices that said: “We write in C, so you don’t have to”.
I took enough C to understand what things like the JDK or the Golang runtime are doing for me wrt things like memory management. I do not know a lot about C, or anything about the C compiler, or the assembly, or the chip, or all the layers underneath. But I do know that there are people who do know that.
The point of all that is: What if we wind up in a world where nobody understands how the JDK or other runtimes work anymore? Or if the only people who do use their knowledge to break or take down systems running one of these runtimes? That is where I think things are heading.
We started this decade with doors falling off of airplanes, and a lot of people seemed to think that is a wonderful idea.