I don’t know- AI is a very fast-moving and ill-defined field, and I think even an expert who tries to predict where it’s going has a larger-than-usual chance of being completely incorrect.

That being said, Big Yud is so clearly bad at talking about AI that I trust him about as much to talk about AI as the average CS undergrad.

Examples from this piece:

I would even say that a neural network is more analyzable [than a symbolic system] - since it does more real cognitive labor on board a computer chip where I can actually look at it, rather than relying on inscrutable human operators who type “|- Human(Socrates)” into the keyboard under God knows what circumstances.

We don’t yet even have a strong grounding in why neural networks work, afaik (feel free to correct if I’m off here, not fully up to date on the field). Like, lots of cutting-edge methods have no rationale beyond “lol it performs better on MNIST / CIFAR-10 / ImageNet.” And the complaint that knowledge bases are complicated? Has he ever seen training data? Data are, in general, awful.

If you believe that connectionist AI is a simple, reasonable thing, take a look at some of the examples of neural networks being tricked into making utterly bizarre image-misclassifications. They’re fantastic.

So I’m just mentioning this little historical note [that going from connectionism to backpropagation took 17 years] about the timescale of mathematical progress, to emphasize that all the people who say “AI is 30 years away so we don’t need to worry about Friendliness theory yet” have moldy jello in their skulls.

He’s talking about how progress in AI can be much slower than expected… in order to argue that we should be worried about immanent general AI? I mean, I know his point is that it would take a long time to develop “Friendliness theory,” but still, he’s undermining his own point.

(Doubly so because this is an essay about how general AI needs to come from a radically new idea we don’t even have yet, so what he proposes to do is develop a theory for an AI system we not only don’t have, but also don’t even understand the basis of? It’s like asking someone who doesn’t know what a stock is to help you write regulations for high-frequency trading.)

This robot ran on a “neural network” built by detailed study of biology.

lol he’s complaining about people getting sucked into incorrect neural network hype but he’s bought into the “neural networks are totally biology” hype.

(Hot take: we should ban machine learning researchers from using the phrase “neural” until they can explain, at an undergraduate level, how a synapse works. They should also be banned from saying “artificial intelligence” until they have a consistent definition of what “intelligence” is, which, given that nobody does afaik, should take a while.)

I don’t know anything about SI/MIRI’s proposals or results. I’ve tended to follow Deep Learning papers for the past couple of years, and I haven’t recognized any work from them, if I’ve seen it. Doesn’t mean much, though… It’s a huge field.

I agree with the overall strategy he seems to be outlining in the OP link, of mixing logical/Bayesian systems with connectionist. There are interesting contemporary architectures which involve something like that: Use the perceptual and decision-making capabilities of neural nets to drive some kind of symbolic or probabilistic system. I think it’s the way forward.

You seem to be sceptical about AI-safety-research-by-amateurs rather than AI.