Greg Egan’s 2010 novel, Zendegi, has a few scenes satirizing rationalists and their “ideas.”
Here are five of those moments:
1. A Rationalist’s Introduction
‘Are you Nasim?’
‘Yes.’
‘I’m Nate Caplan.’ He offered her his hand, and she shook it. In response to her sustained look of puzzlement he added, ’My IQ is one hundred and sixty. I’m in perfect physical and mental health. And I can pay you half a million dollars right now, any way you want it.
[…]
‘Just give me your email address.’
‘Absolutely not.’ Nasim increased her pressure on the door and he started yielding.
‘You can always reach me through my blog!’ he panted. ‘Over-powering Falsehood dot com, the number one site for rational thinking about the future –’
2. Who Needs to Read Books?
‘Not compression for the sake of bandwidth,’ Mike explained, ‘compression to save the reader’s time. Abridgement. Like Reader’s Digest Condensed Books, but fully automated, and based on a rigorous scientific analysis of what readers will actually retain […] surely we can figure out what words can be omitted from a great slab of Melville or Proust without altering the impression that they leave behind. People are far too busy these days to indulge in rambling, discursive novels… but if they can just feel just as Prousty in two hours as they would have in eight, every word lost is time found.’
3. Fast Take-off is Inevitable
‘I have been invited to fund an enterprise known as the Benign Superintelligence Bootstrap Project,’ Churchland explained. ‘Their aim is to build an artificial intelligence capable of such exquisite powers of self-analysis that it will design and construct its own successor, which will be armed with superior versions of all the skills the original possessed. The successor will then produce a still more proficient third version, and so on, leading to a cascade of exponentially increasing abilities. Once this process is set in motion, within weeks – perhaps within hours – a being of truly God-like powers will emerge.’
Nasim resisted the urge to bury her face in her hands. However surreal the spectacle unfolding on the screen, there was, in retrospect, something inevitable about it. The uploading advocates who’d sold Churchland on an imminent digital resurrection hadn’t lost their critical faculties entirely, but their penchant for finessing away any ‘mere technical problems’ that might stretch out the timetable was, nonetheless, intellectually corrosive, to the point where the next step probably didn’t seem like such a great leap anymore: hand-waving all practicalities out of existence, transforming the cyber-eschatologist’s rickety scaffolding of untested assumptions into a cast-iron stairway to heaven.
[… Churchland said, ’]I am leaning towards putting my fate in the hands of an artificial God, for whom such problems will be trivial. The Benign Superintelligence will rule the planet with wisdom and compassion, eliminating war, disease, unhappiness, and of course, death. I am told that it will probably disassemble most of the material in our solar system in order to construct a vast computer that will exploit all the energy of the sun. Perhaps it will spare the Earth, or perhaps the Earth will be reconstructed, more perfectly, within that computerised domain….
There’s no point in fighting it, and the alternative would be far worse. Imagine if one of our country’s enemies did this first. Imagine the kind of despotic superintelligence that Al Qaeda would create.”
4. Fifteen-Years Later, Fast Take-off is Inevitably Even More Inevitable
By carefully studying the HCP data over the last few months, the Superintelligence Project had acquired vital clues that would allow it to construct a Class Three Emergent Godlet within five years.
‘And when that happens, what can we expect?’ Bello asked.
‘Within two or three hours, the planet will be entirely in the hands of the Benign Superintelligence. Human affairs will be reorganised, within seconds, into their optimal state: no more war, no more sorrow, no more death.’
‘But how can we be sure of that?’ Bello probed fearlessly. ‘Computers are capable of all kinds of errors and mistakes.’
‘Computers built and programmed by humans, yes,’ Esch conceded. ‘But remember, by definition, every element in the ascending chain of Godlets will be superior to its predecessor, in both intelligence and benignity. We’ve done the theoretical groundwork; now we’re assembling the final pieces that will start with the chain reaction. The endpoint is simply a matter of logic: God is coming into being. There is no disputing that, and there is no stopping it.’
5. Communicating with Rationalists
Nasim struggled to reorganise her tactics. How did you get through to someone whose entire world view had been moulded by tenth-rate science fiction? Empathy [was … out]; Caplan probably believed that the only consequence of being orphaned at six was that you tried harder than anyone else to reach the top of your class in space academy.
One of those perfect pieces of satire which will have now probably already been said by somebody. Bingo, thanks UN (This report is actually about AI not AGI, but lets not let that get in the way of things ;) (I didn’t read it well enough to see if they mention AGI as a serious threat, which doesn’t seem to be the case.))
Oh my god. This seems to perfectly capture so many of my thoughts and feelings about AI, in better words than I probably could have expressed them. How have I not known about this novel for over a decade? I will definitely check it out. Thanks for sharing!
Yeah Egan is exactly the kind of person I’d otherwise expect to be in with the rationalist crowd and I’m very glad that he’s not (cos his books are really good)
And EY has said Greg’s Permutation City is his favorite SF book ever. That must sting a bit!
I feel like a lot of rats must have felt seriously hurt to see Egan coming for them. Like being stabbed by William Gibson or Bruce Sterling.
Greg Egan is probably my favorite SF author of all time. It’s funny that so much of what the rationalists obsess about seems to be just derivative and overly serious adherence to ideas that Egan wrote novels about 30 years ago.
…so, CliffNotes.
Is he spoofing eliminative materialism with “Churchland” too? I do like the BS (Benign Superintelligence) pun.
Can’t wait to read, this was prescient stuff in 2010, before Bostrom seduced Max Tegmark and Jaan Tallin and Stuart Russell parseltongued Hawking and Musk.
The problem with comedies is that they waste time on jokes and humor, if we had an AI that wrote humorless comedies without any jokes them it would allow people to get through comedies much more efficiently.
This is excellent. Everything is in there.