And the comments about it on Lesswrong in case you want to see all
the but ahkshuallies and second guessing real expertise:
https://www.lesswrong.com/posts/FijbeqdovkgAusGgz/grey-goo-is-unlikely#comments
Eliezer’s default scenario for how the AI bootstraps its vast
intelligence into actual influence and agency in the real world is
nanotech. Fortunately for him and the rest of the world, doing better
than real biology given the resources available in the real world is
actually really hard so a grey goo is basically impossible.
TLDR:
worrying about “grey goo” is a waste of time
Some choice sneers:
But…what if a superintelligence finds something I didn’t think
of?
I know, right? What if it finds a way to travel faster than light and
sets up in Alpha Centauri, then comes back? What if it finds a way to
make unlimited free energy? What if it finds a friendly unicorn that
grants it 3 wishes?
Shoots fired at one of Lesswrong’s idols:
von Neumann (who didn’t invent the “von Neumann architecture” or half
the other stuff he took credit for, but that’s off topic)
And a jab at Drexler at Smalley:
Perhaps I would thank Drexler if he actually pushed people away from
working on carbon nanotubes, but he didn’t.
I find it mildly irritating at the fact that LWers can't seem to grasp BACTERIA AREADY ARE NANOMACHINES. Living cells aren't just random smooshes of organic materials, they contain complex mechanical apparatuses like ratchets, axles, grabbers etc. except at such a small scale it's much more effective to make those mechanisms out of carbon nitrogen and oxygen than little metal doohickeys. It's such a failure of imagination on their part. luckily evolution isn't restricted by what ideas computer touchers find sexually arousing
Some of them acknowledge that point… but then reason that bacteria are an existence proof for nanomachines obviously a super intelligence could do even better than evolution.
No bird flies faster than a 747...so it totally possible to do "better than evolution" - given that evolution doesn't really have a goal or a purpose or even a very strong concept of "better" - just survival.
Having said that, it is hard to see how you can make nanomachines that are dramatically better than biological nanomachines because theres not that much "room" down there below what evolution has already achieved. Yes modern transistors are a couple orders of magnitude smaller than biological machines, but what they gain in efficiency they lose entirely in robustness. Biology may already be at some pareto optimal possible in the universe - where the objective function is a combination of efficiency, robustness, ease of replication, redundancy etc.
I'm actually curious if transistors are actually more efficient though. A quick google says the brain uses something in the low tens of watts, and this is a structure containing tens of billions of neurons. It's also interesting a point that was raised in the OP that bacteria are already close to thermodynamic limits of efficiency for self replication
Individual transistors are definitely more efficient - a modern transistor is something like ~1000 silicon atoms in volume. The chips we build with those transistors may not be the optimal structure though - a major reason for this is that chips are mostly 2D so extracting heat from the system can only happen in two directions, but realistically it happens in only one direction because the bottom is usually the semiconductor wafer (where the top is metallic)
However, even doing a direct comparison to the brain doesn't really make sense. So the brain runs at 10s of watts with billions of neurons, but how does that compare to, say, a modern GPU running at 1kW? We can't really compare the FLOPs directly, the compute is very different. Some people estimate the "clock" frequency of the brain, but I think those estimates are just bogus science.
There are around 600 trillion synapses though, which roughly correspond to parameters, which would equate to something like 2.4 exabytes of parameter data in a neural network using 32 bit floats, and several orders of magnitude larger than even M6 from alibaba.
The article alludes to this, but the absurd thing about “grey goo” is
that nature has already had millions of years and an astounding number
of iterations on single-celled organisms. There is an estimated five
nonillion bacteria in the world, which are reproducing
constantly. To say cells are well-optimized would be a massive
understatement. An AGI would not be able to invent something
tremendously better instantly - and that’s assuming it could even get
anything produced in the real world to begin with.
I like the comment suggesting that Drexler’s 30-year old book,
positing the creation of nanomagical machines that never came to
fruition, is a sound rebuttal. It’s like reading some book
on DNA computing from the 90s and using it to critique someone in
the modern day pointing out that normal computers are far better than
DNA computers for all our normal computational tasks.
As far as the comments, there’s a lot of bizarre assertions, and
generally while this is all interesting it doesn’t seem too different
from the Drexler vs. Whitesides debate from decades ago, and the lack of
any real progress in molecular manufacturing in the intervening time
period is a major critique of the concept. It’s not like fusion where
there has been real progress, it’s that nobody still has any idea how to
solve these basic physical limitations.
Two other bits of crazy is the idea that simply creating 4-base
codons necessarily makes a lifeform immune from all existing viruses (it
doesn’t, many existing viruses can produce their own polymerases and
only rely on tRNA) or that we just need to create acomputer
inside the cell. Nevermind that this isn’t even theoretically
complicated, we have examples of biologically-created magnetic
nanoparticles and force-activated signal
transduction, but then there’s no clear description of what you do
with this newfound power. The wireless
transmission of genetic information sounds spiffy but again, why do
you need that over more normal methods to destroy humanity?
The most reasonable people in that thread fall back on the idea that
actually normal superplagues would be the most plausible risk factor,
and then you combine some sort of superplague with an AI takeover of
autonomous military technology to finish the job and I guess we’re
ultimately back at Skynet.
And the comments about it on Lesswrong in case you want to see all the but ahkshuallies and second guessing real expertise: https://www.lesswrong.com/posts/FijbeqdovkgAusGgz/grey-goo-is-unlikely#comments
Eliezer’s default scenario for how the AI bootstraps its vast intelligence into actual influence and agency in the real world is nanotech. Fortunately for him and the rest of the world, doing better than real biology given the resources available in the real world is actually really hard so a grey goo is basically impossible.
TLDR:
Some choice sneers:
Shoots fired at one of Lesswrong’s idols:
And a jab at Drexler at Smalley:
The article alludes to this, but the absurd thing about “grey goo” is that nature has already had millions of years and an astounding number of iterations on single-celled organisms. There is an estimated five nonillion bacteria in the world, which are reproducing constantly. To say cells are well-optimized would be a massive understatement. An AGI would not be able to invent something tremendously better instantly - and that’s assuming it could even get anything produced in the real world to begin with.
I like the comment suggesting that Drexler’s 30-year old book, positing the creation of nanomagical machines that never came to fruition, is a sound rebuttal. It’s like reading some book on DNA computing from the 90s and using it to critique someone in the modern day pointing out that normal computers are far better than DNA computers for all our normal computational tasks.
As far as the comments, there’s a lot of bizarre assertions, and generally while this is all interesting it doesn’t seem too different from the Drexler vs. Whitesides debate from decades ago, and the lack of any real progress in molecular manufacturing in the intervening time period is a major critique of the concept. It’s not like fusion where there has been real progress, it’s that nobody still has any idea how to solve these basic physical limitations.
Two other bits of crazy is the idea that simply creating 4-base codons necessarily makes a lifeform immune from all existing viruses (it doesn’t, many existing viruses can produce their own polymerases and only rely on tRNA) or that we just need to create a computer inside the cell. Nevermind that this isn’t even theoretically complicated, we have examples of biologically-created magnetic nanoparticles and force-activated signal transduction, but then there’s no clear description of what you do with this newfound power. The wireless transmission of genetic information sounds spiffy but again, why do you need that over more normal methods to destroy humanity?
The most reasonable people in that thread fall back on the idea that actually normal superplagues would be the most plausible risk factor, and then you combine some sort of superplague with an AI takeover of autonomous military technology to finish the job and I guess we’re ultimately back at Skynet.
Raiden:“Why won’t you shut the fuck up? None of your arguments make any sense!”
Yud: “NANOMACHINES, SON”