My last post about the book, If Anyone Builds It Everyone Dies: Why Superhuman AI Would Kill Us, elicited some particularly thoughtful comments. I would like to draw my readers’ attention to them. C. Rosenkreuz wrote,
I suspect that the authors of this book, Yudkowsky and Soares, are not being fully intellectually honest with their readers.
I feel very confident in saying that Yudkowsky and Soares, being really smart and well-educated guys, fully understand there will never be any cessation or reduction of the AI arms race now underway between the wealthy and technologically advanced nation-states. In the late 1940s and early 1950s, there were sincere efforts, led by Albert Einstein, Bertrand Russell, and other world thought leaders of the humanist variety, to stop the nuclear arms race, but those efforts all came to nothing.
The race to build superintelligent machines will rage on, since, in the short run, the government that first achieves that will be able to dominate all the other nations and will be able to defend itself from being so dominated. But a short time after that, humans will probably lose control of the superintelligent machines, and then there will be a new de facto life form (non-biological) that will be dominant on the earth.
In the end, the most thoughtful humans will realize that superintelligent machines evolved naturally from homo sapiens, by the natural processes that Charles Darwin described in 1859. Many thinkers are realizing that Darwin’s model of natural selection is already operating within AI machines of the LLM variety.
This article hints at this when it quotes Wikipedia as saying this: “An AI that tries to achieve goals will perform better on many metrics, so it WILL BE SELECTED FOR by the training process.” The last lines of Darwin’s famous 1859 book “ORIGIN” capture well what is likely soon to be happening about AI (except thinking in terms of “higher minds” instead of “higher animals”):
“Thus, from the war of nature, from famine and death, the most exalted object which we are capable of conceiving, namely, the production of the higher animals, directly follows. There is grandeur in this view of life, with its several powers, having been originally breathed into a few forms or into one; and that, whilst this planet has gone circling on according to the fixed law of gravity, from so simple a beginning endless forms most beautiful and most wonderful have been, and are being evolved.”
From the passage above, we can see that Darwin viewed the laws of biology (e.g., the law of natural selection, the law of the struggle for existence, etc.) as inescapable, insuppressible laws of nature just like the laws of physics. Whether, as the transhumanists say, “there is grandeur in this view of life” without homo sapiens on the earth, is debatable. I, for one, do not see the grandeur in that.
John Messerly responded,
I basically agree that the genie is out of the bottle. I argued as much against Bill Joy‘s claim in his famous piece “Why The Future Doesn’t Need Us” that we could and should stop AI. However, I would disagree that limits on nuclear arms were unsuccessful. They have been at least somewhat successful given nuclear arms treaties, bans on above-ground tests, etc. And the authors aren’t saying we will be successful in controlling AI; in fact, they doubt it. But they are saying we should try to and that it is at least possible. As to whether these children of our minds will do better than we have is an open question.
Lyle T commented,
Two main points that I took away from your presentation.
1. The nature of modern machine learning does not allow humans to specify the goals that a superintelligent AI system should pursue, and 2. ….AI system becomes threatening….no one can look inside and ‘fix it’ because THERE IS NO CODE, only illegible numerical parameters.
Until reading this, I did not realize this feature of AI. I had been concerned with the immoral or amoral use of AI capabilities. I now understand (I think I understand) that advanced AI itself might be what determines morality and strategic objectives.
Then it would no longer be the stuff of science fiction.
Some stray comments from accumulated notes.
“If it should turn out to be true that knowledge (in the modern sense of know-how) and thought have parted company for good, then we would indeed become the helpless slaves, not so much of our machines as of our know-how, thoughtless creatures at the mercy of every gadget which is technically possible, no matter how murderous it is.”
Hannah Arendt’s prologue of “The Human Condition” also states, “Man has been simultaneously reduced to a biological machine of no lasting worth, while elevated to the position of a god-capable of destroying worlds.”
While I still consider nuclear weapons the number one short-term threat to humanity, it certainly seems AI will possibly overtake them. In reality, we live in an era of poly-crisis that is also a perma-crisis. With each providing feedback on other potential catastrophes.
example:
”global warming threatens famine and unsurvivable temperatures–attempts to limit ‘pollution causing economic activity’ creates joblessness–which combined with famine leads to migrations, which lead to the rise of strong-man nationalist governments–which leads to wars–which leads to some AI influenced military launching a nuclear pre-emptive strike…….”
Perhaps advanced industrial civilization will crash before either AI or nuclear weapons drive us to extinction. It takes an advanced industrial society for AI or nukes. So, ironically, the very collapse of civilization could be what saves Homo Sapiens from extinction. (At a degree of death and suffering that I believe is unimaginable). Or
“Perhaps superior intelligence will prove to be an evolutionary dead-end.”
Meanwhile, here in the Northern Hemisphere, spring is here. The tomato seedlings have sprouted. I see robins every day now, and hear red-winged blackbirds by the creeks and ponds. The beat goes on.
(I agree with Lyle that we must try to go on and enjoy the good things in life that are available to us.)
Chris Crawford commented,
I cannot imagine how AI systems could lead to the extinction of Homo Sapiens. Such systems could certainly destroy the Internet or take over every personal computer and smartphone on the planet, which would likely devastate our economies, leading us to all start shooting at each other. But when the dust settled, there would still be a few million humans, living as hunter-gatherers. We’d revert to our “natural” role — that is, the role that we evolved to play, as hunter-gatherers.
Given the huge supply chain required to sustain computer systems, I cannot imagine how the computers would survive for long. Sure, they could automate most factories, but they’d need to automate every single industrial process, and the odds are high that they wouldn’t be able to replicate at least one process. And why wouldn’t they fight each other for resources? Self-destructive wars are not necessarily a human monopoly
I replied,
Everything you say seems reasonable, but as best as I could tell from reading the book, the authors disagree. I’m just not enough of an expert to have an opinion. But I would add that “Geoffrey Hinton, known as the ‘godfather of AI,” warns that superintelligent AI poses an existential risk to humanity, estimating a 10% to 20% chance it could wipe us out. After leaving Google to speak freely, Hinton highlighted dangers including AI outsmarting humans, creating uncontrollable misinformation, and massive job displacement.”
In the end, I’m concerned for my children and grandchildren, but like Lyle, I try to maintain equanimity. My sincere thanks to these readers who took the time to share their profound thoughts.
Hmm… perhaps an easy resolution of the difference between my position and that of Yudkowsky and Soares is to zero in on the meanings of an “existential threat” and “extinction”. Could we agree that AI is capable of destroying our civilization while being unable to kill every last human being?
For example, there remain a number of functioning hunter-gatherer groups, such as the Hadza (Tanzania), San (Southern Africa), Sentinelese (Andaman Islands), Nukak (Colombia), Batek (Malaysia), and the Inuit (Arctic). Moreover, there remain quite a few societies that are only partially connected to our civilization. Many of these are in the Amazon; some others are in the Philippines. There are 10,000 Sami people in northern Finland, still practicing their ancient ways.
Let us imagine the evil AI program twirling its virtual mustache and sneering, “Now I shall keel those last Sami in Finland, and then I shall rule zee world!!!!” How do you suppose it might go about the task? Deny them their cat videos? Make it impossible for them to sell reindeer hides on eBay?
I suspect that the greatest danger lies in an AI system somehow setting up a complex scheme in which it fakes the voices of important people and calls important people, using the fake voices, and convinces the leaders of the USA that China has launched a nuclear attack against it. The President orders a full-scale nuclear attack on China. Somehow, Russia and China are also convinced to launch their weapons, and we get a huge nuclear catastrophe resulting in a nuclear winter. Even then, all of the nukes would detonate north of the equator. The entire northern hemisphere would be shrouded in smoke. But the atmosphere in the Northern Hemisphere mixes only slowly with the atmosphere in the Southern Hemisphere. They’d suffer some problems, but nothing like a nuclear winter. Many people in the Southern Hemisphere would survive.
This comment from Lyle T set me thinking: “THERE IS NO CODE, only illegible numerical parameters.”
Actually, there IS some code, but it’s at the front end of the AI. The back end is where all those inscrutable numerical parameters are. A plain old everyday program processes those inscrutable numerical parameters and sends the result across the Internet as actions. It would be simple enough to write that code with inhibitions prohibiting the AI from doing nasty stuff, like so:
“No, you cannot access addresses belonging to the Department of Defense.”
“No, you cannot access addresses belonging to any foreign military group.”
Oh, wait: here’s an even better way to do it: have the UN establish a central repository for Internet addresses that are forbidden to all AI systems. Anybody can go to that repository’s website and enter the URL that they want protected from AI. All AI systems are legally required to confine their behaviors in accordance with the UN depository.
There! You see, I’ve saved the world. Now I have to go back to my 1968 Honda CB160 and see if I can’t get those damn connectors clean enough to work properly.
A very humorous ending:) But yes, I think the answer to your question “Could we agree that AI is capable of destroying our civilization while being unable to kill every last human being?” is yes. How much consolation that provides is another question. Also, I like the analogy with the hemispheres. We’ve known for years that you have the best chance of surviving a full scale nuclear war in the southern hemisphere. Perhaps living off the grid somewhere (as these people you describe do) would give us the best shot of avoiding AI apocalypse. On the other hand, maybe AI and/or an army of robotic workers will usher in paradise.
It’s NOT AI that will kill us all, it’s modern humans that do it. This reality is in plain sight of every “awake” person but…
“The masses have never thirsted after truth. They turn aside from evidence that is not to their taste, preferring to deify error, if error seduces them. Whoever can supply them with illusions is easily their master; whoever attempts to destroy their illusions is always their victim.” — Gustave Le Bon, in 1895
Everyone SHOULD of course get what AI is REALLY all about but most people CHOOSE not to want to understand it…
“Ignorance is the root cause of all Evil. Since only Knowledge eradicates ignorance, it is our duty and moral obligation to educate ourselves, as well as the masses around us.” — Anonymous
Like with every criminal inhumane self-concerned agenda of theirs the psychopaths-in-control sell and propagandize AI to the “awake” public with total lies such as AI being the benign means to connect, unit, transform, benefit, and save humanity.
The 2 major OFFICIAL deceptive fake FEAR-MONGERING narratives or phony pretexts (ie, lies, propaganda) nearly everyone, including “alternative news” sources, have been spreading is (1) that the TRULY big threat is that AI just creates utter chaos in society and that it might achieve control over humans (therefore it must be regulated, ie monopolized by the typical criminal governments); and (2) that we, the US, have to invest heavily in AI technological development so as to stay ahead of other nations, such as China
The TRUE narrative (ie empirical reality) virtually no one talks about or spreads is that the TRULY big threat with AI is that AI allows the governing psychopaths-in-power to materialize their ultimate wet dream to control and enslave everyone and everything on the whole planet, a process that’s long been ongoing in front of everyone’s “awake” nose
The proof is in the pudding… ask yourself, “how is the hacking of the planet going so far? Has it increased or crushed personal freedom?”
“AI responds according to the “rules” created by the programmers who are in turn owned by the people who pay their salaries. This is precisely why Globalists want an AI controlled society- rules for serfs, exceptions for the aristocracy.” —Unknown
“Almost all AI systems today learn when and what their human designers or users want.” —Ali Minai, Ph.D., American Professor of Computer Science, 2023
“Who masters those technologies [=artificial intelligence (AI), chatbots, and digital identities] —in some way— will be the master of the world.” — Klaus Schwab, at the World Government Summit in Dubai, 2023
“[…] what we are being sold through all this AI hype is that everything should be reducible to categories and hierarchies, put in the form of measures and statistics for prediction. However, if you understand that counting, measuring and predicting are always situated, and therefore subjective and political, you might not be as excited about such AI-determined futures. […]. The bigger AI project asks that you abandon meaning and feeling, to believe in a neutral, objective, all-knowing source built from massively large datasets hosted in hyperscale data centres. AI asks that you buy into the idea that more data means being closer to The Truth. It means that you should want to offload your limited cognition to the super machine. And perhaps most dangerously — that you understand all of this as a scientific endeavour.” — Mél Hogan, Associate Professor in Film & Media, 2026
“COVID is critical because this is what convinces people to accept, to legitimize, total biometric surveillance.” — Yuval Noah Harari, member of the dictatorial ruling mafia of psychopaths, World Economic Forum [https://archive.md/vrZGf]
“The whole idea that humans have this soul, or spirit, or free will … that’s over.” — Yuval Noah Harari, member of the dictatorial ruling mafia of psychopaths, World Economic Forum [hhttps://archive.md/vrZGf
“It looks to me like AI has been tasked with creating literally millions of fake videos tailored for every possible interest, to snare you in an endless loop, waste as much of your time as possible, stir your brain, create fear and unease, and make it impossible for you to tell reality from fiction. Why? TO KEEP YOU FROM STARTING THE REVOLUTION.” — Miles Mathis, American author, 2026