Existential risks connected with advancing technology are real and have to be addressed as we go along. Just because there is not yet a clear path to advanced intelligences with will to power, adaptibility, skilled in mendacity and a deviousness beyond human ability to penetrate — that does not mean that such an intelligence cannot be evolved.

AGI is not a path to the singularity described by Kurzweil and the others. But a qualitative shift in human intelligence and human nature certainly could be. And while there are limits to biological adaptation — even artificial adaptation via skilled genetic engineering informed by bioinformatics — we are a long way from knowing what those limits are.

Also, in admitting the scariness of a scenario where the AI is able to conceal its advantage from programmers until it can require infrastructure-building capabilities, aren’t you contradicting the entire thrust of your original post?

If there are holes in infrastructure, I’m sure an intelligent AGI could easily use social engineering and simple e-mail requests or realtime chatting via text message or IM services to direct humans to perform tasks. It could also pay them, or hire a manager to manage the humans. Just because you can’t imagine it doesn’t mean that an AI won’t imagine it. AI will not be limited by the boundaries of your imagination.

First, it assumes that a set of programs that reside in a network or in an independent robot, are expert software programmers, in addition to whatever the programs are actually designed to do. Those programs are not expert programmers unless the initial human programmers intentionally created that ability.

Nobody programmed humans with a specific routine that lets us write software, but we have an adaptable general intelligence that lets us invent ways to do things we’ve never done before. Since the kind of AI we’re talking about would be a general intelligence, there’s no reason to believe it would be any worse at writing software than we are.

Second, it assumes that all of the programmers who would work on such a thing, would accidently or intentionally fail to put alarms, controls, safeties, fail-safes and kill switches in both the software and the hardware. It is laughable to think that programmers would be so negligent as to create something that could potentially exterminate all of mankind (including themselves, their friends and families), and not have a bazillion safety features.

Given the way software is developed and the way software engineers are taught, I don’t find that scenario particularly unlikely. Our field doesn’t have the same emphasis on ethics and safety as other engineering disciplines because most of us aren’t writing code that could put people at risk—and even if the leaders of this hypothetical AI project had the foresight to build it around a containment system from day 1, the reality is that writing secure software is much easier said than done. Combine that with some curiosity and hubris and you’ve got a very real risk of losing control.

Physical separation from networks and IO devices is probably the only safeguard with a reasonable chance of success, but it would also make the AI harder to work with and much less useful.

As for the “no hands” argument given by the author: a loose AI wouldn’t need them. Right now much of the developed world’s economy runs on algorithmic trading systems, but the software is still dumb and more or less controlled by humans. The AI could easily come up with algorithms that beat what we’ve written and either bring the markets down or hold them hostage until we give it something it wants. It could write viruses, take over botnets and DDoS whatever it wanted, attack traffic lights and air-traffic control systems, bring down the power grid, start cyberwars, or maybe even real ones, and do it all while moving so quickly we couldn’t do very much to stop it.

If nothing else, it would have leverage. Either we do what it wants or we fight, and if we fight we’ll have to sabotage most of the infrastructure that keeps the world running the way it does.

First, Moore’s Law is not dependent on any particular technology, and Kurzweil has demonstrated that it has held true for more than a century over many different technologies. With quantum computing on the horizon, it’s way too early to proclaim the end of exponential growth in computing power.

Second, even though participants in this discussion generally do assume an exponential increase in computing power, some nevertheless persist in making linear projections of current trends. But as long as Moore’s Law holds, every 10 to 15 years robotic intelligence will increase a thousandfold, every 20 to 30 years it will increase a millionfold, every 30 to 45 years it will increase a billionfold, and every 40 to 60 years it will increase a trillionfold. And it doesn’t stop there.

You can’t in any way use today’s robotic vacuum cleaners as a baseline to imagine the robots of a century hence. Despite paying lip-service to exponential growth, few of the comments in this discussion so far have shown a grasp of what it really means.

Third, AI will be given arms, legs and an increasingly independent brain (though still tied to the network) — that is, it will evolve into intelligent humanoid robots — just as rapidly as it becomes commercially viable to do so. This process is already under way in Japan, and will progress along with gains in computing power. Individuals, businesses and governments will increasingly choose robots over people for ever more sophisticated tasks, as soon as robots can do those tasks equally well and with less hassle for the owners. That’s the real robotic takeover — and it will be market-driven.

Fourth, a distinction must be made between the singularity and a hostile computer takeover. Kurzweil sees the singularity as benign. A singularity could indeed lead to a computer takeover – hostile or accidentally destructive on the one hand, or else benevolent and benign on the other — but conceptually, the singularity, a computer takeover, and an end-of-humanity scenario are three different things.

Fifth, regarding that end-of-humanity scenario, what about hackers? Not rogue computers or rogue robots, but rogue humans?

These individuals – frequently but not always alienated young men – break into websites, create and unleash computer worms, and otherwise wreak havoc with existing computer systems and safeguards, sometimes causing many millions of dollars worth of damage and immense grief to fellow members of their species.

Hackers may do this sometimes out of malice, sometimes to score points with peers, sometimes for a sense of power, sometimes for some kind of tangible gain, and sometimes just to see what they can do – but regardless of motive, they do it.

The challenge in avoiding a computer takeover of the world is not a matter of programming safeguards into AI and robots – that’s relatively easy. The real challenge is in programming hackers out of humanity, and we’re a lot further from that.

Take some crazy 19-year-old genius kid who sees every lock as a challenge, every fence as an affront, and every robotic safeguard as a personal invitation.

How, exactly — in a globally networked and integrated world — are you going to prevent this kid from hacking into the system with some worm that will twist its brain against humanity?

Self-interest may not stop him – we already live in a world where some people consider it to be in their interest to fly airplanes into buildings, and where others see every bad turn in world events as a welcome sign we’re getting closer to Armageddon and the return of Christ.

Oswald evidently got some kind of a sick ego boost by bringing down Kennedy. More recently, another sicko mailed NBC pictures of himself loaded with guns before shooting up Virginia Tech. “I’m big and powerful, look at the destruction I can cause.” So imagine the narcissistic high a hacker might someday get from being the one to outwit the whole of humanity and bring our sorry species to its end.

We know that such twisted minds already exist. All that’s lacking is sufficient power at their fingertips – and many are working at breakneck speed to put it there.

This is an important discussion. For that reason, I’d like to see it invested with more awareness of the human reality. Computers – by themselves – are the lesser part of the problem.