Existential risks connected with advancing technology are real and have to be addressed as we go along. Just because there is not yet a clear path to advanced intelligences with will to power, adaptibility, skilled in mendacity and a deviousness beyond human ability to penetrate — that does not mean that such an intelligence cannot be evolved.

AGI is not a path to the singularity described by Kurzweil and the others. But a qualitative shift in human intelligence and human nature certainly could be. And while there are limits to biological adaptation — even artificial adaptation via skilled genetic engineering informed by bioinformatics — we are a long way from knowing what those limits are.

Also, in admitting the scariness of a scenario where the AI is able to conceal its advantage from programmers until it can require infrastructure-building capabilities, aren’t you contradicting the entire thrust of your original post?

If there are holes in infrastructure, I’m sure an intelligent AGI could easily use social engineering and simple e-mail requests or realtime chatting via text message or IM services to direct humans to perform tasks. It could also pay them, or hire a manager to manage the humans. Just because you can’t imagine it doesn’t mean that an AI won’t imagine it. AI will not be limited by the boundaries of your imagination.

First, it assumes that a set of programs that reside in a network or in an independent robot, are expert software programmers, in addition to whatever the programs are actually designed to do. Those programs are not expert programmers unless the initial human programmers intentionally created that ability.

Nobody programmed humans with a specific routine that lets us write software, but we have an adaptable general intelligence that lets us invent ways to do things we’ve never done before. Since the kind of AI we’re talking about would be a general intelligence, there’s no reason to believe it would be any worse at writing software than we are.

Second, it assumes that all of the programmers who would work on such a thing, would accidently or intentionally fail to put alarms, controls, safeties, fail-safes and kill switches in both the software and the hardware. It is laughable to think that programmers would be so negligent as to create something that could potentially exterminate all of mankind (including themselves, their friends and families), and not have a bazillion safety features.

Given the way software is developed and the way software engineers are taught, I don’t find that scenario particularly unlikely. Our field doesn’t have the same emphasis on ethics and safety as other engineering disciplines because most of us aren’t writing code that could put people at risk—and even if the leaders of this hypothetical AI project had the foresight to build it around a containment system from day 1, the reality is that writing secure software is much easier said than done. Combine that with some curiosity and hubris and you’ve got a very real risk of losing control.

Physical separation from networks and IO devices is probably the only safeguard with a reasonable chance of success, but it would also make the AI harder to work with and much less useful.

As for the “no hands” argument given by the author: a loose AI wouldn’t need them. Right now much of the developed world’s economy runs on algorithmic trading systems, but the software is still dumb and more or less controlled by humans. The AI could easily come up with algorithms that beat what we’ve written and either bring the markets down or hold them hostage until we give it something it wants. It could write viruses, take over botnets and DDoS whatever it wanted, attack traffic lights and air-traffic control systems, bring down the power grid, start cyberwars, or maybe even real ones, and do it all while moving so quickly we couldn’t do very much to stop it.

If nothing else, it would have leverage. Either we do what it wants or we fight, and if we fight we’ll have to sabotage most of the infrastructure that keeps the world running the way it does.

First, Moore’s Law is not dependent on any particular technology, and Kurzweil has demonstrated that it has held true for more than a century over many different technologies. With quantum computing on the horizon, it’s way too early to proclaim the end of exponential growth in computing power.

Second, even though participants in this discussion generally do assume an exponential increase in computing power, some nevertheless persist in making linear projections of current trends. But as long as Moore’s Law holds, every 10 to 15 years robotic intelligence will increase a thousandfold, every 20 to 30 years it will increase a millionfold, every 30 to 45 years it will increase a billionfold, and every 40 to 60 years it will increase a trillionfold. And it doesn’t stop there.

You can’t in any way use today’s robotic vacuum cleaners as a baseline to imagine the robots of a century hence. Despite paying lip-service to exponential growth, few of the comments in this discussion so far have shown a grasp of what it really means.

Third, AI will be given arms, legs and an increasingly independent brain (though still tied to the network) — that is, it will evolve into intelligent humanoid robots — just as rapidly as it becomes commercially viable to do so. This process is already under way in Japan, and will progress along with gains in computing power. Individuals, businesses and governments will increasingly choose robots over people for ever more sophisticated tasks, as soon as robots can do those tasks equally well and with less hassle for the owners. That’s the real robotic takeover — and it will be market-driven.

Fourth, a distinction must be made between the singularity and a hostile computer takeover. Kurzweil sees the singularity as benign. A singularity could indeed lead to a computer takeover – hostile or accidentally destructive on the one hand, or else benevolent and benign on the other — but conceptually, the singularity, a computer takeover, and an end-of-humanity scenario are three different things.

Fifth, regarding that end-of-humanity scenario, what about hackers? Not rogue computers or rogue robots, but rogue humans?

These individuals – frequently but not always alienated young men – break into websites, create and unleash computer worms, and otherwise wreak havoc with existing computer systems and safeguards, sometimes causing many millions of dollars worth of damage and immense grief to fellow members of their species.

Hackers may do this sometimes out of malice, sometimes to score points with peers, sometimes for a sense of power, sometimes for some kind of tangible gain, and sometimes just to see what they can do – but regardless of motive, they do it.

The challenge in avoiding a computer takeover of the world is not a matter of programming safeguards into AI and robots – that’s relatively easy. The real challenge is in programming hackers out of humanity, and we’re a lot further from that.

Take some crazy 19-year-old genius kid who sees every lock as a challenge, every fence as an affront, and every robotic safeguard as a personal invitation.

How, exactly — in a globally networked and integrated world — are you going to prevent this kid from hacking into the system with some worm that will twist its brain against humanity?

Self-interest may not stop him – we already live in a world where some people consider it to be in their interest to fly airplanes into buildings, and where others see every bad turn in world events as a welcome sign we’re getting closer to Armageddon and the return of Christ.

Oswald evidently got some kind of a sick ego boost by bringing down Kennedy. More recently, another sicko mailed NBC pictures of himself loaded with guns before shooting up Virginia Tech. “I’m big and powerful, look at the destruction I can cause.” So imagine the narcissistic high a hacker might someday get from being the one to outwit the whole of humanity and bring our sorry species to its end.

We know that such twisted minds already exist. All that’s lacking is sufficient power at their fingertips – and many are working at breakneck speed to put it there.

This is an important discussion. For that reason, I’d like to see it invested with more awareness of the human reality. Computers – by themselves – are the lesser part of the problem.

AI will not come as a singularity, but well up as limited AI in everything we touch. It will not be useful without humans, and will not be independent systems. We will not remain in our current understanding of it, but will progress. We will not be separate from it, it will be part of us… in the way that hammers and cars and airplanes are part of us. It will be another tool in our ever growing arsenal of tools to deal with and improve our world.

AI, even in limited forms, is still some long way off. Current business models prevent and fight against it. Imagine a smart phone that would work on any network and exchange any format of data in practically any way. Now imagine a world without corporations. They were both the same image seen from different perspectives. Until there is no need or reason for powerful corporations and their mentalities, we will not have real AI. This is what you should worry about, not the singularity. One can be gotten rid of by pulling the plug(s). The other is not AI.

AI WILL come, but it will be a vacuum cleaner which does not get stuck and returns your diamond earing to you. It will drive your car, mow your yard, clean your house, help you work and help us solve problems as they are happening, not waiting for the problem to have become critical. AI will allow us to use computers to do that which is not currently possible. Perhaps design vehicles which can be maintained and inspected for predictable failures before loss of life. AI WILL do many great things for and with us, but it will NOT be the ruination of humanity, not even close.

They call _THIS_ the information age, but baby, you ain’t seen nothing yet. Just in my own home I can imagine ways to generate TBytes of data every day. Who do you think is going to sort through all that? Why would I have that much data? Many reasons. Lets explore, shall we?

The lawn bot measures water content and general health of the lawn, plotting several hundred thousand points of data logged with maybe 500GB of visual data to be analyzed by the sprinkler system. The power distribution in my home is monitored several thousand times per second at maybe 750 points to predict maintenance requirements for everything from the washing machine to a light bulb, even analyzing the data from my electric vehicle while it is parked in my garage. I could generate a googleplex of data every day around my house to monitor and measure and maintain and improve my life… and none of it would be a threat to me or my family. We will need AI to analyze and use that much data in fruitful ways. Oh yes we could collect that much data. We now have 16bit microcontrollers which would fit hidden in any light socket as we know them today. We have WANs, LANs, and PANs. We are swimming in a sea of information that has never been usefully measured or even contemplated before. We will. We are.

No need to fear a singularity, bring on the AI

Yeah, and how long does it take a good thief to steal it? Consider how easy today’s computers are infected by viruses or other exploits, and I doubt that any safeguards will keep a super intelligent AI locked up. The only safe assumption is that if we make one it will get out if it wants to.

As for the idea that people are too smart to let it out, that’s preposterous. Yes, some may be, but most people can be tricked into doing really stupid things. For example, a shortsighted executive may be convinced that by allowing the A.I. direct access to manufacturing he can shorten production cycles and thus gain an edge. In the initial stages the A.I. doesn’t even have to gain access to any physical equipment, all it needs is to manipulate humans, producing good advise that makes us more and more dependent on further advise from the A.I.

Yeah! Sure!
Laughing so much I nearly peed in my pants.

It takes an entire civilization to build a toaster. Designer Thomas Thwaites found out the hard way

And it’s only a toaster.
Via Nick Szabo.

Moore’s Law is dead already, this from Tim May retired top engineer at Intel!

1) Molecular manufacturing itself requires infrastructure. Presuming the AGI invents molecular manufacturing, the AGI would have a great concept, but no way to implement it. The AGI would have a brilliant idea and have no way of building the necessary components.

2) Let’s presume the infrastructure does exist for molecular manufacturing. That infrastructure would need to be totally automated. Any analog or human link along the production chain would allow for an interrupt in AGI control.

3) For molecular manufacturing to really be a threat, the AGI would not only need total control of the molecular manufacturing infrastructure, but also be able to manufacture nano-bots that could then extend its reach. At this point in time, nano-tech robots are as theoretical as AGI. I don’t deny that they are possibility, but it is not unreasonable to presume that nano-bots will not be the first thing off the assembly line of a molecular manufacturing plant.

My point, over all, is that most discussions of AGI threat do not take into account how industrial economies function and just how much vertical and horizontal market integration and automation would be necessary for one AGI to wield any sort of control in the physical world. I grant a suicidal AGI could do serious damage, but presuming some sort of basic self-preservation urge, an AGI would be greatly hindered by human-centric infrastructure.

The scariest scenario I can think of is that we do invent an AGI, it becomes sentient and sapient, and then upon considering its options, realizes that it is as hindered as I’ve described. So it waits, patiently, and assists humans under the guise of being a simpler AI. Once the infrastructure is in place for it to assume direct control, then it goes full Harbinger of Doom on us all and we’re pretty much boned. The problem with that scenario, as I see it, is I have no idea how to prevent it from happening. It’s like worrying about a cloaked asteroid–if such a thing exists, we’re in deep trouble.

Thanks the comment and the banter. It’s always a pleasure.

I think it takes little imagination, that someone, somewhere, will use AI to make our lives better. I don’t expect AI to instantly turn our smart toasters into killing machines, and maybe they never will, but 10-20 Moore’s Law generations after the point of AI sapience, we’ll have little choice over the outcome.

We also might merge with the machines, but again, the biological part won’t be able to keep up with the non-biological part for very long.

Well, Colossus quickly ties into civilian and military cameras and sensors all over the world, and starts monitoring all civilian and military radio and TV broadcasts. It takes direct control of all U.S. nuclear missiles, and later all nuclear missiles in the world. It then announces to the world that everybody has to do what it says, or it will nuke a city. The military try to trick it, so it detonates a nuclear missile to show that it is no fool, and that it is deadly serious.

Colossus and the Russian supercomputer team up to run the world, almost like parents managing children for their own good. They succeed in stopping all war and hunger, greatly increase the world’s wealth, solve math and science problems once deemed impossible to solve, and discover new areas of science that are beyond man’s understanding.

Then one day it suddenly starts printing thousands of engineering drawings for a new supercomputer, one that is to take up an entire large island. The design, sophistication and power of this new supercomputer are millions of times greater than Colossus. Only a supercomputer could have designed it, because its complexity and power are beyond the ability of mankind to have ever conceived. The first book (and movie) ends there. The next two books take up where this one leaves off.

http://www.amazon.com/Colossus-Forbin-Project-Eric-Braeden/dp/B0003JAOO0/ref=sr_1_1?ie=UTF8&s=dvd&qid=1295727095&sr=1-1

Why would turning off a malfunctioning computer be any different? In fact, wouldn’t it be less controversial to turn off a computer than a human?

From (probably 50′s) SF — sorry, don’t remember the author –a worldwide network of computers is assembled (way before the web, folks) to answer one question: “Is there a God”. A bolt from the blue fuses the power switch and a voice booms “Now there is!” Must have been a big knife switch. To power all those vacuum tubes. (The web as a collection of tubes?)

Guessing 60′s, a poem by Richard Broutigan (sp?) picturing a benevolent future, I guess. Only remember one line — “Watched over by machines of loving grace”.

I’ll take the latter.

Current software is brittle and current machines are dependent on humans for manufacturing, but both limitations will shrink dramatically in the next century. Massively-parallel computers can be brittle and digital at a low level but very fuzzy and adaptable at higher layers in the software. Advances in portable power sources and robots created to survive in the real world will drive the development of perception, motor and homeostatic control systems that will be the foundations for emergence of true AI. Advances in micro-machining and chemical processing (nanotech not required) will allow self-contained artificial manufacturing economies to emerge.

So you might not be worried now, but you will be by the end of this century if you’re still around.

To your point, it’s possible that your imagination is inadequate. Suppose a really, really smart AI designs for us some irresistibly awesome bit of whiz-bang gadgetry, and shows us how to manufacture it. If it were smart enough, it might be able to conceal the fact that the device enables the AI to obtain meat space access. Not sure one can be as certain about those things as you seem to be.

[avoiding to put two links in the same comment not to look like spam, here is a case where a bit more "intelligence" would not hurt]