(and lethal) technologies, like cars, are for all intents and purposes irreversible due to
social and economic forces. Within genetics, a red line makes us ban or avoid genetically
modified foods but embrace genetically modified bacteria making insulin, or genetically
modified humans—witness mitochondrial therapies approved in Europe for human adults
and embryos.
The line for germline manipulation seems less sensible than the usual, practical
line drawn at safety and efficacy. Marriages of two healthy carriers of the same genetic
disease have a choice between no child of their own, 25-percent loss of embryos via
abortion (spontaneous or induced), 80-percent loss via in-vitro fertilization, or potential
zero-percent embryo loss via sperm (germline) engineering. It seems premature to
declare this last option unlikely.
171
For “human subject research,” we refer to the 1964 Declaration of Helsinki,
keeping in mind the 1932-1972 Tuskegee syphilis experiment, possibly the most
infamous biomedical research study in U.S. history. In 2015, the Nonhuman Rights
Project filed a lawsuit with the New York State Supreme Court on behalf of two
chimpanzees kept for research by Stony Brook University. The appellate court decision
was that chimps are not to be treated as legal persons since they “do not have duties and
responsibilities in society,” despite Jane Goodall’s and others’ claim that they do, and
despite arguments that such a decision could be applied to children and the disabled. 46
What prevents extension to other animals, organoids, machines, and hybrids? As
we (e.g., Hawking, Musk, Tallinn, Wilczek, Tegmark) have promoted bans on
“autonomous weapons,” we have demonized one type of “dumb” machine, while other
machines—for instance, those composed of many Homo sapiens voting—can be more
lethal and more misguided.
Do transhumans roam the Earth already? Consider the “uncontacted peoples,”
such as the Sentinelese and Andamanese of India, the Korowai of Indonesia, the Mashco-
Piro of Peru, the Pintupi of Australia, the Surma of Ethiopia, the Ruc of Vietnam, the
Ayoreo-Totobiegosode of Paraguay, the Himba of Namibia, and dozens of tribes in
Papua New Guinea. How would they or our ancestors respond? We could define
“transhuman” as people and culture not comprehensible to humans living in a modern,
yet un-technological culture.
Such modern Stone Age people would have great trouble understanding why we
celebrate the recent LIGO gravity-wave evidence supporting the hundred-year-old
general theory of relativity. They would scratch their heads as to why we have atomic
clocks, or GPS satellites so we can find our way home, or why and how we have
expanded our vision from a narrow optical band to the full spectrum from radio to
gamma. We can move faster than any other living species; indeed, we can reach escape
velocity from Earth and survive in the very cold vacuum of space.
If those characteristics (and hundreds more) don’t constitute transhumanism, then
what would? If we feel that the judge of transhumanism should not be fully paleo-culture
humans but recent humans, then how would we ever reach transhuman status? We
“recent humans” may always be capable of comprehending each new technological
increment—never adequately surprised to declare arrival at a (moving) transhuman
target. The science-fiction prophet William Gibson said, “The future is already here—
it’s just not very evenly distributed.” While this underestimates the next round of
“future,” certainly millions of us are transhuman already—with most of us asking for
more. The question “What was a human?” has already transmogrified into “What were
the many kinds of transhumans?. . . And what were their rights?”
46
https://www.nbcnews.com/news/us-news/lawyer-denying-chimpanzees-rights-could-backfire-disabled-
n734566.
172
Caroline A. Jones’ interest in modern and contemporary art is enriched by a willingness
to delve into the technologies involved in its production, distribution, and reception. “As
an art historian, a lot of my questions are about what kind of art we can make, what kind
of thought we can make, what kind of ideas we can make that could stretch the human
beyond our stubborn, selfish, ‘only concerned with our small group’ parameters. The
philosophers and philosophies I’m drawn to are those that question the Western
obsession with individualism. Those are coming from so many different places, and
they’re reviving so many different kinds of questions and problems that were raised in the
1960s.”
She has recently turned her attention to the history of cybernetics. Her MIT
course, “Automata, Automatism, Systems, Cybernetics,” explores the history of the
human/machine interface in terms of feedback, exploring the cultural rather than
engineering uptake of this idea. She begins with primary readings by Wiener, Shannon,
and Turing and then pivots from the scientists and engineers to the work and ideas of
artists, feminists, postmodern theorists. Her goal: to come up with a new central
paradigm of evolution that’s culture-based—“communalism and interspecies symbiosis
rather than survival of the fittest.”
As a historian, Caroline draws a distinction between what she has termed “left
cybernetics” and “right cybernetics”: “What do I mean by left cybernetics? In one
sense, it’s a pun or a joke: the cybernetics that was ‘left’ behind. On another level, it’s a
vague political grouping connoting our Left Coast: California, Esalen, the group that
Dave Kaiser calls the ‘hippie physicists.’ It’s not an adequate term, but it’s a way of
recognizing that there was a group beholden to the military-industrial complex,
sometimes very unhappily, who gave us the tools to critique it.”
173
THE ARTISTIC USE OF CYBERNETIC BEINGS
Caroline A. Jones
Caroline A. Jones is a professor of art history in the Department of Architecture at MIT
and author of Eyesight Alone: Clement Greenberg’s Modernism and the
Bureaucratization of the Senses; Machine in the Studio: Constructing the Postwar
American Artist; and The Global Work of Art.
Cybernated art is very important, but art for cybernated life is more important.
— Nam June Paik, 1966
Artificial intelligence was not what artists first wanted out of cybernetics, once Norbert
Wiener’s The Human Use of Human Beings: Cybernetics and Society came out in 1950.
The range of artists who identified themselves with cybernetics in the fifties and sixties
initially had little access to “thinking machines.” Moreover, craft-minded engineers had
already been making turtles, jugglers, and light-seeking robot babes, not giant brains.
Using breadboards, copper wire, simple switches, and electronic sensors, artists followed
cyberneticians in making sculptures and environments that simulated interactive
sentience—analog movements and interfaces that had more to do with instinctive drives
and postwar sexual politics than the automation of knowledge production. Now obscured
by an ideology of a free-floating “intelligence” untethered by either hardware or flesh, AI
has forgotten the early days of cybernetics’ uptake by artists. Those efforts are worth
revisiting; they modeled relations with what the French philosophers Gilles Deleuze and
Félix Guattari have called the “machinic phylum,” having to do with how humans think
and feel in bodies engaged with a physical, material, emotionally stimulating, and
signaling world.
Cybernetics now seems to have collapsed into an all-pervasive discourse of AI
that was far from preordained. “Cybernetics,” as a word, claimed postwar newness for
concepts that were easily four centuries old: notions of feedback, machine damping,
biological homeostasis, logical calculation, and systems thinking that had been around
since the Enlightenment (boosted by the Industrial Revolution). The names in this
lineage include Descartes, Leibniz, Sadi Carnot, Clausius, Maxwell, and Watt. Wiener’s
coinage nonetheless had profound cultural effects. 47 The ubiquity today of the prefix
“cyber-” confirms the desire for a crisp signifier of the tangled relations between humans
and machines. In Wiener’s usage, things “cyber” simply involved “control and
communication in the animal and the machine.” But after the digital revolution, “cyber”
moved beyond servomechanisms, feedback loops, and switches to encompass software,
algorithms, and cyborgs. The work of cybernetically inclined artists concerns the
emergent behaviors of life that elude AI in its current condition.
As to that original coinage, Wiener had reached back to the ancient Greek to
borrow the word for “steersman” (κυβερνήτης / kubernétés), a masculine figure
channeling power and instinct at the helm of a ship, who read the waves, judged the
wind, kept a hand on the tiller, and directed the slaves as they mindlessly (mechanically)
churned their oars. The Greek had already migrated into modern English via Latin, going
47
Wiener later had to admit the earlier coinage of the word in 1834 by André-Marie Ampère, who had
intended it to mean the “science of government,” a concept that remained dormant until the 20th century.
174
from kuber- to guber—the root of “gubernatorial” and “governor,” another term for
masculine control, deployed by James Watt to describe his 19th-century device for
modulating a runaway steam engine. Cybernetics thus took ideas that had long
analogized people and devices and generalized them to an applied science by adding that
“-ics.” Wiener’s three c’s (command, control, communication) drew on the mathematics
of probability to formalize systems (whether biological or mechanical) theorized as a set
of inputs of information achieving outputs of actions in an environment—a muscular,
fleshy agenda often minimized in genealogies of AI.
But the etymology does little to capture the excitement felt by participants, as
mathematics joined theoretical biology (Arturo Rosenblueth) and information theory
(Claude Shannon, Walter Pitts, Warren McCulloch) to produce a barrage of
interdisciplinary research and publications viewed as changing not just the way science
was done but the way future humans would engage with the technosphere. As Wiener
put it, “We have modified our environment so radically that we must now modify
ourselves in order to exist.” 48 The pressing question is: How are we modifying
ourselves? Are we going in the right direction or have we lost our way, becoming the
tools of our tools? Revisiting the early history of humanist/artists’ contribution to
cybernetics may help direct us toward a less perilous, more ethical future.
The year 1968 was a high-water mark of the cultural diffusion and artistic uptake
of the term. In that year, the Howard Wise gallery opened its show of Wen-Ying Tsai’s
“Cybernetic Sculpture” in midtown Manhattan, and Polish émigré Jasia Reichardt opened
her exhibition “Cybernetic Serendipity” at London’s ICA. (The “Cybernetic” in her title
was intended to evoke “made by or with computers,” even though most of the artworks
on view had no computers, as such, in their responsive circuits.) The two decades
between 1948 and 1968 had seen both the fanning out of cybernetic concepts into a
broader culture and the spread of computation machines themselves in a slow migration
from proprietary military equipment, through the multinational corporation, to the
academic lab, where access began to be granted to artists. The availability of cybernetic
components—“sensor organs” (electronic eyes, motion sensors, microphones) and
“effector organs” (electronic “breadboards,” switches, hydraulics, pneumatics)—on the
home hobbyist front rendered the computer less an “electronic brain” than an adjunct
organ in a kit of parts. There was not yet a ruling metaphor of “artificial intelligence.”
So artists were bricoleurs of electronic bodies, interested in actions rather than calculation
or cognition. There were inklings of “computer” as calculator in the drive toward Homo
rationalis, but more in aspiration than achievement.
In light of today’s digital convergence in art/science imaging tools, Reichardt’s
show was prophetic in its insistence on confusing the boundaries between art and what
we might dub “creative applied science.” According to the catalog, “no visitor to the
exhibition, unless he reads all the notes relating to all the works, will know whether he is
looking at something made by an artist, engineer, mathematician, or architect.” So the
comically dysfunctional robot by Nam June Paik, Robot K-456 (1964), featured on the
catalog’s cover and described as “a female robot known for her disturbing and
idiosyncratic behavior,” would face off against a balletic Colloquy of Mobiles (1968)
from second-order cybernetician Gordon Pask. Pask worked with a London theater
48
The Human Use of Human Beings (1954 edition), p. 46.
175
designer to craft a spindly “male” apparatus of hinges and rods, set up to communicate
with bulbous “female” fiberglass entities nearby. Whether anyone could actually map the
quiddities of the program (or glean its reactionary gender theater) without reading the
catalog essay is an open question. What is significant is Pask’s focus on the behaviors of
his automata, their interactivity, their responsiveness within an artificially modulated
environment, and their “reflection” of human behaviors.
The ICA’s “Cybernetic Serendipity” introduced an important paradigm: the
machinic ecosystem, in which the viewer was a biological part, tasked with figuring out
just what the triggers for interaction might be. The visitors in those London galleries
suddenly became “cybernetic organisms”—cyborgs—since to experience the art
adequately, one needed to enter a kind of symbiotic colloquy with the servomechanisms.
This turn toward human-machine interactive environments as an aesthetic becomes
clearer when we examine a few other artworks from the period, beginning with one
constituting an early instance of emergent behavior—Senster, the interactive sculpture by
artist/engineer Edward Ihnatowicz (1970), celebrated by medical robotics engineer Alex
Zivanovic, editor of a Web site devoted to Ihnatowicz’s little-known career, as “one of
the first computer controlled interactive robotic works of art.” Here, “the computer”
makes its entry (albeit a twelve-bit, limited device). But rather than “intelligence,”
Ihnatowicz sought to make an avatar of affective behavior. Key to Senster’s uncanny
success was the programming with which Ihnatowicz constrained the fifteen-foot-long
hydraulic apparatus (its hinge design and looming appearance inspired by a lobster claw)
to convey shyness in responding to humans in its proximity. Senster’s sound channels
and motion sensors were set to recoil at loud noises and sudden aggressive movements.
Only those humans willing to speak softly and modulate their gestures would be
rewarded by Senster’s quiet, inquisitive approach—an experience that became real for
Ihnatowicz himself when he first assembled the program and the machine turned to him
solicitously after he’d cleared his throat.
In these artistic uses of cybernetic beings, we sense a growing necessity to train
the public to experience itself as embedded in a technologized environment, modifying
itself to communicate intuitively with machines. This necessity had already become
explicit in Tsai’s “Cybernetic Sculpture” show. Those experiencing his immersive
installation were expected to experiment with machinic life: What behaviors would
trigger the servomechanisms? Likely, the human gallery attendant would have had to
explain the protocol: “Clap your hands—that gets the sculptures to respond.” As an early
critic described it:
A grove of slender stainless-steel rods rises from a plate. This base vibrates at 30
cycles per second; the rods flex rapidly, in harmonic curves. Set in a dark room,
they are lit by strobes. The pulse of the flashing lights varies—they are
connected to sound and proximity sensors. The result is that when one
approaches a Tsai or makes a noise in its vicinity, the thing responds. The rods
appear to move; there is a shimmering, a flashing, an eerie ballet of metal, whose
apparent movements range from stillness to jittering and back to a slow,
indescribably sensuous undulation. 49
49
Robert Hughes, Time magazine (October 2, 1972) review of Tsai exhibition at Denise René gallery.
176
Like Senster, the apparatus stimulated (and simulated) an affective rather than
rational interaction. Humans felt they were encountering behaviors indicative of
responsive life; Tsai’s entities were often classed as “vegetal” or “aquatic.” Such
environmental and kinetic ambitions were widespread in the international art world of the
time. Beyond the stable at Howard Wise, there were the émigrés forming the collective
GRAV in Paris, the “cybernetic architectures” of Nicolas Schöffer, the light and plastic
gyrations of the German Zero Gruppe, and so on—all defining and informing the genre
of installation art to come.
The artistic use of cybernetic beings in the late sixties made no investment in
“intelligence.” Knowing machines were dumb and incapable of emotion, these creators
were confident in staging frank simulations. What interested them were machinic
motions evoking drives, instincts, and affects; they mimicked sexual and animal
behaviors, as if below the threshold of consciousness. Such artists were uninterested in
the manipulation of data or information (although Hans Haacke would move in that
direction by 1972 with his “Real-Time Systems” works). The cybernetic culture that
artists and scientists were putting in place on two continents embedded the human in the
technosphere and seduced perception with the graceful and responsive behaviors of the
machinic phylum. “Artificial” and “natural” intertwined in this early cybernetic
aesthetic.
But it wouldn’t end here. Crucial to the expansion of this uncritical, largely
masculine set of cybernetic environments would be a radical, critical cohort of
astonishing women artists emerging in the 1990s, fully aware of their predecessors in art
and technology but perhaps more inspired by the feminist founders of the 1970 journal
Radical Software and the cultural blast of Donna Haraway’s inspiring 1984 polemic, “A
Cyborg Manifesto.” The creaky gender theater of Paik and Pask, the innocent creatures
of Ihnatowicz and Tsai, were mobilized as savvy, performative, and postmodern, as in
Lynn Hershman Leeson’s Dollie Clone Series (1995-98) consisting of the interactive
assemblages CyberRoberta and Tillie, the Telerobotic Doll, who worked the
technosphere with the professionalism of burlesque, winking and folding us viewers into
an explicit consciousness of our voyeuristic position as both seeing subjects and objectsto-be-looked-at.
The “innocent” technosphere established by male cybernetic sculptors of the
1960s was, by the 1990s, identified by feminist artists as an entirely suffusive condition
demanding our critical attention. At the same time, feminists tackled the question of
whose “intelligence” AI was attempting to simulate. For an artist such as Hershman
Leeson, responding to the technical “triumph” of cloning Dolly the sheep, it was crucial
to draw the connection between meat production and “meat machines.” Hershman
Leeson produced “dolls” as clones, offering a critical framing of the way contemporary
individuation had become part of an ideological, replicative, plastic realm.
While the technofeminists of the 1990s and into the 2000s weren’t all cyber all
the time, their works nonetheless complicated the dominant machinic and kinetic
qualities of male artists’ previous techno-environments. The androgynous tele-cyborg in
Judith Barry’s Imagination, Dead Imagine (1991), for example, had no moving parts:
He/she was comprised of pure signals, flickering projections on flat surfaces. In her
setup, Barry commented on the alienating effects of late-20th-century technology. The
image of an androgynous head fills an enormous cube made of ten-foot-square screens on
177
five sides, mounted on a ten-foot-wide mirrored base. A variety of viscous and
unpleasant-looking fluids (yellow, reddish-orange, brown), dry materials (sawdust?
flour?), and even insects drizzle or dust their way down the head, whose stoic sublimity is
made gorgeously virtual on the work’s enormous screens. Dead Imagine, through its
large-scale and cubic “Platonic” form, remains both artificial and locked into the body—
refusing a detached “intelligence” as being no intelligence at all.
Artists in the new millennium inherit this critical tradition and inhabit the current
paradigms of AI, which has slid from partial simulations to claims of intelligence. In the
1955 proposal thought to be the first printed usage of the phrase “artificial intelligence,”
computer scientist John McCarthy and his colleagues Marvin Minsky, Nathaniel
Rochester, and Claude Shannon conjectured that “every aspect of learning or any other
feature of intelligence can in principle be so precisely described that a machine can be
made to simulate it.” This modest theoretical goal has inflated over the past sixty-four
years and is now expressed by Google DeepMind as an ambition to “Solve intelligence.”
Crack the code! But unfortunately, what we hear cracking is not code but small-scale
capitalism, the social contract, and the scaffolding of civility. Taking away the jobs of
taxi and truck drivers, roboticizing direct marketing, hegemonizing entertainment,
privatizing utilities, and depersonalizing health care—are these the “whips” that Wiener
feared we would learn to love?
Artists can’t solve any of this. But they can remind us of the creative potential of
the paths not taken—the forks in the road that were emerging around 1970, before
“information” became capital and “intelligence” equaled data harvesting. Richly
evocative of what can be done with contemporary tools when revisiting earlier
possibilities is French artist Philippe Parreno’s “firefly piece,” so nicknamed to avoid
having to iterate its actual title: With a Rhythmic Instinction to Be Able to Travel Beyond
Existing Forces of Life (2014). Described by the artist as “an automaton,” the sculptural
installation juxtaposes a flickering projection of black-and-white drawings of fireflies
with a band of oscillating green-on-black binary figures. The drawings and binary
figures are animated using algorithms from mathematician John Horton Conway’s 1970
Game of Life, a “cellular automaton.”
Conway set up parameters for any square (“cell”) to be lit (“alive”) or dark
(“dead”) in an infinite, two-dimensional grid. The rules are summarized as follows: A
single cell will quickly die of loneliness. But a cell touching three or more other “live”
cells will also die, “due to crowding.” A cell survives and thrives if it has just two
neighbors . . . and so on. As one cell dies, it may create the conditions for other cells to
survive, yielding patterns that appear to move and grow, shifting across the grid like
evanescent neural impulses or bioluminescent clusters of diatoms. In Stephen Hawking’s
2012 film The Meaning of Life, the narrator describes Conway’s mathematical model as
simulating “how a complex thing like the mind might come about from a basic set of
rules,” revealing the overweening ambitions that characterize contemporary AI: “[T]hese
complex properties emerge from simple laws that contain no concepts like movement or
reproduction,” yet they produce “species,” and cells “can even reproduce, just as life does
in the real world.” 50
Just as life does? Artists know the blandishments of simulation and
representation, the difference between the genius of artifice and the realities of what “life
50
Narration in Stephen Hawking’s The Meaning of Life (Smithson Productions, Discovery Channel, 2012).
178
does.” Parreno’s piece is an intuitive assembly of our experience of “life” through
embodied, perspectival engagement. Our consciousness is electrically (cybernetically)
enmeshed, yet we don’t respond as if this human-generated set of elegant simulations had
its own intelligence.
The artistic use of cybernetic beings also reminds us that consciousness itself is
not just “in here.” It is streaming in and out, harmonizing those sensory, scintillating
signals. Mind happens well outside the limits of the cranium (and its simulacrum, the
“motherboard”). In Mary Catherine Bateson’s paraphrase of her father Gregory’s
second-order cybernetics, mind is material “not necessarily defined by a boundary such
as an envelope of skin.” 51 Parreno pairs the simulations of art with the simulations of
mathematics to force the Wiener-like point that any such model is not, by itself, just like
life. Models are just that—parts of signaling systems constituting “intelligence” only
when their creaturely counterparts engage them in lively meaning making.
Contemporary AI has talked itself into a corner by instrumentalizing and particularizing
tasks and subroutines, confusing these drills with actual wisdom. The brief cultural
history offered here reminds us that views of data as intelligence, digital nets as “neural,”
or isolated individuals as units of life, were alien even to Conway’s brute simulation.
We can stigmatize the stubborn arrogance of current AI as “right cybernetics,” the
path that led to current automated weapons systems, Uber’s ill-disguised hostility to
human workers, and the capitalist dreams of Google. Now we must turn back to left
cybernetics—theoretical biologists and anthropologists engaged with a trans-species
understanding of intelligent systems. Gregory Bateson’s observation that corporations
merely simulate “aggregates of parts of persons,” with profit-maximizing decisions cut
off from “wider and wiser parts of the mind,” has never been more timely. 52
The cybernetic epistemology offered here suggests a new approach. The
individual mind is immanent, not only in the body but also in pathways outside the body,
and there is a larger Mind, of which the individual mind is only a subsystem. This larger
Mind, Bateson holds, is comparable to God, and is perhaps what some people mean by
“God,” but it is still immanent in the total interconnected social system and planetary
ecology. This is not the collective delusion of an exterior “God” who speaks from
outside human consciousness (this long-seated monotheistic conceit, Bateson suggests,
leads to views of nature and environment as also outside the “individual” human,
rendering them as “gifts to exploit”). Rather, Bateson’s “God” is a placeholder for our
evanescent experience of interacting consciousness-in-the-world: larger Mind as a result
of inputs and actions that then become inputs for other actions in concert with other
entities—webs of symbiotic relationships that form patterns we need urgently to sense
and harmonize with. 53
From Tsai in the 1970s to Hershman Leeson in the 1990s to Parreno in 2014,
artists have been critiquing right cybernetics and plying alternative, embodied,
environmental experiences of “artificial” intelligence. Their artistic use of cybernetic
beings offers the wisdom of symbionts experienced in the kinds of poeisis that can be
achieved in this world: rhythms of signals and intuitive actions that produce the
51
Mary Catherine Bateson, 1999 foreword to Gregory Bateson, Steps to an Ecology of Mind (Chicago:
University of Chicago Press, 1972): xi.
52
Steps to an Ecology of Mind, p. 452.
53
Ibid., pp. 467-8.
179
movements of life partnered with an electro-mechanical and -magnetic technosphere.
Life, in its mysterious negentropic entanglements with matter and Mind.
180
Over nearly four decades, Stephen Wolfram has been a pioneer in the development and
application of computational thinking and responsible for many innovations in science,
technology and business.
His 1982 paper “Cellular Automata as Simple Self-Organizing Systems,” written
at the age of twenty-three, was the first of numerous significant scientific contributions
aimed at understanding the origins of complexity in nature.
It was around this time that Stephen briefly came into my life. I had established
The Reality Club, an informal gathering of intellectuals who met in New York City to
present their work before peers in other disciplines. (Note: In 1996, The Reality Club
went online as Edge.org). Our first speaker? Stephen Wolfram, a “wunderkind” who
had arrived in Princeton at the Institute for Advanced Study. I distinctly recall his
focused manner as he sat down on a couch in my living room and spoke uninterrupted for
about an hour before the assembled group.
Since that time, Stephen has become intent making the world’s knowledge easily
computable and accessible. His program Mathematica is the definitive system for
modern technical computing. Wolfram|Alpha computes expert-level answers using AI
technology. He considers his Wolfram Language to be the first true computational
communication language for humans and AIs.
I caught up with him again four years ago, when we arranged to meet in
Cambridge, Massachusetts, for a freewheeling conversation about AI. Stephen walked
in, said hello, sat down, and, looking at the video camera set up to record the
conversation for Edge, began to talk and didn’t stop for two and a half hours.
The essay that follows is an edited version of that session, which was a Wolfram
master class of sorts and is an appropriate way to end this volume—just as Stephen’s
Reality Club talk in the ’80s was a great way to initiate the ongoing intellectual
enterprise whose result is the rich community of thinkers presenting their work to one
another and to the public in this book.
181
ARTIFICIAL INTELLIGENCE AND THE FUTURE OF CIVILIZATION
Stephen Wolfram
Stephen Wolfram is a scientist, inventor, and the founder and CEO of Wolfram
Research. He is the creator of the symbolic computation program Mathematica and its
programming language, Wolfram Language, as well as the knowledge engine
Wolfram|Alpha. He is also the author of A New Kind of Science.
The following is an edited transcript from a live interview with him conducted in
December 2015.
I see technology as taking human goals and making them automatically executable by
machines. Human goals of the past have entailed moving objects from here to there,
using a forklift rather than our own hands. Now the work we can do automatically, with
machines, is mental rather than physical. It’s obvious that we can automate many of the
tasks we humans have long been proud of doing ourselves. What’s the future of the
human condition in that situation?
People talk about the future of intelligent machines and whether they’ll take over
and decide what to do for themselves. But the inventing of goals is not something that
has a path to automation. Someone or something has to define what a machine’s purpose
should be—what it’s trying to execute. How are goals defined? For a given human, they
tend to be defined by personal history, cultural environment, the history of our
civilization. Goals are uniquely human. Where the machine is concerned, we can give it
a goal when we build it.
What kinds of things have intelligence, or goals, or purpose? Right now, we
know one great example, and that’s us—our brains, our human intelligence. Human
intelligence, I once assumed, is far beyond anything else that exists naturally in the
world; it’s the result of an elaborate process of evolution and thus stands apart from the
rest of existence. But what I’ve realized, as a result of the science I’ve done, is that this is
not the case.
People might say, for instance, “The weather has a mind of its own.” That’s an
animist statement and seems to have no place in modern scientific thinking. But it’s not
as silly as it sounds. What does the human brain do? A brain receives certain input, it
computes things, it causes certain actions to happen, it generates a certain output. Like
the weather. All sorts of systems are, effectively, doing computations—whether it’s a
brain or, say, a cloud responding to its thermal environment.
We can argue that our brains are doing vastly more sophisticated computations
than those in the atmosphere. But it turns out that there’s a broad equivalence between
the kinds of computations that different kinds of systems do. This renders the question of
the human condition somewhat poignant, because it seems we’re not as special as we
thought. There are all those different systems of nature that are pretty much equivalent,
in terms of their computational capabilities.
What makes us different from all those other systems is the particulars of our
history, which give us our notions of purpose and goals. That’s a long way of saying that
when the box on our desk thinks as well as the human brain does, what it still won’t have,
intrinsically, are goals and purposes. Those are defined by our particulars—our particular
biology, our particular psychology, our particular cultural history.
182
When we consider the future of AI, we need to think about the goals. That’s what
humans contribute; that’s what our civilization contributes. The execution of those goals
is what we can increasingly automate. What will the future of humans be in such a
world? What will there be for them to do? One of my projects has been to understand
the evolution of human purposes over time. Today we’ve got all kinds of purposes. If
you look back a thousand years, people’s goals were quite different: How do I get my
food? How do I keep myself safe? In the modern Western world, for the most part you
don’t spend a large fraction of your life thinking about those purposes. From the point of
view of a thousand years ago, some of the goals people have today would seem utterly
bizarre—for example, like exercising on a treadmill. A thousand years ago that would
sound like a crazy thing to do.
What will people be doing in the future? A lot of purposes we have today are
generated by scarcity of one kind or another. There are scarce resources in the world.
People want to get more of something. Time itself is scarce in our lives. Eventually,
those forms of scarcity will disappear. The most dramatic discontinuity will surely be
when we achieve effective human immortality. Whether this will be achieved
biologically or digitally isn’t clear, but inevitably it will be achieved. Many of our
current goals are driven in part by our mortality: “I’m only going to live a certain time, so
I’d better get this or that done.” And what happens when most of our goals are executed
automatically? We won’t have the kinds of motivations we have today. One question I’d
like an answer for is, What do the derivatives of humans in the future end up choosing to
do with themselves? One of the potential bad outcomes is that they just play video games
all the time.
~ ~ ~
The term “artificial intelligence” is evolving, in its use in technical language. These
days, AI is very popular, and people have some idea of what it means. Back when
computers were being developed, in the 1940s and 1950s, the typical title of a book or a
magazine article about computers was “Giant Electronic Brains.” The idea was that just
as bulldozers and steam engines and so on automated mechanical work, computers would
automate intellectual work. That promise turned out to be harder to fulfill than many
people expected. There was, at first, a great deal of optimism; a lot of government
money got spent on such efforts in the early 1960s. They basically just didn’t work.
There are a lot of amusing science-fiction-ish portrayals of computers in the
movies of that time. There’s a cute one called Desk Set, which is about an IBM-type
computer being installed in a broadcasting company and putting everybody out of a job.
It’s cute because the computer gets asked a bunch of reference-library questions. When
my colleagues and I were building Wolfram|Alpha, one of the ideas we had was to get it
to answer all of those reference-library questions from Desk Set. By 2009, it could
answer them all.
In 1943, Warren McCulloch and Walter Pitts came up with a model for how
brains conceptually, formally, might work—an artificial neural network. They saw that
their brainlike model would do computations in the same way as Turing Machines. From
their work, it emerged that we could make brainlike neural networks that would act as
general computers. And in fact, the practical work done by the ENIAC folks and John
183
von Neumann and others on computers came directly not from Turing Machines but
through this bypath of neural networks.
But simple neural networks didn’t do much. Frank Rosenblatt invented a learning
device he called the perceptron, which was a one-layer neural network. In the late sixties,
Marvin Minsky and Seymour Papert wrote a book titled Perceptrons, in which they
basically proved that perceptrons couldn’t do anything interesting, which is correct.
Perceptrons could only make linear distinctions between things. So the idea was more or
less dropped. People said, “These guys have written a proof that neural networks can’t
do anything interesting, therefore no neural networks can do anything interesting, so let’s
forget about neural networks.” That attitude persisted for some time.
Meanwhile, there were a couple of other approaches to AI. One was based on
understanding, at a formal level, symbolically, how the world works; and the other was
based on doing statistics and probabilistic kinds of things. With regard to symbolic AI,
one of the test cases was, Can we teach a computer to do something like integrals? Can
we teach a computer to do calculus? There were tasks like machine translation, which
people thought would be a good example of what computers could do. The bottom line is
that by the early seventies, that approach had crashed.
Then there was a trend toward devices called expert systems, which arose in the
late seventies and early eighties. The idea was to have a machine learn the rules that an
expert uses and thereby figure out what to do. That petered out. After that, AI became
little more than a crazy pursuit.
~ ~ ~
I had been interested in how you make an AI-like machine since I was a kid. I was
interested particularly in how you take the knowledge we humans have accumulated in
our civilization and automate answering questions on the basis of that knowledge. I
thought about how you could do that symbolically, by building a system that could break
down questions into symbolic units and answer them. I worked on neural networks at
that time and didn’t make much progress, so I put it aside for a while.
Back in mid-2002 to 2003, I thought about that question again: What does it take
to make a computational knowledge system? The work I’d done by then pretty much
showed that my original belief about how to do this was completely wrong. My original
belief had been that in order to make a serious computational knowledge system, you first
had to build a brainlike device and then feed it knowledge—just as humans learn in
standard education. Now I realized that there wasn’t a bright line between what is
intelligent and what is simply computational.
I had assumed that there was some magic mechanism that made us vastly more
capable than anything that was just computational. But that assumption was wrong. This
insight is what led to Wolfram|Alpha. What I discovered is that you can take a large
collection of the world’s knowledge and automatically answer questions on the basis of
it, using what are essentially merely computational techniques. It was an alternative way
to do engineering—a way that’s much more analogous to what biology does in evolution.
In effect, what you normally do when you build a program is build it step-by-step.
But you can also explore the computational universe and mine technology from that
universe. Typically, the challenge is the same as in physical mining: That is, you find a
supply of, let’s say, iron, or cobalt, or gadolinium, with some special magnetic properties,
184
and you turn that special capability to a human purpose, to something you want
technology to do. In the case of magnetic materials, there are plenty of ways to do that.
In terms of programs, it’s the same story. There are all kinds of programs out there, even
tiny programs that do complicated things. Could we entrain them for some useful human
purpose?
And how do you get AIs to execute your goals? One answer is to just talk to
them, in the natural language of human utterances. It works pretty well when you’re
talking to Siri. But when you want to say something longer and more complicated, it
doesn’t work well. You need a computer language that can represent sophisticated
concepts in a way that can be progressively built up and isn’t possible in natural
language. What my company spent a lot of time doing was building a knowledge-based
language that incorporates the knowledge of the world directly into the language. The
traditional approach to creating a computer language is to make a language that
represents operations that computers intrinsically know how to do: allocating memory,
setting values of variables, iterating things, changing program counters, and so on.
Fundamentally, you’re telling computers to do things in your own terms. My approach
was to make a language that panders not to the computers but to the humans, to take
whatever a human thinks of and convert it into some form that the computer can
understand. Could we encapsulate the knowledge we’d accumulated, both in science and
in data collection, into a language we could use to communicate with computers? That’s
the big achievement of my last thirty years or so—being able to do that.
Back in the 1960s, people would say things like, “When we can do such-andsuch,
we’ll know we have AI. When we can do an integral from a calculus course, we’ll
know we have AI. When we can have a conversation with a computer and make it seem
human. . . ,” et cetera. The difficulty was, “Well, gosh, the computer just doesn’t know
enough about the world.” You’d ask the computer what day of the week it was, and it
might be able to answer that. You’d ask it who the President was, and it probably
couldn’t tell you. At that point, you’d know you were talking to a computer and not a
person. But now when it comes to these Turing Tests, people who’ve tried connecting,
for example, Wolfram|Alpha to their Turing Test bots find that the bots lose every time.
Because all you have to do is start asking the machine sophisticated questions and it will
answer them! No human can do that. By the time you’ve asked it a few disparate
questions, there will be no human who knows all those things, yet the system will know
them. In that sense, we’ve already achieved good AI, at that level.
Then there are certain kinds of tasks easy for humans but traditionally very hard
for machines. The standard one is visual object identification: What is this object?
Humans can recognize it and give some simple description of it, but a computer was just
hopeless at that. A couple of years ago, though, we brought out a little imageidentification
system, and many other companies have done something similar—ours
happens to be somewhat better than the rest. You show it an image, and for about ten
thousand kinds of things, it will tell you what it is. It’s fun to show it an abstract painting
and see what it says. But it does a pretty good job.
It works using the same neural-network technology that McCulloch and Pitts
imagined in 1943 and lots of us worked on in the early eighties. Back in the 1980s,
people successfully did OCR—optical character recognition. They took the twenty-six
letters of the alphabet and said, “OK, is that an A? Is that a B? Is that a C?” and so on.
185
That could be done for twenty-six different possibilities, but it couldn’t be done for ten
thousand. It was just a matter of scaling up the whole system that makes this possible
today. There are maybe five thousand picturable common nouns in English, ten thousand
if you include things like special kinds of plants and beetles which people would
recognize with some frequency. What we did was train our system on 30 million images
of these kinds of things. It’s a big, complicated, messy neural network. The details of
the network probably don’t matter, but it takes about a quadrillion GPU operations to do
the training.
Our system is impressive because it pretty much matches what humans can do. It
has about the same training data humans have—about the same number of images a
human infant would see in the first couple of years of its life. Roughly the same number
of operations have to be done in the learning process, using about the same number of
neurons in at least the first levels of our visual cortex. The details are different; the way
these artificial neurons work has little to do with how the brain’s neurons work. But the
concept is similar, and there’s a certain universality to what’s going on. At the
mathematical level, it’s a composition of a very large number of functions, with certain
continuity properties that let you use calculus methods to incrementally train the system.
Given those attributes, you can end up with something that does the same job human
brains do in physiological recognition.
But does this constitute AI? There are a few basic components. There’s
physiological recognition, there’s voice-to-text, there’s language translation—things
humans manage to do with varying degrees of difficulty. These are essentially some of
the links to how we make machines that are humanlike in what they do. For me, one of
the interesting things has been incorporating those capabilities into a precise symbolic
language to represent the everyday world. We now have a system that can say, “This is a
glass of water.” We can go from a picture of a glass of water to the concept of a glass of
water. Now we have to invent some actual symbolic language to represent those
concepts.
I began by trying to represent mathematical, technical kinds of knowledge and
went on to other kinds of knowledge. We’ve done a pretty good job of representing
objective knowledge in the world. Now the problem is to represent everyday human
discourse in a precise symbolic way—a knowledge-based language intended for
communication between humans and machines, so that humans can read it and machines
can understand it, too. For instance, you might say “X is greater than 5.” That’s a
predicate. You might also say, “I want a piece of chocolate.” That’s also a predicate. It
has an “I want” in it. We have to find a precise symbolic representation of the desires we
express in human natural language.
In the late 1600s, Gottfried Leibniz, John Wilkins, and others were concerned
with what they called philosophical languages—that is, complete, universal, symbolic
representations of things in the world. You can look at the philosophical language of
John Wilkins and see how he divided up what was important in the world at the time.
Some aspects of the human condition have been the same since the 1600s. Some are very
different. His section on death and various forms of human suffering was huge; in
today’s ontology, it’s a lot smaller. It’s interesting to see how a philosophical language
of today would differ from a philosophical language of the mid-1600s. It’s a measure of
our progress. Many such attempts at formalization have happened over the years. In
186
mathematics, for example: Whitehead and Russell’s Principia Mathematica in 1910 was
the biggest showoff effort. There were previous attempts by Gottlob Frege and Giuseppe
Peano that were a little more modest in their presentation. Ultimately, they were wrong
in what they thought they should formalize: They thought they should formalize some
process of mathematical proof, which turns out not to be what most people care about.
With regard to a modern analog of the Turing Test, it’s an interesting question.
There’s still the conversational bot, which is Turing’s idea. That one hasn’t been solved
yet. It will be solved—the only question is, What is the application for which it is
solved? For a long time I would ask, “Why should we care?”—because I thought the
principal application would be customer service, which wasn’t particularly high on my
list. But customer service, where you’re trying to interface, is just where you need this
conversational language.
One big difference between Turing’s time and ours is the method of
communicating with computers. In his time, you typed something into the machine and it
typed back a response. In today’s world, it responds with a screen—as for instance, when
you want to buy a movie ticket. How is a transaction with a machine different from a
transaction with a human? The main answer is that there’s a visual display. It asks you
something, and you press a button, and you can see the result immediately. For example,
in Wolfram|Alpha, when it’s used inside Siri, if there’s a short answer, Siri will tell you
the short answer. But what most people want is the visual display, showing the
infographic of this or that. This is a nonhuman form of communication that turns out to
be richer than the traditional spoken, or typed, human communication. In most humanto-human
communication, we’re stuck with pure language, whereas in computer-tohuman
communication we have this much higher bandwidth channel—of visual
communication.
Many of the most powerful applications of the Turing Test fall away now that we
have this additional communication channel. For example, here’s one we’re pursuing
right now. It’s a bot that communicates about writing programs: You say, “I want to
write a program. I want it to do this.” The bot will say, “I’ve written this piece of
program. This is what it does. Is this what you want?” Blah-blah-blah. It’s a back-andforth
bot. Devising such systems is an interesting problem, because they have to have a
model of a human if they’re trying to explain something to you. They have to know what
the human is confused about.
What has long been difficult for me to understand is, What’s the point of a
conventional Turing Test? What’s the motivation? As a toy, one could make a little chat
bot that people could chat with. That will be the next thing. The current round of deep
learning—particularly, recurrent neural networks—is making pretty good models of
human speech and human writing. We can type in, say, “How are you feeling today?”
and it knows most of the time what sort of response to give. But I want to figure out
whether I can automate responding to my email. I know the answer is “No.” A good
Turing Test, for me, will be when a bot can answer most of my email. That’s a tough
test. It would have to learn those answers from the humans the email is connected to. I
might be a little bit ahead of the game, because I’ve been collecting data on myself for
about twenty-five years. I have every piece of email for twenty-five years, every
keystroke for twenty. I should be able to train an avatar, an AI, that will do what I can
do—perhaps better than I could.
187
~ ~ ~
People worry about the scenario in which AIs take over. I think something much more
amusing, in a sense, will happen first. The AI will know what you intend, and it will be
good at figuring out how to get there. I tell my car’s GPS I want to go to a particular
destination. I don’t know where the heck I am, I just follow my GPS. My children like
to remind me that once when I had a very early GPS—the kind that told you, “Turn this
way, turn that way”—we ended up on one of the piers going out into Boston Harbor.
More to the point is that there will be an AI that knows your history, and knows
that when you’re ordering dinner online you’ll probably want such-and-such, or when
you email this person, you should talk to them about such-and-such. More and more, the
AIs will suggest to us what we should do, and I suspect most of the time people will just
go along with that. It’s good advice—better than what you would have figured out for
yourself.
As far as the takeover scenario is concerned, you can do terrible things with
technology and you can do good things with technology. Some people will try to do
terrible things with technology, and some people will try to do good things with
technology. One of the things I like about today’s technology is the equalization it has
produced. I used to be proud that I had a better computer than anybody I knew; now we
all have the same kind of computers. We have the same smartphones, and pretty much
the same technology can be used by a decent fraction of the planet’s 7 billion people. It’s
not the case that the king’s technology is different from everybody else’s. That’s an
important advance.
The great frontier five hundred years ago was literacy. Today, it’s doing
programming of some kind. Today’s programming will be obsolete in a not very long
time. For example, people no longer learn assembly language, because computers are
better at writing assembly language than humans are, and only a small set of people need
to know the details of how language gets compiled into assembly language. A lot of
what’s being done by armies of programmers today is similarly mundane. There’s no
good reason for humans to be writing Java code or JavaScript code. We want to
automate the programming process so that what’s important goes from what the human
wants done to getting the machine, as automatically as possible, to do it. This will
increase that equalization, which is something I’m interested in. In the past, if you
wanted to write a serious piece of code, or program for something important and real, it
was a lot of work. You had to know quite a bit about software engineering, you had to
invest months of time in it, you had to hire programmers who knew this or you had to
learn it yourself. It was a big investment.
That’s not true anymore. A one-line piece of code already does something
interesting and useful. It allows a vast range of people who couldn’t make computers do
things for them, make computers do things for them. Something I’d like to see is a lot of
kids around the world learn the new capabilities of knowledge-based programming and
then produce code that’s effectively as sophisticated as what anybody in the top ranks can
produce. This is within reach. We’re at the point where anybody can learn to do
knowledge-based programming, and, more important, learn to think computationally.
The actual mechanics of programming are easy now. What’s difficult is imagining things
in a computational way.
188
How do you teach computational thinking? In terms of how to do programming,
it’s an interesting question. Take nanotechnology. How did we achieve nanotechnology?
Answer: We took technology as we understand it on a large scale and we made it very
small. How to make a CPU chip on the atomic scale? Fundamentally, we use the same
architecture as the CPU chip we know and love. That isn’t the only approach one can
take. Looking at what simple programs can do suggests that you can take even simple
impoverished components and with the right compiler you can make them do interesting
things. We don’t do molecular-scale computing yet, because the ambient technology is
such that you’d have to spend a decade building it. But we’ve got the components that
are enough to make a universal computer. You might not know how to program with
those components, but by doing searches in the space of possible programs, you’d start to
amass building blocks, and you could then create a compiler for them. The surprising
thing is that impoverished stuff is capable of doing sophisticated things, and the
compilation step is not as gruesome as you might expect.
Just searching the computational universe and trying to find programs—building
blocks—that are interesting is a good approach. A more traditional engineering
approach—trying by pure thought to figure out how to build a universal computer—is a
harder row to hoe. That doesn’t mean it can’t be done, but my guess is that we’ll be able
to do some amazing things just by finding the components and searching the possible
programs we can make with them. Then it’s back to the question about connecting
human purposes to what is available from the system.
One question I’m interested in is, What will the world look like when most people
can write code? We had a transition, maybe five hundred years ago or so, when only
scribes and a small set of the population could read and write natural language. Today, a
small fraction of the population can write code. Most of the code they write is for
computers only. You don’t understand things by reading code. But there will come a
time when, as a result of things I’ve tried to do, the code is at a high enough level that it’s
a minimal description of what you’re trying to do. It will be a piece of code that’s
understandable to humans but also executable by the machines.
Coding is a form of expression, just as writing in a natural language is a form of
expression. To me, some simple pieces of code are poetic—they express ideas in a very
clean way. There’s an aesthetic aspect, much as there is to expression in a natural
language. One feature of code is that it’s immediately executable; it’s not like writing.
When you write something, somebody has to read it, and the brain that’s reading it has to
absorb the thoughts that came from the person who did the writing. Look at how
knowledge has been transmitted in the history of the world. At level zero, one form of
knowledge transmission is essentially genetic—that is, there’s an organism, and its
progeny has the same features that it had. Then there’s the kind of knowledge
transmission that happens with things like physiological recognition. A newborn creature
has some neural network with some random connections in it, and as the creature moves
around in the world, it starts recognizing kinds of objects and it learns that knowledge.
Then there’s the level that was the big achievement of our species, which is
natural language. The ability to represent knowledge abstractly enough that we can
communicate it brain to brain, so to speak. Arguably, natural language is our species’
most important invention. It’s what led, in many respects, to our civilization.
189
There’s yet another level, and probably one day it will have a more interesting
name. With knowledge-based programming, we have a way of creating an actual
representation of real things in the world, in a precise and symbolic way. Not only is it
understandable by brains and communicable to other brains and to computers, it’s also
immediately executable.
Just as natural language gave us civilization, knowledge-based programming will
give us—what? One bad answer is that it will give us the civilization of the AIs. That’s
what we don’t want to happen, because the AIs will do a great job communicating with
one another and we’ll be left out of it, because there’s no intermediate language, no
interface with our brains. What will this fourth level of knowledge communication lead
to? If you were Caveman Ogg and you were just realizing that language was starting,
could you imagine the coming of civilization? What should we be imagining right now?
This relates to the question of what the world would look like if most people
could code. Clearly, many trivial things would change: Contracts would be written in
code, restaurant recipes might be written in code, and so on. Simple things like that
would change. But much more profound things would also change. The rise of literacy
gave us bureaucracy, for example, which had already existed but dramatically
accelerated, giving us a greater depth of governmental systems, for better or worse. How
does the coding world relate to the cultural world?
Take high school education. If we have computational thinking, how does that
affect how we study history? How does that affect how we study languages, social
studies, and so on? The answer is, it has a great effect. Imagine you’re writing an essay.
Today, the raw material for a typical high school student’s essay is something that’s
already been written; students usually can’t generate new knowledge easily. But in the
computational world, that will no longer be true. If the students know something about
writing code, they’ll access all that digitized historical data and figure out something
new. Then they’ll write an essay about something they’ve discovered. The achievement
of knowledge-based programming is that it’s no longer sterile, because it’s got the
knowledge of the world knitted into the language you’re using to write code.
~ ~ ~
There’s computation all over the universe: in a turbulent fluid producing some
complicated pattern of flow, in the celestial mechanics of planetary interactions, in
brains. But does computation have a purpose? You can ask that about any system. Does
the weather have a goal? Does climate have a goal?
Can someone looking at Earth from space tell that there’s anything with a purpose
there? Is there a civilization there? In the Great Salt Lake, in Utah, there’s a straight
line. It turns out to be a causeway dividing two areas of the lake with different colors of
algae, so it’s a very dramatic straight line. There’s a road in Australia that’s long and
straight. There’s a railroad in Siberia that’s long, and lights go on when a train stops at
the stations. So from space you can see straight lines and patterns.
But are these clear enough examples of obvious purpose on Earth as viewed from
space? For that matter, how do we recognize extraterrestrials out there? How do we tell
if a signal we’re getting indicates purpose? Pulsars were discovered in 1967, when we
picked up a periodic flutter every second or so. The first question was, Is this a beacon?
190
Because what else would make a periodic signal? It turned out to be a rotating neutron
star.
One criterion to apply to a potentially purposeful phenomenon is whether it’s
minimal in achieving a purpose. But does that mean that it was built for the purpose?
The ball rolls down the hill because of gravitational pull. Or the ball rolls down the hill
because it’s satisfying the principle of least action. There are typically these two
explanations for some action that seems purposeful: the mechanistic explanation and the
teleological. Essentially all of our existing technology fails the test of being minimal in
achieving its purpose. Most of what we build is steeped in technological history, and it’s
incredibly non-minimal for achieving its purpose. Look at a CPU chip; there’s no way
that that’s the minimal way to achieve what a CPU chip achieves.
This question of how to identify purposefulness is a hard one. It’s an important
question, because radio noise from the galaxy is very similar to CDMA transmissions
from cell phones. Those transmissions use pseudo-noise sequences, which happen to
have certain repeatability properties. But they come across as noise, and they’re set up as
noise, so as not to interfere with other channels. The issue gets messier. If we were to
observe a sequence of primes being generated from a pulsar, we’d ask what generated
them. Would it mean that a whole civilization grew up and discovered primes and
invented computers and radio transmitters and did this? Or is there just some physical
process making primes? There’s a little cellular automaton that makes primes. You can
see how it works if you take it apart. It has a little thing bouncing inside it, and out
comes a sequence of primes. It didn’t need the whole history of civilization and biology
and so on to get to that point.
I don’t think there is abstract “purpose,” per se. I don’t think there’s abstract
meaning. Does the universe have a purpose? Then you’re doing theology in some way.
There is no meaningful sense in which there is an abstract notion of purpose. Purpose is
something that comes from history.
One of the things that might be true about our world is that maybe we go through
all this history and biology and civilization, and at the end of the day the answer is “42,”
or something. We went through all those 4 billion years of various kinds of evolution
and then we got to “42.”
Nothing like that will happen, because of computational irreducibility. There are
computational processes that you can go through in which there is no way to shortcut that
process. Much of science has been about shortcutting computation done by nature. For
example, if we’re doing celestial mechanics and want to predict where the planets will be
a million years from now, we could follow the equations, step-by-step. But the big
achievement in science is that we’re able to shortcut that and reduce the computation.
We can be smarter than the universe and predict the endpoint without going through all
the steps. But even with a smart enough machine and smart enough mathematics, we
can’t get to the endpoint without going through the steps. Some details are irreducible.
We have to irreducibly follow those steps. That’s why history means something. If we
could get to the endpoint without going through the steps, history would be, in some
sense, pointless.
So it’s not the case that we’re intelligent and everything else in the world is not.
There’s no enormous abstract difference between us and the clouds or us and the
cellular automata. We cannot say that this brainlike neural network is qualitatively
191
different from this cellular-automaton system. The difference is a detailed difference.
This brainlike neural network was produced by the long history of civilization, whereas
the cellular automaton was created by my computer in the last microsecond.
The problem of abstract AI is similar to the problem of recognizing
extraterrestrial intelligence: How do you determine whether or not it has a purpose? This
is a question I don’t consider answered. We’ll say things like, “Well, AI will be
intelligent when it can do blah-blah-blah.” When it can find primes. When it can
produce this and that and the other. But there are many other ways to get to those results.
Again, there is no bright line between intelligence and mere computation.
It’s another part of the Copernican story: We used to think Earth was the center of
the universe. Now we think we’re special because we have intelligence and nothing else
does. I’m afraid the bad news is that that isn’t a distinction.
Here’s one of my scenarios. Let’s say there comes a time when human
consciousness is readily uploadable into digital form, virtualized and so on, and pretty
soon we have a box of a trillion souls. There are a trillion souls in the box, all virtualized.
In the box, there will be molecular computing going on—maybe derived from biology,
maybe not. But the box will be doing all kinds of elaborate stuff. And there’s a rock
sitting next to the box. Inside a rock, there are always all kinds of elaborate stuff going
on, all kinds of subatomic particles doing all kinds of things. What’s the difference
between the rock and the box of a trillion souls? The answer is that the details of what’s
happening in the box were derived from the long history of human civilization, including
whatever people watched on YouTube the day before. Whereas the rock has its long
geological history but not the particular history of our civilization.
Realizing that there isn’t a genuine distinction between intelligence and mere
computation leads you to imagine that future—the endpoint of our civilization as a box of
trillion souls, each of them essentially playing a video game, forever. What is the
“purpose” of that?
192