Trelograms #18 — What Do You Worry About?

I just finished listening to Life 3.0: Being Human in The Age of Artificial Intelligence, by Max Tegmark. This brilliant book which i encourage everybody to read argues that we badly underestimate the existential risk posed by the imminent onset of artificial superintelligence, and calls for a global and collaborative safety research program.

From the book:

“Not only can non-living matter have goals, at least in [the sense that it was designed to accomplish a goal], but it increasingly does. If you’d been observing Earth’s atoms since our planet formed, you’d have noticed three stages of goal-oriented behavior:

  1. All matter seemed focused on dissipation [of energy].
  2. Some of the matter came alive and instead focused on replication and subgoals of that.
  3. A rapidly growing fraction of matter was rearranged by living organisms to help accomplish their goals.

(…) not only do we now contain more matter than all other mammals except cows (which are so numerous because they serve our goals of consuming beef and dairy products), but the matter in our machines, roads, buildings and other engineering projects appears on track to soon overtake [in weight] all living matter on Earth.”

So, if we overcome our anthropocentric bias and understand human intelligence as simply one of the many possible instantiations of intelligence in our Universe, then all it takes for superintelligent machines to be a serious concern is for them to have goals misaligned with our own, and we might end up being treated by those machines much like we treat ants today — we largely don’t care about whatever might be their goals. Importantly, those machines need not at all be rebellious — in fact, they don’t even need to be ‘aware’ of what they’re doing — especially not subjectively so.

That worries me.

How about you — what do you worry about?


On the photo: a pig head at a market in L’viv (January ’18)


got no time or money to travel? — i beg to differ!

I am hard at work on a budget long-term traveling tutorial consolidating all i’ve learned over my past two years on the road — sign up for the Not Mad Yet mailing list, and be among the first to hear about it when it’s out!

You will also get this weekly Trelograms series of inspiring one-minute reads delivered straight to your inbox two days before they’re published here on the blog 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s