Anil Ananthaswamy: AI’s Nobel Moment

SPEAKERS

Sarah Webb, Anil Ananthaswamy

Anil Ananthaswamy  00:00

You get the feeling that we’re witnessing a very, very epochal event in human history. There’s something happening, and no one’s quite clear about how to categorize it. And people have said that this must be how it must have felt in the 1920s when physicists were coming up with quantum mechanics, when they were aware that things are happening in physics that they couldn’t quite explain, and literally, they were witnessing the birth of an entire new field of physics.

Sarah Webb  00:28

That’s Anil Ananthaswamy talking about artificial intelligence and large language models. He’s a science journalist and author and my guest on this episode of Science in Parallel. I’m your host Sarah Webb, and we’re wrapping up our creativity in computing series with two episodes discussing AI and the 2024 Nobel Prizes in Physics and Chemistry.  [theme music]

Sarah Webb  00:55

John Hopfield and Geoffrey Hinton shared the physics prize this year for building the foundations that made machine learning with artificial neural networks possible. And the chemistry prize highlighted AI’s most notable science achievement so far, helping researchers predict protein structures and design new ones. Anil Ananthaswamy is the author of Why Machines Learn: The Elegant Math Behind Modern AI, which was published in July. He is also journalist in residence at the Simons Institute for the theory of computing at the University of California, Berkeley. While working on his book, Anil spent several years diving into the mathematical and historical context of artificial intelligence. We talked about his book, his assessment of the 2024 Nobel Prizes, the significance of this moment in scientific history and the challenges it brings.

Sarah Webb  01:52

Anil, it is great to have you on the podcast.

Anil Ananthaswamy  01:55

Thank you very much for having me. Sarah,

Sarah Webb  01:57

I want to set the stage for this conversation by talking a bit about your book. Can you tell me a little bit about this book, why you wrote it, and how it came together.

Anil Ananthaswamy  02:07

The book started in earnest sometime in late 2020. So I used to be a software engineer a long, long time ago, before I became a journalist, and for the longest time, I was writing about physics and neuroscience and things other than technology because I used to be in technology, and I wasn’t particularly keen on writing about it. But then sometime around 2017, 2018, I noticed that I was beginning to write more and more stories that involved machine learning and AI. It was unavoidable. It was all over the place. And every time I would interview my sources to understand the machine learning aspects of the story, I think the software engineer part of me got really interested and wanted to kind of get my hands dirty again, so to say.

Anil Ananthaswamy  02:58

And so what happened was, sometime in 2019 when I started doing the Knight Science Journalism Fellowship at MIT, I went back to my roots. I literally went back to school, sat with teenagers in MIT’s Python programming classes, and taught myself coding again, and started playing around with trying to build deep neural networks based systems, very simple things. For me, very complicated at the time, but for people in the field, it would have been trivial. Got interested in machine learning as a field, and somewhere decided that I needed to understand the basics even more than I did just by programming. So started doing courses, and this was during the pandemic. I was stuck, you know, in my apartment all by myself, you know, listening to these machine learning lectures and got fascinated by the mathematics of all of that. I think there was some very beautiful stuff that I was learning. And then the writer in me woke up again, saying, oh, I need to communicate these ideas to my readers. And I ended up pitching a book about the mathematics, or the mathematical history, of machine learning, to my editor with the desire to actually have a book full of equations. I wanted to write something that was technical and yet had narrative, journalistic storytelling also. So that’s how it began.

Sarah Webb  04:20

Your book came out over the summer, right? And then the physics Nobels are announced this year. This was the Nobel year for AI; it’s hard to call it anything else. What was your initial reaction when you heard the news?

Anil Ananthaswamy  04:35

I was very pleasantly surprised. And in a sense, it didn’t seem like a stretch to me. You know, for instance, one of the Nobels you know, went to Hopfield for Hopfield networks. And if you look at the chapter in my book, it’s actually called “With a Little Help from Physics” because John Hopfield did his work and was very heavily influenced by his physics background. He used to be a condensed matter physicist first, and he took a lot of ideas from condensed matter physics, the Ising model of magnetic materials, to inform his work when he designed his Hopfield networks. I was pleasantly surprised in a nice way. I thought it was wonderful to see that.

Sarah Webb  05:14

Let’s just lay the groundwork here. What was the physics behind this prize and its contributions to artificial intelligence?

Anil Ananthaswamy  05:23

John Hopfield’s work was very strongly tied to his physics background, so maybe we can start with that first. You know, Hopfield used to be, like I said, a condensed matter physicist. He felt like he had run the course in terms of making contributions to physics and was looking for other fields in which to make contributions. He then did some seminal work in trying to look at the dynamics of biochemical reactions. And for a while he was working on that. He was still, still looking for a big problem to solve, problem in all caps, as he likes to say. And then he found something which was in neuroscience. He figured out that there was a problem about associative memories that actually he could bring his knowledge about physics to bear on that problem. Like, how did a neural network store a memory and then recall that memory based on some fragment of the original information? For instance, if you have experienced something where there was a strong smell involved, or a very familiar song involved, and that becomes part of your memory somewhere, and then you just sometime later, just have a hint of that smell enter your experience, and that entire memory gets recalled. So that’s associative memory.

Anil Ananthaswamy  06:35

And Hopfield figured out that his work with the dynamics of biochemical reactions, his knowledge of the Ising model of magnetic materials could be used to construct artificial neural networks that were able to store these associative memories. And the way he designed them was that if you stored a memory, then that stored memory represented, quote, unquote, the lowest energy state for that network. And then if you perturbed that network, which means that you corrupted the memory, then the dynamics of the network were such that it would find its way back to its energy minimum. And when it was at the minimum, if you just read out the outputs of the neurons, it was as if you had retrieved the memory. So he essentially built this network, and it was, like I said, very strongly influenced by the Ising model of magnetic materials. And then the Hopfield networks were deterministic networks.

Anil Ananthaswamy  07:28

And Hinton, Terry Sejnowski and others then took this idea and they built something called Boltzmann machines. And Boltzmann machines bring stochasticity into the picture, and they are based on this idea of Boltzmann distributions. And in Hopfield’s case, the networks were storing memories and then were able to retrieve the memories when perturbed by dynamically finding their way down to the energy minimum. In the case of Boltzmann machines, what the machines were learning were probability distributions over data. So if there was some data on which they were being trained, they were not remembering the data itself, or it was not the memory of the data, but rather, they were trying to model the probability distribution over that data. So they were stochastic in nature, very strongly influenced by the thermodynamics of Boltzmann distributions and things like that. So there was a lot of physics, or at least concepts from physics, that were involved in these networks.

Sarah Webb  08:29

You’ve had a chance to survey this whole history of artificial intelligence up to this point. Which pieces do you find especially creative?

Anil Ananthaswamy  08:38

There are many amazing things that have happened in machine learning. One of the things that people who have come into AI recently think that AI is only deep neural networks, and that’s not the case at all. Machine learning has been around for decades, and when, when it began in the late 1950s and early 1960s, we had these things called single-layer neural networks that were kind of good at solving some simple classification problems where you had one set of data that represented, let’s say, images of dogs, and another set of data that represented images of cats. And as long as these two clusters of data were fairly well separated in some high dimensional space, these single layer neural networks could find a linear boundary to separate the two, and you could then classify a new image as being that of a dog or a cat based on which side of the divide it fell on. But these were technically quite simple, in the sense that the moment you had problems where the boundary that separated the data was nonlinear, you couldn’t solve the problem anymore using single-layer neural networks.

Anil Ananthaswamy  09:46

So for a long time, research in neural networks kind of faded away because people couldn’t quite figure out how to build multilayer neural networks. So in the meantime, there were lots of other developments in machine learning that were quite amazing and very seminal for the field. And for instance, sometime in the 80s and 90s, there were these machine learning models called support vector machines, which are extremely amazing in the sense that when you’re trying to do a classification problem in some, let’s say, 100-dimensional space, and you have a cluster of data points that represent cats and the cluster of data points that represent dogs, but in that 100-dimensional space, let’s say you cannot find a linear boundary, that the separation is nonlinear, but you want to use your linear classifier, because they work really well. And so what support vector machines do is they, first of all, are capable of finding an optimal linear boundary between two clusters of data. But in this example we’re talking about, if you have 100-dimensional space and you cannot find the optimal linear boundary, you have the ability to somehow project these data into higher dimensions, in even infinite dimensions, where you’re guaranteed to find a linear boundary. In that higher dimensional space, you can find an optimal linear boundary.

Anil Ananthaswamy  11:08

But the creative part is that when you enter these high dimensions, computation can be extremely expensive, because you’re doing dot products or vectors and things like that. And you know, if you end up in million-dimensional spaces, or even higher or infinite dimensional spaces. Computation can be extremely difficult and expensive, and they use, then something called kernel methods, or these kernel machines, where you project the data into high dimensions, but the actual computation is done in the lower dimensions. So you can literally do your linear classification in high dimensions, but the computation that you’re doing still remains anchored to the low-dimensional space that you started off in. And that was extremely creative. That sort of stuff, when you look at it and you see how powerful it is. And this was all, you know, not neural networks. All of that stuff happened in the 90s.

Anil Ananthaswamy  11:58

I think, when it comes to deep neural networks, and the way we train them probably the most creative thing that has happened in the last maybe decade or so is also what has led to large language models, and it’s this idea of self-supervised learning. So so far we’ve been talking of things like clusters of data, where, you know, one cluster represents images of dogs and the other cluster represents images of cats. Somebody has to label these data points as cats and dogs. Now, what if you just had a whole bunch of images and you wanted to learn something about the features that exist in those images, but no human has annotated those images as being that of cats or dogs? There’s this whole field of self-supervised learning where you take in some data, let’s say an image, and you mask off some portion of the image, let’s say 20% of the image, and then you ask your neural network to learn to predict the missing part of the image.

Anil Ananthaswamy  12:57

You kind of know what the missing part is, because you have the original image, but you’re asking the neural network to learn by masking different parts of the image and asking it to keep trying to recreate the original image. It actually learns features that exist in the image or in the data that wouldn’t be possible if all you told it in the beginning was, oh, this is a cat. This is a dog. Here you’re asking it to learn something much richer about the image, and this, this whole idea is called self-supervised learning, where you take data and you mask some portion of the data, and you ask the machine learning model to learn how to recreate the masked portion of the data.

Anil Ananthaswamy  13:33

Large language models are doing exactly that. You take a sentence. You mask the last word of the sentence, and you say, Okay, tell me what that last word should be. And of course, the model in the beginning makes mistakes, and you slowly tweak the model so that it learns how to predict the masked word. And if you can do that for one sentence, you can do that for every sentence on the internet. And eventually, if you do that for all of the written text that exists on the internet, you have essentially built a model that has learned the statistical structure of human written language and also the knowledge that is contained therein. And this is completely without requiring humans to annotate data. It’s just, you know, just you take the data that’s out there, hide some portion of it, ask the model to learn how to predict what is hidden, and as it gets better and better at predicting that masked part of the data, it essentially learns something very, very important about the statistical structure of the data that you presented.

Sarah Webb  14:25

There was this period where we didn’t have the computational power to look at multilayer models, and the hardware has now advanced, too. And so I want to get your take on where the ideas and the machines to help carry them out play off of each other here.

Anil Ananthaswamy  14:44

That interplay has always been very important. For instance, we talked about how single layer neural networks in the 60s couldn’t solve certain kinds of problems, and they needed multilayer neural networks to be able to solve such problems, these non. Linear boundary problems, but we didn’t know how to train them. So it took until 1986, mid 1980s, for people to figure out, you know, and this is also work that Hinton was very strongly involved in, in terms of coming up with an algorithm that could be then used to train these multilayer neural networks. And that algorithm is the workhorse of today’s deep neural networks, and it’s called the backpropagation algorithm. But even though we had that algorithm in the mid 1980s, people didn’t know what to do with it because it was computationally extremely heavy to train these neural networks. All we had were CPUs, central processing units, and they were not good enough to crunch the numbers in order to be able to train these networks, because a lot of the training, pretty much anything to do with deep neural networks, involves manipulating vectors and matrices, and it’s just these huge arrays of numbers that you have to multiply and manipulate.

Anil Ananthaswamy  15:59

And through the 1980s and through the mid 1990s even though the algorithms were there, we just didn’t have the compute to pull it off. Someone eventually, and again, this was Hinton’s team and others simultaneously who realized that we could use GPUs to do the number crunching. And these graphical processing units had actually been designed for video gaming. If you think of what’s happening when you’re updating your screen, computer screen, that’s essentially a huge matrix of numbers. So there’s a lot of matrix manipulation that is going on in order to do the graphics updates for a video game. And people realize that they could just take that GPU and co-opt it for doing the training of the neural networks. So once that happened, that was one very big piece of the puzzle. Now we had the compute that you needed to be able to do the calculations.

Anil Ananthaswamy  16:52

The other thing that was missing was data. Neural networks are extremely data hungry. They need a lot of data to train. And we just didn’t have that in the 1990s and even the early 2000s and then, you know, sometime towards 2010, people started building these massive sort of data sets: ImageNet being one of the classic ones. And so it was the confluence of the ability to use GPUs to have the algorithm, the backpropagation algorithm to train your neural network, and the availability of training data, all of which came together. And I think it was 2011 or 2012 when Hinton’s team came up with AlexNet, which won the ImageNet competition. And basically that was the first sign that neural networks were onto something.

Sarah Webb  17:38

I also want to talk some about the chemistry prize, because I felt like these two things were hand in hand. You had the physics with the fundamentals, and then chemistry with this obviously explosive application of being able to understand proteins in a different way. How do you see the implications of the chemistry prize and and the timing, one right after the other?

Anil Ananthaswamy  18:00

Yeah. I mean, a lot of people have pointed out that the timing was very interesting. On one hand, the physics prize is being given to people who did some basic work in neural networks, and then the chemistry prize is about how the same neural networks are now being used to solve an applied chemistry problem, protein folding problem, and the fact that they happen in the same year, it’s been pretty amazing. I think the chemistry prize just tells you that neural networks are here to stay, that machine learning and deep learning is going to be used to solve some very, very big problems. And this was an example of a very difficult problem that people in chemistry and biology had been tackling for a long time, and along comes this machine learning solution that helps you deal with otherwise intractable problem. So it’s just a sign of the times. I think we’re going to apply machine learning, deep learning, in particular, to problems that have proven difficult otherwise that once we have enough data on which to train these systems, we will be able to solve problems in ways that we hadn’t anticipated. And I think this is just the first step. We’re going to see many more of these things. I don’t know about Nobels, but we’re certainly going to see solutions like that.

Sarah Webb  19:13

I spend a lot of time talking with people who are thinking about artificial intelligence, some using it more than others. I want to get your take on the significance of this moment for research. Do you think this changes the game for how much people will be curious about using AI? Want to use AI?

Anil Ananthaswamy  19:37

I think the Nobels certainly have shown a light on this whole issue, but scientists were already onto it. They might become an incentive to pay more attention to this way of doing science. But scientists, whether it’s in physics or chemistry or material science, quantum physics, people are already using machine learning and deep learning to help them with their science. Yes, this will certainly bring more attention to the fact that you can use AI to do your science. And questions are being asked about, what is AI really being used for? At this point right now, AI is more a tool that helps you do better science. There’s a big question about, when will AI become good enough to act as a scientist itself? And at this point AI is a long way from being able to do that.

Anil Ananthaswamy  20:28

But people are already thinking about such things like, will it be 50 years or 100 years from now when AI will be able to come up with theories that humans couldn’t? Possibly. We don’t know that for a fact, but the advances are happening along multiple prongs. One is just using AI to do your science. So for instance, if you want to design a new kind of quantum optics circuit to generate new kinds of quantum states, you can use AI to search through an enormous space of possible solutions that humans may not be able to and it might find an optimal circuit for you in ways that we couldn’t have found. So there are lots and lots of ways in which machine learning is already being used to do the science. But there’s also this interest in trying to figure out whether we can actually build AI that itself becomes capable of doing science the way humans do where it hypothesizes, theorizes, tests and comes up with new theories, etc. We are still some ways away from that, but that’s also one big focus of where science is going with AI.

Sarah Webb  21:32

So what the role of the human and the interaction between the human and the machine in terms of asking and answering these questions?

Anil Ananthaswamy  21:41

Yeah. I mean, it’s going to be a big question, right? I have talked to scientists who have wondered about the whole question of why we do science. Some scientists do science because it brings them joy, and there’s something very personal about the act of doing science. Now, if you use an AI to do something that is bringing you joy, and you’re just giving that off to the AI, it raises existential questions for scientists, certainly. So there are going to be a lot of different responses to this whole question of, how do scientists and AI interact, and what is the role of the human scientists versus the role of the AI scientists? And I think different scientists are going to have different answers to that question about that interaction.

Sarah Webb  21:45

As we look forward, what trends are you watching? What do you find most interesting? Is there anything that we haven’t talked about that you think is important to mention about this moment in scientific history?

Anil Ananthaswamy  22:35

I think there is something very significant happening with AI. So anytime I’ve been part of conferences where researchers are discussing, for instance, what’s happening with large language models, it’s very, very clear that people are not quite sure how to explain the behavior of these models. So we know the mathematics involved in building these systems and training them, but the mathematics necessary to explain their behavior is not at all clear. And so when you sit in these conferences, you get the feeling that we’re witnessing a very, very epochal event in human history. There’s something happening, and no one’s quite clear about how to categorize it. And people have said that this must be how it must have felt in the 1920s when physicists were coming up with quantum mechanics, when they were aware that things are happening in physics that they couldn’t quite explain, and they literally they were witnessing the birth of an entire new field of physics.

Anil Ananthaswamy  23:35

You get that feeling in these AI conferences, especially in AI conferences that are grappling not with the building of these systems or the users of these systems, but conferences that are trying to grapple with a more basic understanding of why these models are working the way they do. And yet it’s also becoming pretty clear after the initial hype has kind of died down, about large language models, for instance, is that these systems are not really reasoning the way humans reason. Even though their behavior is suggestive of extraordinary abilities to do reasoning and solve math problems and all these things, if you look at the behavior carefully, it turns out that they’re not really reasoning the way we reason. They are very, sophisticated pattern matching machines, and they’re leveraging this extraordinary ability of finding very sophisticated correlations in data and using that to extract some amazing information and answer some very, very difficult questions. But they’re still not doing what we do as humans.

Anil Ananthaswamy  24:39

So in terms of what I’m looking at or watching out for, there’s this sense that we are one or two breakthroughs away from getting to AIs that are as capable as humans are in terms of being general-purpose problem-solving machines. And obviously we don’t know what those breakthroughs are going to be like. So something like what happened in 2017 When we got the transformer architecture that was proposed in a single paper that came out of Google, I think it was called “Attention is All You Need.” And that was basically the foundation for everything we are seeing now with large language models. So maybe there are a couple of such breakthroughs on their way. We don’t know and when they might happen, but that is a space that I’m watching very keenly.

Sarah Webb  25:23

I’m going to ask you the more general question, but it’s a question that I have been asking all our guests on this creativity and computing series. I’m wondering what creativity means to you.

Anil Ananthaswamy  25:34

What creativity means to me?

Sarah Webb  25:37

Yeah, to you.

Anil Ananthaswamy  25:39

It’s hard. I think human beings are also machines, very, very sophisticated machines. And I personally think that our creativity is overblown. People argue that machines are never going to be as creative as us. But if we are also machines, which I think we are, very sophisticated ones, way more sophisticated than the machines that we’re building, but machines nonetheless, then I think creativity, to me, would be this ability to you encounter a certain amount of information in your immediate environment, and somehow you abstract out some knowledge, you abstract out the patterns that you’ve seen in the data, and then you’re able to somehow compose new things based on the abstractions that you’ve learned. Whether you do that consciously or whether that happens subconsciously, is a different story. But something about our brains are capable of doing that in ways that leads to new insights or new paintings or new music or it doesn’t matter what it is. That would be creativity to me. And I think machines are going to get there. They’re not there yet right now. They don’t compose abstractions in the way humans do. So there’s something missing. There’s no reason to think in principle that they won’t get there too. Yeah, which is both exciting and scary.

Sarah Webb  26:57

Well, it will be interesting, right? I think it goes back to my question about how we interact with them, it seems like we’re in a foundational place where how we work with the built machines around us is a really big question.

Anil Ananthaswamy  27:13

I agree. And because I also write a lot about neuroscience and think a lot about all these things in evolutionary terms, I’m not surprised that the intelligence that has evolved in us over evolutionary time is then going to produce new intelligence, maybe even smarter things than we are. And that to me, seems just another step in the chain of evolution doing its thing. So maybe in the future, we will look back and we won’t recognize ourselves. We don’t know. We don’t know how this is going to pan out. A lot of it will depend on us as humans and the decisions we make about how we choose to deploy these machines, what kind of machines we choose to build. But it seems like a natural consequence of intelligence evolving and taking its course. So there will come a point where biological machines and these artificial machines will not be as indistinguishable as they are today.

Sarah Webb  28:11

We’ve gotten to this point of talking about this interface of biology, and I have heard people talk about the ethical space of this, and some people have equated it to CRISPR and some of the ethical discussions about cloning in the last decade. Do you think that’s a reasonable analogy for thinking about how we want this field to move forward in terms of all of these big ethical questions we’ve been talking about?

Anil Ananthaswamy  28:36

I think the ethical questions are probably even more stark when it comes to AI because we’re dealing with systems that are going to have enormous power, and the speed at which things are happening is also quite frightening. Sometimes it’s very hard to keep up with how quickly things are changing. But the ethical questions are no different than the ones that we confront in society regularly. It’s just that the speed at which we have to confront them, and the speed at which we have to come up with answers has changed.

Anil Ananthaswamy  29:05

But eventually it’s about human societies being good societies. And if we’re not good to each other, regardless of AI, we’re doing terrible things to each other without AI in the picture, AI just makes things worse. If we bring the nasty human tendencies to bear on society, and we use AI to just further those, the problems that we already have. I think it’s just going to get worse. We’re just going to accelerate the nastiness. So it seems like the ethics problems is a problem for us as humans. We need to figure out how we’re going to cope with that, regardless of AI, and then realizing that AI just makes things harder, because it’s so much easier to deploy bad stuff. It’s so much easier now for bad actors to use these technologies for evil or dangerous things. So we need to be really wary of how we deploy these technologies, but it’s also a time where we have to look at ourselves. Say, why are we the way we are, regardless of AI?

Sarah Webb  30:03

What I hear you saying is that we have to be good stewards of technology, no matter what the technology is and no matter what we’re doing.

Anil Ananthaswamy  30:09

Yeah, we have to be good stewards, with the caveat that this technology now is way more powerful than anything else that has come before. So we have much less time to adapt and to cope with the changes that are coming. So we probably have to be more agile in how we shepherd this technology going forward.

Sarah Webb  30:27

Is there anything you think we’ve missed?

Anil Ananthaswamy  30:29

I think one of the things that made me want to write my book in the first place, and also one of the things that gets forgotten in all this, is some of this stuff is actually quite amazing. It’s when you think about human intelligence that has then brought about another kind of intelligence, not yet at the level of human intelligence, but nonetheless, something very smart in terms of the machines we are building. And there is so much lovely math. The fact that all of this has calculus and linear algebra and probability and statistics and things that you learned in high school and first year undergrad, and never thought we would have any bearing on anything. Turns out that the machines that we’re building today are grounded in math that was done by Newton and Leibniz and all these people going back centuries. That’s quite beautiful. So I don’t think we should completely forget just how amazing it is that we are at this place in history, while we are cognizant of the dangers that are afoot.

Sarah Webb  31:24

That seems like a wonderful place to wrap up. Thank you, Anil. This has been such a pleasure.

Anil Ananthaswamy  31:29

Thank you. Thank you for having me. It’s been a pleasure, too.

Sarah Webb  31:33

To learn more about Anil Ananthaswamy, his book Why Machines Learn and this year’s Nobel Prizes, check out our show notes at science inparallel.org. Stay tuned for our next episode, which will feature conversations with computational scientists about this year’s AI Nobels and their impact.

Sarah Webb  31:54

Science in Parallel is produced by the Krell Institute and is a media project of the Department of Energy Computational Science Graduate Fellowship program. Any opinions expressed are those of the speaker and not those of their employers, the Krell Institute or the U.S. Department of Energy. Our music is by Steve O’Reilly. This episode was written, produced and edited by me: Sarah Webb.