Tim Hwang: It’s 2027. Has an AI-generated work won the Nobel Prize for Literature? Chris Hay is a distinguished engineer and the CTO for customer transformation. Chris, welcome to the show. What do you think?

Chris Hay: Absolutely. And while we’re at it, AI is going to win a few Oscars and an Emmy as well.

Tim Hwang: Okay, all right. Next up, Edward Calvesbert is a vice president of product management for the Watson X platform. Edward, welcome to the show. What do you think?

Edward Calvesbert: No way. I don’t think the Nobel institution will allow it.

Tim Hwang: All right, cool. Well, with a difference of opinion like that, you know it’s going to be a good show. All that and more on today’s Mixture of Experts.

I’m Tim Hwang, and it’s Friday, which means that it’s time again to take a deep dive with our experts into the week’s news in AI. We’re going to talk about OpenAI’s new DGX B200, a new round of funding for a company called Unstructured, but we’re going to start today with the big news story of the week: AI has been taking the Nobel Prizes by storm this year. In the prize for Chemistry, David Baker, Demis Hassabis, and John Jumper took the prize, with Hassabis and Jumper winning in part for DeepMind’s work on AlphaFold. And then in the Nobel Prize in Physics, one of the founding fathers of modern AI, Jeff Hinton, and John Hopfield, another leading light of the field, won for their work in neural networks.

So Chris, I want to start with you first. What do we make of this? Are the Nobel Prizes basically succumbing to AI hype, or is this the start of something way bigger?

Chris Hay: I love it, actually. I think, well... if I’m being completely honest, the Nobel Prize has been a little theoretical and not hitting with the real world for a while. So actually, recognizing AI and the impact that it’s going to have in multiple fields like physics and chemistry... if we think about the big innovations going forward in the next few years, it is going to be more AI-led. It’s going to be AI and humans in collaboration. How do you distinguish that? Well, is it fair to say that the people who founded AI in the first place don’t get rewarded for that work? Of course not. So I actually think it’s a good thing because this is the cornerstone for the next few years, where AI is going to massively help in these areas. So I’m all for it. Go for it. And while we’re at it, I think my next AI means I’m going to become the MVP of the NFL soon as well, you know? So Aaron Rodgers, watch out. Here I come.

Tim Hwang: I guess, Edward, to bring you into the conversation, Chris has already taken a very strong stance that basically in a few years, AI will be winning every single Nobel Prize award. Edward, there are two questions from what you said in the opening. You said, “I don’t know if the Nobel institution will really allow it.” Does that mean you don’t think it’s deserved, or that you think the institution won’t be into awarding AI its due?

Edward Calvesbert: Yeah, I mean I think... mainly the former. I do think that AI is going to be an incredible tool in almost every aspect of our lives, and it’s going to do amazing good for society, well-being, quality of life, and basically all the things the institution stands for. But I do think the human contribution to those outputs, those deliverables, is really essential. So whether it’s 10% AI or 90% AI, if it’s 100% AI, maybe it goes a little bit too far.

Tim Hwang: Uh-huh, right. Chris, one thing I wanted to do... I take a lot of pride in the fact that Mixture of Experts is a good way for people who might not be reading every single arXiv paper to learn more deeply about the AI space. It’s interesting—you can often get lost with all the stuff happening at the enterprise layer around AI. I think there are a lot of people listening who may not even know who Jeff Hinton is. I’m curious if you feel comfortable giving our listeners a quick explanation for why someone like Hinton is so important to the field and what exactly he contributed here.

Chris Hay: No, absolutely. So I think the first thing I would say is Hinton really is considered the OG, the GOAT, the godfather of AI—which I love. He’s been doing AI, machine learning specifically, for a very long time. He was there before it was cool, right back in the 1980s.

Tim Hwang: Really uncool, basically, right?

Chris Hay: Yeah, very uncool, exactly. But if we think of the modern foundations of what we have today—things like deep learning—that all comes down to these really deep, massive neural networks with billions, even trillions of parameters. I’m not going to go into the massive details, but if we look at the work Hinton has done, even as far back as 2011 when AlexNet came out (and ILSVRC was part of that), that was when the deep learning revolution really kicked off. It was the first CNN on a GPU for training against images. If we go further back in time, look at work he did on things like backpropagation, which is a key cornerstone of what we do today with deep learning. All of this goes back to the 80s. Jeff Hinton was doing this when it was uncool. If he hadn’t done that work, we wouldn’t be where we are today. So I think you have to recognize that fact. And as I said earlier, the impact AI is having and will have in the future is incredible. So I think the impact it’s having in these different fields... he should be recognized for the work he’s done.

Tim Hwang: Right, even in physics. I know some physicists on my Twitter feed were grumpy, like, “What’s this computer science person doing here?” But ultimately, you’re saying this is significant enough that it should be recognized in this context.

Chris Hay: Absolutely. And I think it opens things up. It’s a good thing for physics as well. You don’t want to be seen as this boring thing—“Here’s a bunch of formulas, oh look, here’s another telescope in the sky.” You know what I mean?

Tim Hwang: This is stuff...

Chris Hay: Exactly. AI is impacting every field, and therefore I think it’s a really good move by the Nobel Institute.

Tim Hwang: Edward, I’d love to bring you into this. One thing I love about Jeff Hinton is how down-to-earth and open he is about his views. There’s a great quote the Nobel committee posted on Twitter about how he was in a low-rent hotel room when he got the news and had to reschedule medical appointments to deal with the win. One thing Hinton has been strong on in the last few years is warning about the risks of AI, and people take him seriously because he’s been at this for so long—the “GOAT hipster of neural nets,” as Chris explained. I’m curious how you think about that. As a leading light in the field, do you take his dark warnings seriously? Do you think he’s on the right track? He made it a centerpiece of some interviews around this prize, so I wanted to talk about it before we move on.

Edward Calvesbert: Yeah, I mean I think he’s raising the warning just to make sure that voice is always considered, that risk is always part of the math that enterprises, individuals, and institutions do when applying AI to a particular problem or use case. I really think that’s what it comes down to. When we think about risk assessments or AI governance, it’s really at the intersection of the technology and the use case. It’s not the same thing to apply AI to creative writing, as we mentioned, as it is to do credit underwriting for a bank, or a hiring decision for an organization. These are very different use cases with very different impacts on individuals and society, and they pose totally different risks. So it’s really not just about the technology; it’s what the technology is being applied to that needs to be assessed. The more this technology makes its way into national security, defense... obviously, that’s a much different consideration than poetry.

Tim Hwang: Yeah, for sure. Are you hearing this... because you work close to the metal, with the Watson X platform that customers use and rely on. Sometimes I feel discussions about AI being dangerous take place in a totally different domain. It sounds like you’re suggesting that day-to-day, you’re hearing from customers and the market that these risks and concerns are things people are thinking about.

Edward Calvesbert: Yeah, I mean they’re not existential risks, but there are definitely risks to brands, business risks, regulatory compliance risks. Managing these risks is definitely one of the top considerations that is acting as an inhibitor to more wide-scaled adoption of the technology. It’s something you can’t really do after the fact because so much of managing this risk is the end-to-end lifecycle. It starts with the data that goes into the model, what model you selected, how you customized and tuned it, all the way to monitoring, guardrails, separation of duties between development and deployment. You have to start thinking about these things from the beginning; if you don’t, they become a wall or a real obstacle to reconstitute post-fact. That’s what we’ve been working with clients on, with that approach, and that’s what’s leading to some of these use cases making their way into production. I wasn’t being hypothetical about credit risk underwriting and hiring decisions; these are real-world use cases where the risk is being assessed and mitigated in order to implement these workflows.

Tim Hwang: Yeah, for sure. Chris, do you want to get a final shot here? What do you think about Hinton’s late-career turn as a voice of warning around these technologies?

Chris Hay: I like to think of this as the difference between waterfall and agile, which is probably a weird way of putting it. If we think of waterfall projects, nobody does them anymore because we realize we are too dumb—and I mean that in the nicest possible way—to figure out every requirement in advance and plan everything because the world is too complicated. I sort of feel that way about AI risks. I think we are too dumb to figure out every single risk and exploitation and get ahead of everything in advance with a pre-planned approach. Therefore, I think like a software project: we need to be agile. You need to experiment and then discover, in a safe and controlled fashion, what those risks are and let them evolve. That means we’re going to do dumb things. We really are. But in the process of doing dumb things—like sticking your fingers in a wall socket—you realize, “Oh, I better not do that,” and then you put safety things in place. I’m not saying we should go that far with AI, but I hope history tells us that in human existence, we’ve done enough dumb things that we’ll do enough small dumb things before they become catastrophic dumb things. So I think a little bit of agility will help us discover that stuff. We need to have control, but I don’t think we’re all going to blow ourselves up because I think we’re going to do much dumber things much earlier. That is my opinion.

Tim Hwang: I’m going to move us on to our next topic. There was an incredible photo that OpenAI put out on social media of the team celebrating their receipt of the new NVIDIA DGX B200. It’s a great photo because you can see everybody is so jazzed to be standing next to this fresh piece of compute. It’s like Christmas morning; people are thrilled to get this computer in their hands. It’s a nice hook to talk about this next generation of platform that NVIDIA is rolling out, which is having a material effect on the compute market. There’s a great chart I saw earlier in the week about how prices for NVIDIA H100s—last season’s must-have hardware—are dropping suddenly as these new boards come available.

So, maybe Edward, I’ll turn to you first. Is what we’re seeing here just more speed? One point of view is it’s Christmas morning because it’s cool to stand next to an F1 racing car for compute. But is what we’re getting largely just faster? If not, what’s different about it?

Edward Calvesbert: Yeah, I mean I think it connects back to the first topic we talked about: the evolution of this technology, trying to build it in a way that somewhat models how our brain works. This kind of almost infinity—I know that’s a big word—of nodes and connections and relative strengths between them. So it’s not just speed; it’s scale. The capacity to consume more data and have more nuanced relationships within that data. I’m not a hardware expert, but I definitely think it’s a great time in technology when hardware matters again. We go through cycles where hardware becomes totally commoditized, then it matters again, and eventually becomes commoditized again. We’re definitely in a stage where it matters. I think that’s a signal that the innovation frontier is active and moving rapidly, and I think that’s all very positive.

Tim Hwang: Yeah, I mean, Chris, it’s stunning. I was talking to a friend working on these clusters, and he said the hardware is moving so quickly they can only afford to do one big training run on a cluster before almost immediately starting to build the next cluster for training. As someone who researches this space, where is this all going? Is the cluster just going to get bigger and faster? Does this top out at some point? What’s the trend in the next 12 to 24 months?

Chris Hay: I think there are a couple of trends going on. I might have said this on another episode, but it’s almost like following the Bitcoin trend. Everything started on CPU, then moved to GPUs, then to FPGAs, and then to ASICs. You went from compute being CPU-bound to GPU-bound, then to custom hardware. We’re kind of seeing the same thing again because people need to bring the cost of compute down, the cost of training down. You’ve got bigger models to train, but I think the bigger thing is on inference. You’ve got to run these models at low cost and high speed. If there’s one criticism of NVIDIA, it’s that the speed (tokens per second) and cost on these GPUs are quite expensive. You’ve seen this in the marketplace already; that’s where folks like Groq have been coming in, releasing chips that go really fast. And then IBM has its NorthPole chip, and Google has its TPU chip. Everybody’s trying to bring down the cost of inference because if you’re running massive models on the cloud, you need it to be as cheap and fast as possible. The big thing with these new NVIDIA boxes is yes, training speed is much faster, but if you look at the chart, the cost of inference and the speed of inference came down massively. They’ve obviously put a focus on that because they know if they don’t improve inference speed and cost, all these other providers will start eating their lunch. So this push and pull between general-purpose GPUs and custom chips is really important. But again, from a training perspective—different from inference—everybody’s just focused on, “I need to get the biggest, fastest model out really quickly.” So you throw away your last card and put in the latest card. There’s a different dynamic going on. Over time, you’re going to get faster architectures, different architectures; it’s going to get cheaper. These cost/speed/performance ratios are going to change over time.

Tim Hwang: Yeah, the architecture bit is really interesting. One theme that’s popped up on a lot of Mixture of Experts episodes is that customers want smaller, faster models. They don’t want the gigantic, expensive model. So there’s pressure there. But it feels like there are two ways to get there: one is we start marketing just smaller models, lowering demand. The other, which you’re arguing, is the chips get good enough that the cost of inference finally falls for running larger models. The two are in a bit of a race. Predictions on who wins? You can imagine the market might eventually settle and say, “These smaller models do 99% of what we need; we don’t need near-AGI models.” Or, if the cost were cheap enough, you’d still go bigger. I’m curious how you think about that relationship. It’s complex and unclear where it lands.

Chris Hay: Today, I think it’s just going to keep pushing and pulling. We are going to want to run our models on-device. If you think of things like Apple Intelligence, etc., you’re going to need both smaller models and faster compute for a while. Will one win out? I don’t think so. The smaller you can make the models and the chips, the more you can put them on embedded devices, which opens up low-latency scenarios. You even see that this week: Llama 3.2 was out last week, and they released their 1 billion and 3 billion parameter models. Again, smaller models. The big thing is folks are getting really good at taking larger models and distilling them down into much smaller models. That’s going to continue. We’re looking at 1 billion parameter models now, but let’s project forward six months or a year—you’re going to start getting into million-parameter models, and the chips will get faster. We’re just going to go back and forth, back and forth. It’s forever.

Tim Hwang: Yeah. Edward, are you seeing that in the market? It feels like one interesting outcome is a lot of market pressure to have models more on edge devices. Part of the idea of a platform is running it in the cloud, but there seem to be powerful economic incentives pushing us toward on-device. Do you think that’s a real possibility going forward?

Edward Calvesbert: I mean I think it’s going to be “all the above.” We’re the hybrid cloud company, so “edge” to us is definitely a continuum. The data center compared to the hyperscaler cloud is effectively a type of edge, and then you go down to facilities and eventually devices. So yes, it’s going to be all of the above. Finding the right balance is always specific to the use case requirements. What we see a lot right now is clients, to get started, use a big model because that accommodates a very broad range of requirements, use cases, languages, etc. So you prove out the business case with a big model to accelerate. But then when you’re there, you think, “Okay, how can I do this as cheaply and with the least latency as possible?” Then you start to really optimize and customize once you’ve validated the business case and want to scale it with real economics behind it. It’s like using a Swiss Army knife—it gives you flexibility, but eventually you want to use the fit-for-purpose tool to get the job done.

Tim Hwang: Yeah, that’s super interesting. I never thought about it as a lifecycle. It’s interesting that for the pilot, you use the biggest, baddest model for optionality, and then as the organization tunes the use case, it gets more discrete, smaller, and optimized for cost. Very interesting. Is this the time to mention agents? I realize we haven’t said “agents” in this episode. We’re not contractually obligated, but if you want to, Chris, go for it. Give the final hot take before we move to the last topic.

Chris Hay: This is needed for agents! Because agents are going to be highly specialized; they’re going to work together and need low latency, etc. So actually, smaller models, running on-device, whether in a data center or on a device, running in different locations, is 100% necessary for this agentic world we’re moving into. So... it’s a good thing. Agents, for sure.

Tim Hwang: Agents! A lot more to get into there, for sure. So for our final story today, I wanted to talk about a company called Unstructured, which recently closed a USD 40 million round led by IBM and NVIDIA, among other prominent investors. What’s most interesting about Unstructured is it focuses purely on transforming unstructured data into structured data—not something you’d normally think of as a USD 40 million investment. Edward, if I can bring you in, can you resolve that mystery? Why is unstructured data important, and why is structuring it incredibly important for AI?

Edward Calvesbert: Yeah, well, unstructured data is most data nowadays. I think the most relatable, usable type of unstructured data today for LLMs is document data. It could be content on the internet, Word docs, or PowerPoint presentations. Effectively, document data is enterprise knowledge. That’s what runs the world. It’s these documents, this language information, that large language models are built on, trained on, and excellent at processing, summarizing, and making usable. Bringing that enterprise, institutional knowledge to the models is how an organization can make it their own, customize it to the knowledge, language, tone, entities, relationships, and values of their business. To put a model in service of a business or a goal, you need to teach it with your data. That’s what this company focuses on. I’ve met them; they’re very focused, which I think is part of their strength. They’re focused on taking unstructured data from different locations and formats and making it available for the models, particularly in vector stores for retrieval-augmented generation (RAG) as an initial use case—which is almost universal now. But beyond that, identifying relationships in the data for graph RAG, or putting data into structured format to increase the precision and accuracy of queries. I think RAG is very popular and valuable, but it’s already running out of gas a little for the next evolution of use cases. That’s really about continuing to unlock the value of the data in those documents.

Tim Hwang: Huh, that’s really interesting. Can you go into that a bit more? Why is RAG running out of steam? Twelve months ago it was the new hotness, the key strategy for retrieval. What’s missing? What cracks are appearing?

Edward Calvesbert: Yeah, I think it’s a great starting point and essential in most cases. But for example, graph RAG is going to give you the ability to have richer contextualization by identifying non-obvious relationships. If I prompt the model with a certain set of words, it’s only going to limit its ability to reason and retrieve knowledge to that domain. There may be hidden relationships. For example, if I search for something about Facebook but don’t get a response about Instagram, I’m not getting the whole picture. The model might not know Facebook and Instagram are related because those relationships could be non-obvious. The graph RAG pattern gives you strength in identifying non-obvious relationships, providing richer contextualization that is more relevant to the question, even if it’s not asked with those specific words. It’s mimicking how our brain identifies relationships. But even that isn’t perfectly accurate. There’s data about transactions that may have a SKU number or a particular ID with no semantic value—it’s just a bunch of characters, like a license plate. You need that type of data in structured formats and combine RAG or semantic search with SQL, with structured queries. That gives you more accurate responses for questions about transactions or other data important to a business but lacking semantic value in a conversational sense. So you have to complement RAG with a different dimension—structured data. Those are just two examples of how you need to complement classic RAG to make it more accurate.

Tim Hwang: That’s really helpful. Chris, this story made me think about the market for data structuring. We normally think of the supply chain as people who generate data, people who do the training, people who offer models. One link I haven’t thought much about is this layer between raw data and usable data. There are lots of ways to do that. There are companies like Unstructured, a specialized service. You might imagine foundation models themselves become good enough to structure data out-of-the-box. You could imagine synthetic data gets good enough we don’t even need unstructured data. How do you size that up? Do you think synthetic data will get so good that you won’t need data structuring, or will there always be a niche for this kind of business? Where do you think this market is going?

Chris Hay: Oh goody, I get to say the word “agents” again! My favorite.

Tim Hwang: Yes, please do! We’ve got to get a few more in before the episode’s up.

Chris Hay: So actually, I think everything is going to move into a marketplace in the future. I think we’re going to have a marketplace of data, a marketplace of agents, and a marketplace of models. We are going to get more outcome-focused. Specifically on data, I think we’re doing a lot of human work now to curate it. Even looking at companies like Unstructured, they do great work because they take away the complexity of getting your data into vector databases for RAG. It’s really hard; you have to do chunking, you’re constrained by the model’s context window (its short-term memory), you have to work out which data is associated with what, and as Edward said, you need to build relations. You have to get data from an S3 bucket, from here, from there—it’s really complicated. Even though that’s a faster process, humans are still figuring this out, doing transformations, doing ETL pipelines. If I project a bit forward—back to our earlier discussion—where models are smaller, have lower latency, and faster tokens per second, you’ll be able to train these smaller models to do that restructuring work for you. I think you’re going to be in a world where agents are going to help you get your data into a structured format. Once your data is structured, you’ll be able to train your model, and then you’ll loop around in a nice virtuous circle. Will there be a marketplace for this? Absolutely. At the end of the day, people own data. Publishing companies, media companies—they’re all sitting on gold mines because that data is highly valuable, highly creative. There are things that can probably be synthetically generated, like all the math data; you could argue that will be commoditized over time. But human data, especially in creative spaces, will still be highly valued. I don’t see record companies or songwriters giving up their ownership. So that’s going to be the push and pull. We are going to be moving into a marketplace where that soft IP is the big thing that distinguishes companies. One example I like: if you have a model trained on all Spanish legal texts, structured etc., and your model can answer Spanish legal queries better than any general-purpose model... if I’m going into court, I want the model that’s really good at Spanish law, not one with a vague understanding. That’s the difference between getting a large fine or going to jail. So there’s huge value on that locality. I think that will be one of the biggest trends. Models are going to get more specialized. We’re going to have benchmarks for everything you can imagine, Tim. There will be a Spanish legal benchmark, a car parking benchmark... you name it, there will be benchmarks everywhere. We’re just going to be in this massive marketplace of specialization.

Edward Calvesbert: I love the image of hiring an AI agent attorney to defend you in a case. I think that is a feature I can get behind.

Chris Hay: I used it myself! I had an insurance claim. I looked at the document and had no clue what it meant—it was a medical condition thing. I ran it through an LLM; it gave me the key points. I brought it to the insurance company, they paid out, and I thought, “This could go somewhere.” That’s what you want from these things. But we are going to be on a wild ride. We are going to have Uber-style marketplaces matching AIs to people, AIs to AIs. It’s going to be wild over the next few years.

Tim Hwang: And Edward, do you want to close us out for the day?

Edward Calvesbert: Agents! Absolutely. Some of the work we’re doing at IBM with agents is super exciting. It really is going to be a step function in terms of the complexity of workloads and use cases, the creativity in solving problems, potentially beyond our current approaches. The automation—you’re going to have so much work happening 24/7, 365. A lot of stuff already works that way, but this will take it to the next level. I think it’s exciting, productive. I think it’s going to level the playing field for consumers and for smaller institutions. We’re excited to be part of this future and to be co-creating it with our clients and our community.

Tim Hwang: Well, gentlemen, this has been wonderful. Chris, you’re always welcome back on Mixture of Experts. Edward, I hope to have you on again in the future. Listeners out there, if you enjoyed what you heard, you can get Mixture of Experts on Apple Podcasts, Spotify, and podcast platforms everywhere. We will see you next week for another roundup.