Tim Hwang: Are we at peak pre-training? Vagner Santana is a Staff Research Scientist and Master Inventor on the Responsible Tech team. Vagner, welcome back to the show. What say you?

Vagner Santana: Given that we don’t have methods for detecting synthetic data, maybe the biggest risk in the past, right? Because now people are realizing... and maybe we’ve done that already.

Tim Hwang: Volkmar Uhlig is Vice President, AI Infrastructure Portfolio Lead. Uh, Volkmar, how about you?

Volkmar Uhlig: I think we need to give it a couple more years.

Tim Hwang: And Abraham Daniels is joining us for the very first time as a Senior Technical Product Manager on Granite. Abraham, welcome to the show. You’re now an expert on “Mixture of Experts.” Uh, tell us what you think.

Abraham Daniels: I wouldn’t say that it’s over. I guess I’m not 100 percent sure that it’s over, but I think we’re less reliant on it going forward.

Tim Hwang: All right. Awesome. All that and more on today’s “Mixture of Experts.” I’m Tim Hwang, and welcome to “Mixture of Experts.”

Tim Hwang: Each week, MoE is dedicated to bringing the breaking news and analysis you need to understand what’s going on in the world of artificial intelligence. Today is another jam-packed episode. We’re going to talk about the latest release out of Granite, weird ways of stealing models, and NVIDIA’s tiny supercomputer. But first, let’s talk a little bit about pre-training.

As many of you may know, this is the kind of week of the big machine learning conference, NeurIPS. Ilya Sutskever, a big, actually prominent thinker and kind of intellectual and entrepreneur in the AI space, gave a keynote talk in which he claimed that we are at peak pre-training. Essentially, pre-training is over, and to create improvements on AI going forwards, we’re going to have to employ a bunch of different methods.

And I guess maybe Vagner, I’ll start with you because I think you nodded towards kind of one solution that Ilya mentioned in his talk, which is synthetic data. Um, are you ultimately really optimistic? I mean, it kind of sounds like you’ve got kind of almost a paranoid view that we might already be living in synthetic data land. But, uh, tell us more about why synthetic data might be a way forwards if we think that we’re literally running out of the data that we need to do pre-training.

Vagner Santana: I, in fact, I’m concerned about it because when we see the projections of how synthetic data is populating the web, and we don’t have methods for actually detecting and, for pre-training, saying, “Okay, this is synthetic, this is not synthetic.” So, that is why I answered the way I did, because we may already been pre-training models with content generated by LLMs, because we don’t have a way to properly filter when something was generated by an LLM or not. So that’s my concern. So we may be living already in the world that pre-training is happening using data generated by LLMs.

Tim Hwang: Abraham, maybe I’ll try to turn to you. ‘Cause I know you work on the Granite team. I think one really interesting kind of aspect of all this is that a lot of the data that we’ve relied on for pre-training in the past has been open data, right? People use Common Crawl and what have you to kind of pre-train their models. And I guess part of the idea here is that in this world where basically all the data is available and being used, proprietary data becomes a lot more valuable. And I’m curious about how you think about that at Granite. I mean, I know IBM’s taken a real stand towards openness, and I’m wondering if that also applies to how you guys think about the data sets that are gonna become more and more important here?

Abraham Daniels: Yeah. So great question. Um, in terms of data, I think we’ve done a lot of work in terms of partnering with third parties to be able to backfill some of the more specific, like domain or enterprise data that’s key to our not only our models, but our commercial roadmap. So I think that’s going to be a really big pillar for us: where can we find data that may not necessarily be open source, but that is kind of central to our roadmap, focusing on domain-specific models.

But kind of back to what I said earlier in terms of where pre-training might not be over, but I think we’re... there’s a shifting paradigm in terms of how we inference our models, more specifically what’s called “test-time compute.” And you’re starting to see this with some of the newer models that came out, like ChatGPT o1 as well as Qwen, where we’re less reliant on the static knowledge that we get as part of our pre-training and really focusing on how do we make models more capable at inferencing? Having them think a little bit more about their answers, as opposed to using a system-one thinking where the first answer they get is what they respond with.

Tim Hwang: Yeah. And I think that’s a really interesting approach. It’s almost kind of like the real action now is almost “data-less” in some ways, right? It assumes the pre-training happens, and then all the real optimization is going to be like doing these kind of inference tricks, which is going to be a very, very different way of thinking about some of this stuff.

Abraham Daniels: Yeah, it’s a lot more system-two thinking. So reflecting on your answer, having the ability to go back, change your answer, or better understand if there was a misstep in your thought process.

Tim Hwang: Volkmar, maybe I’ll bring you into this discussion. We’ve had you on a couple of shows at this point. I think I’m starting to get the Volkmar vibe on how you answer questions. I don’t know if I’m reading too much into what you said, where you said, “Well, we got a few more years.” Do you think this is just hype? This is just Ilya doing his thought-leading? Like, we’re not really at peak pre-training. You know, this is like, he’s describing a trend, but it’s probably more hyped than anything else.

Volkmar Uhlig: So I think what he’s touching on is—and this has been a trend over the last couple of years—that the amount of information you can get in the open... in fact, we downloaded everything and we trained everything into the models. And so we are now at a point where, you know, how do you differentiate data which is computer-generated versus what is actually human-generated? And that’s the fundamental question.

First of all, we assume that human-generated data is good and machine-generated data is maybe good, and I think that is not true, right? So human-generated data is misleading and is wrong. If you just crawl the internet, what do you know, right? You just download some stuff and people make stuff up and you train that into the model and you declare that’s reality.

So I think we are at a point—Abraham touched on this, and I’m a strong believer in the system-one, system-two thing—where we need to actually test the data which we download from the internet, or we just synthetically generate it. And that’s kind of like test-time compute behind it, right? So I spit out a bunch of answers, and some of them are wrong, some of them are right, and I pick the right one. So I can apply the same mechanism to stuff I download from the internet. And now it doesn’t matter anymore if it’s human-generated, where we kind of assumed humans generate good data, and so anything we could download was good.

I think we are now in a world where we can produce so much bad data at a really high velocity that we actually, when we are creating those pre-training data corpora, we need to actually go through the data, sifted and actually classified as, you know, “this is garbage” and “this is true.” And I think the only way to do that is actually using large language models and the same thing we are using for test-time compute, where we are picking the right answer. We will probably have to apply that to the data corpus we train on.

I think still the pre-training is the system two where... I don’t think for a long time... but I just recall, and I think in any case, we will have to build that core model, and the model architectures change and how we are training changes. And so I don’t think pre-training per se goes away, but I think that the focus will shift. The pre-training is the basis so that you know something, and then you can apply it in the system-two type, where you’re spending more time thinking.

Tim Hwang: Yeah, just like it gets you to first base, basically. But like going further is going to depend on all these other techniques.

Volkmar Uhlig: Correct. I think the further is the hard one. And this is where I think reinforcement comes in. We are actually discovering new knowledge, and we will publish that new knowledge. And I think new knowledge will not necessarily come from humans all the time, but it will come from models.

Tim Hwang: Yeah, I love this observation that, you know, in some ways, the gains from just having more data have been so strong for so long that the incentive has just been like, “Dump more data. We really don’t care. We don’t really look too closely because the more data we put into this magic machine, it just gets better and better and better.” I guess kind of you’re saying is that the meta is kind of shifting now that we are concerned about quality. And then I think the nuance you’re adding to it was very interesting is that synthetic data may very well be much more high quality than what we get out of humans, which I think is an interesting outcome.

I guess, Vagner, how do you think about that? Because I think in some ways, from the point of view of like a responsible AI or responsible tech ethics person, you know, I think that the bias has always been towards human-generated content. We say, “Oh, well, we don’t trust the synthetic generated content because who knows, it might embed all this bias and it might have all these problems.” Do you kind of buy what Volkmar is saying? Like, that actually, we are now rapidly approaching this point where, hey, our kind of long-standing prejudice in favor of human data is actually misplaced.

Vagner Santana: My concern is that if we think about the most popular LLM used now by people to generate content, it could be OpenAI’s ChatGPT, right? And if we have this data—and I agree with Volkmar in terms of the quality, it’s difficult to assess quality of data, human-generated or synthetic data—but imagine that we have a lot of people using ChatGPT to create data, and then this data is used again to pre-train ChatGPT version plus one, right? So now the biases that the data had before are coming in again to the model. So my concern is that this feedback of bias, considering that we don’t have yet good methods for detecting when the data was generated by an LLM... I think that’s my concern: how to prevent that the bias that a previous version has comes into the next version of any LLM pre-training on these data.

Tim Hwang: Yeah, for sure. I mean, I think this goes to the question, which is like, I guess quality is a little bit under-theorized in this context, right? Like, what do we actually mean by better quality here and against what types of tasks? I guess, Vagner, you’ve got certain concerns, right, around this data. You know, I guess if the main use case is like, I don’t know, math or something like that, right? Like, we may actually say that the synthetic data is actually better, but it really depends on use case in some ways.

I’m going to move us to our next topic, I think. We had a big launch this week, which is Granite 3.1. Abraham, you’re on the show in part because you’re from the team. Uh, do you want to kind of tell our listeners, I guess, what’s coming out and what people should be paying attention to?

Abraham Daniels: Yeah. Actually, what’s already come out. So as of yesterday, we released Granite 3.1, our latest model in the series, in the Granite series family of models. So it’s built on top of Granite 3.0. And as part of that release, we’ve pushed out our Granite 8B Instruct as well as our Granite 2B Instruct. Our Granite 8B is really kind of our workhorse model for, you know, 80-90 percent of use cases, enterprise as well as any sort of specific domain cases. What we’re really excited about in terms of our workhorse model is that we know we’ve seen great improvements in instruction following as well as multi-step reasoning.

Along with our Granite 8B and 2B dense models, we’ve released a suite of MoE, or Mixture of Expert, models. These come in 1B (one billion) as well as 3B. And these are really focused on resource-constraint environments, any sort of low-latency applications, edge computing—which we’ll talk a little bit about in terms of the NVIDIA.

And the big kind of release or the capability that we’re launching with the Granite 3.1 is we now support 128K context length. So what that really means is we can input larger tokens into the model. So what that supports is long documents or multiple documents to support QA, as well as any sort of code bases. So you can input a full code base repository. And it also lends to more LLM-powered autonomous agents.

Along with our language models, we’ve released our Granite Guardian series. So these are our guardian models that support detection across a number of different biases and hallucinations. So specifically, the most recent Granite Guardian 3.1 supports function-calling hallucinations. So again, a great feature for agentic workflows.

And then lastly, as part of our release, we have pushed out our Granite embedding models. So these are efficient, robust models that support semantic search. They’ve come in four sizes, cross-English and language—and in terms of language, we support all 12 languages included as part of our language model.

So we’re super excited to have them out in the market. They can be found on Hugging Face as well as our watsonx platform. We also have them on a number of different partner platforms: Ollama, Replicate, they’ll be pushed to NVIDIA as well. And we’re just really excited for what’s to come. And we’re looking forward to scaling out our MoE models in 2025, as well as introducing some multimodal capabilities.

Tim Hwang: Yeah, that’s awesome. So a lot there, obviously, to go through. I mean, I think maybe the first thing I’ll kind of bring you back to talk a little bit about is context window. I know that was a big part of the launch, a big part that IBM is touting as part of this release. Can you paint a little bit more of a picture of kind of like what this means for enterprise customers? Like, what does a long context window actually mean in practice?

Abraham Daniels: So it’s basically how many words you can input as part of your model inferencing. So our initial models were 4K, and there’s really no one-to-one in terms of tokens to words, but we’d say about 1.5. So a 128K context length would be about 300 pages, give or take. And what that really means is now you can ingest multiple documents that supports any sort of particular QA or legal documents, anything that spans multiple pages or multiple corpuses of information. And it opens up a couple different capabilities for users. Again, specifically, more specifically, LLM-supported agent support, which is prominent—kind of like the buzzword right now is agents. Otherwise, it also supports new possibilities around repository-level code understanding, as well as self-reflection. So we kind of talked about it a little bit where you can now start to ask your models to reflect on the input or the output and start to have that little bit of system-two thinking where it can start to better understand its answers and potentially shift its answers if necessary.

Tim Hwang: Right, for sure. Vagner, maybe one thing I can bring you to talk a little bit about here is, I think the models that are focused specifically on safety are pretty interesting here. I think for a very long time, one of the concerns around open models is, “Well, they’re going to be used for all sorts of bad purposes, right?” And I think one of the really interesting questions has been, can you achieve openness and all the things that we like out of openness while still ensuring safety in the model ecosystem? And I take it that these models that are safety-focused are kind of like an attempt to do so. I guess my question for you is like, do you think that we’re on track, that eventually we will be able to have our cake and eat it too, to get the openness and the safety at the same time?

Vagner Santana: I think so. And if we compare even the topics that we discussed in the last episodes, always when we talk about certain attacks that we discovered, they are mostly connected to proprietary models, right? Because when we see that happening for open models, people contribute and people try to fix it. We have a community around these assets. So I think that, in that sense, the open-source strategy makes a lot of sense in my opinion. And it’s an interesting way, and the way that, for instance, for Granite, the Granite Guardian is structured as also from the very first barrier in terms of what is being sent to the model, right, working on the prompt level, and also after generation, I think that’s a good strategy also. And we see that also in other open platforms, like in LLaMA Guard, they have an open-source model to detect also these types of harms, right? And I think that, again, when we see new ways of attacking, as the ones that we discussed a few episodes ago, they are because we don’t know a lot about architecture, about the code, about the flow of information and generation, and how prompts are going in and the outcome is coming out. So that’s, I think that, again, open source is a good approach to tackle this.

Tim Hwang: That’s right. Yeah. So final element I want to kind of touch on before we move to the next topic is, you know, Volkmar, you work on AI infrastructure. And I think one of the observations we talked about a few episodes ago about all the announcements Amazon was making was there’s sort of one interpretation that in the future, infrastructure wins because the models become more commodity over time, right? Someone could say, “Oh, for a few months, I want to try out the IBM model. I’m now going to try out the Llama model. I’m going to go back to the IBM model.” Like we’re living in a world where it seems like increasingly the models will be things that we kind of switch in, switch out, you know, kind of at will. I don’t know if you agree with that’s kind of like how the future will look, or if it really will be, you know, in the future, a customer will say, “Oh, we’re going to build entirely on the IBM model stack,” and that will be kind of the future. It’s kind of a question I guess for you about like how much software will become the key platform, or if it really will be actually more kind of like an infrastructure thing where we build on AWS and sort of the models are very interchangeable.

Volkmar Uhlig: So I think that from just from experience of switching from one model family to another model family, it’s very hard. So I think there is actually a substantial lock-in into a specific model, and it’s primarily because of prompt tuning. And we’ve seen just by doing better prompt tuning for a specific model family, you know, like a 30-40% improvement in accuracy, sometimes like 60%. And then when you did that and you spent three months of your engineering team getting your prompts correct, and you switch to a different model family, all that prompt-tuning work actually is not transferable. And so I think there is a—it’s almost like you are betting on a particular programming language, and then you’re saying, “Oh, you know, programming language is programming language, so I can switch from Java to Python to C-sharp, and it’s just, you know, you just need to rewrite it a little bit.” I think there is actually a quite substantial lock-in into the models.

And so I think what will happen is that the location where the computation happens, that’s totally commoditized. So if you’re on Amazon, you run on Amazon. And if you want to run it in Google Cloud or IBM Cloud, then that’s what you do. I think that the model families have a much higher stickiness. It’s like the way the skills and knowledge get trained, and it’s not directly transferable.

Abraham Daniels: Yeah, I don’t know if I necessarily agree with that. I think the past you’re 100% correct. There’s been kind of like a moat around specific models given the prompt template. But I think with things like the Model Context Protocol that Anthropic released—that again supports Anthropic but is bound to be an open-source tool—and kind of what we’re building, or what the community is building across agents where you have to be able to rely on multiple models given inference costs and capabilities within your particular workflow... I think right now you’re accurate, but I think going forward, model developers are going to be handcuffing themselves if they try to build an infrastructure around only using their models to support use cases where it’s going to be more so, “What is the plumbing going to look like in order to be able to have interconnectedness with LLMs, given the particular use case or agent?” But I mean, that’s just my opinion.

Volkmar Uhlig: Do you think that this is a counter-reaction to model vendor lock-in? Because that’s usually what you see, right? It’s like, “Oh, we have a cloud here, and we build an abstraction layer.”

Abraham Daniels: Yeah, no, that’s fair. I just... I feel like model development became about trying to lock in, and then this democratization, or this race to the bottom in terms of model development, has almost hinders model developers if they don’t allow their specific models or their ecosystem to play well with others. I think that’s where you’re seeing the proliferation of open-source models. It’s driving a community, given that the community decides what is good and what is bad. And from a commercial standpoint, it’s more so, “How do we make money?” But I don’t necessarily—again, my personal views—I don’t think going forward, model developers are going to be able to lock people in and still have the adoption that they want.

Tim Hwang: Yeah. I think it’s gonna be really interesting to see how it plays out. I know a lot of engineer types that I know are like, “Oh, well in the future, there’s not gonna be much prompting, you know, we’ll just run an optimizer and the prompt will be perfect for it.” And I don’t know, I also know people who work in this stuff who are just like, “It’s very hard to imagine that we’re gonna get that level of optimization where effectively the models become commodity because you can always optimize.”

I’m gonna move us on to our next topic. So a really sort of interesting news came out this week on a new type of model exfiltration attack. And I think Vagner, you flagged this for us. It was a super fun story because normally we talk about prompt hacking and how do we do stuff from just the inputs and outputs of the model. But this is great because I think this is the type of attack that you see from time to time, which is a side channel, right? Which is we’re going to just monitor a TPU chip as it does its thing, and then from that, we’re going to extract all the intel we need to reproduce your model. So pretty interesting. And I guess, Vagner, I mean, you’re the one who flagged it. Like, what did you find most interesting about this story?

Vagner Santana: When I read the story, I thought that the interesting aspect—and I started thinking about the money and all the resources that it takes to train a model and to deploy it to a TPU, for instance—and then these researchers, they found a way to use electromagnetic field to do a reverse engineering of the layers of the model that are deployed to that. And they do that by comparing with a data set they have of over 5,000 layer architectures. And so by trying layer by layer, they can mimic one model to a different TPU with 99% accuracy. So that caught my attention and I said, “Whoa, that’s something that tells a lot about how certain strategies may be at risk in terms of this kind of attack.” And again, I’m advocating open source again because if you have open source, then that will be a way to be less susceptible to this kind of attack.

But, yeah, that’s what caught my attention in terms of how... and last month, I think I saw also an attack that was published on PC World that was about something related, like exploring the HDMI cables to detect what people were seeing on their monitors. So I think it’s interesting how certain attacks go beyond our more immediate thoughts about how attacks may happen, and they’re exploring these capabilities of hardware and also the electromagnetic field around them, and also these other properties and capabilities that are there, but sometimes we don’t pay attention.

Tim Hwang: Yeah, definitely. I love this kind of collection of attacks. I mean, this NC State University report is also... it reminds me of like, there’s a really old one from DEF CON a number of years ago, where it listens to the sound your keyboard makes in order to extract what it is that you’re typing, which I think is really fascinating. I guess Volkmar, you know, working on AI infrastructure, one reaction is like, “This is cool, but like, is it practical? Is it a security surface that we need to be worried about?” Because, you know, who’s going to stand outside the data center, you know, do this monitoring?

Volkmar Uhlig: I think there are two questions. So one is like, you know, what’s the value of the model? Is the model structure in itself valuable? I don’t think that... this is the one thing which everybody understands the math. I think there’s not huge amounts of gain right now in looking at the model structure. I think the true value are the numbers, right? So if you don’t have the weights, then ultimately that’s the information that turned into a model. That’s the expensive part. The model structure itself is the cheap part.

I think that there is, in computer science, over the last five decades, we figured out how to secure computer infrastructure. So you have the same thing: extracting databases and database content out of computers. We have cryptography now on pretty much every link in the computer system, so that you can even run it in the cloud, and so there’s a whole confidential compute movement where the PCIe link from the host to the GPU is fully encrypted, so that you cannot intercept anything which travels over it. And so I think we have very standard defense mechanisms against just brute force attacks on the physical hardware.

I think where we are a little bit less aware today is how models get exchanged. And so the model in itself, in particular, if you look from a proprietary perspective, contain your business information, right? So let’s say you fine-tune it and you take your proprietary data and you stick it in the model. You don’t want to give your database to your competitors, and so you wouldn’t want to give your model, which knows everything in your database, to your competitor. And so we are still in a world right now, I think, where we haven’t really figured out what the end-to-end process of confidentiality around models is.

And so there are pieces missing in the infrastructure, just like how we evolved to where we are today. Like, can I have fully encrypted models which only get decrypted, for example, inside of the GPU? Can I have... so this is the weights, this may be the code, if I want to make that confidential, and then it also means like the stimulus or the inputs and the outputs, are they making it over the wire in an encrypted or unencrypted fashion? So we have the mechanisms; the problem is that they are not pervasively deployed or available in hardware or used. And so I think over the next couple of years, we will see much more effort being put in to make sure that we actually protect the asset in a much more stringent way. I mean, we have data at rest, everything’s encrypted; data in flight, everything’s encrypted; and here comes the model, right? And it’s like, “Good luck.”

Tim Hwang: Yeah, it’s really interesting. I mean, if I’m hearing you right, it feels like a little bit like, as true with many things in security, it’s like we have the techniques. Now the question is, can we get uniform adoption in a way that actually offers security here? And I think a whole ecosystem of software just has been written and it’s kind of this way to the market. And now we are like, “Oh my God, all that proprietary stuff, we are wide open.”

Volkmar Uhlig: And so I think we will actually start locking things down. And, you know, from an enterprise perspective, what we do at IBM, this is like a good chunk of what’s next is protecting your data assets, protecting your model assets, making sure that you are actually compliant. And I think that whole workflow of building these things... it has been an afterthought because, you know, we just download the internet and train it in a model, but I think we are now getting into a world where there are really expensive assets which you must protect. And so everything which follows in the enterprise will effectively be employed or deployed with AI, I think it’s a totally...

Tim Hwang: I think it’s like, and it actually ties to what we were talking about a little bit earlier, right? If the trend is we sort of run out of all the common data, there’s also just more proprietary stuff that becomes a component of how we go about doing this in a way that raises the risks here, right? Actually raises the incentive.

Abraham, maybe I can ask you is like, you know, so there are these interesting security questions about model exfiltration and all that. And, you know, Granite, of course, it’s an open model. But I’m kind of curious about how you guys think about security on a release like 3.1. You know, is there a separate team that does that analysis? Just kind of interested in hearing a little bit more about how you guys think about that in the context of like an open-source launch.

Abraham Daniels: Yeah. So we think about—to answer your question, yeah, there’s a dedicated team led by Ian Molloy focused on security. And they do a lot of work in terms of being able to better identify vulnerabilities in the model. There’s also a wide swath of safety and red teaming that we do, more so from the safety perspective, ensuring that our models don’t harm.

But one of the big things we’ve seen in terms of security, from our cybersecurity team, is that models are most vulnerable... or ways we found models are vulnerable is that in the training data, we may be able to train on, for instance, a large corpus of a language that’s not in our intended use case. So if a user was to prompt our model on that particular use case, or that particular language, given that that’s not in the scope of our security framework, it makes it a little bit more vulnerable to either jailbreak or to get a response that maybe is not in scope of what the model should be used for.

From the infrastructure standpoint, I’ll be totally transparent, this is kind of out of my purview. In reading the article, it was just really interesting to see that, one, this was models on the edge. So things that are... TPUs are not typically used to inference larger-scale LLMs. Also, it was... what I found interesting was this was a vulnerability specific to an infrastructure provider. It wasn’t necessarily an external attack that was successful. So I think it just brought up the question in terms of, is this something where model infrastructure providers need to provide more robust guardrails around how potential bad actors inside the company could potentially infiltrate their models?

And then just to the point that Volkmar made, a lot of the models, the infrastructure, the architecture of the model, i.e., the layers and other aspects of the model, they’re usually provided as part of the open-source community. So a lot of these things are on hand for people to use. So in terms of whether this is a risk that we need to kind of dive into full scope, I can’t say yes or I can’t say no, but I think my take from it was it opened up a little bit more of the question of how are infrastructure providers ensuring that bad actors inside the company aren’t trying to infiltrate any of the models that they serve.

Tim Hwang: That’s right. Yeah. And I think it goes to a little bit of what Volkmar was talking about, which is a number of these security issues are known security issues for just the infrastructure, even before you get to AI infrastructure. And so, the insider threat problem is big in any case, and it’s kind of just like, well, how much of this kind of folds into traditional security work? I think this is one of the reasons I’m asking Abraham is I’m really interested in how an organization decides to deal with safety and security on these models. Who’s responsible for bits of this? And I think it’s assigned in very different parts of the organization. It’s very interesting.

Abraham Daniels: And it’s new. These are things that, you know, as much as AI has proliferated over the last couple of years, it’s only... “Attention is All You Need” only came out seven years ago. Actually, it came out three and a half years ago? No, seven. So, we’re still trying to figure out a lot of things.

Tim Hwang: That’s right. I mean, one of the things I love to say on MoE is, “in the old days,” by that I mean like five years ago... six months ago.

Abraham Daniels: Yeah, that’s right.

Tim Hwang: So I’m going to move us on to our final topic for the day, which is Jetson. Background on all this is that NVIDIA announced a small, dare I say, cute little board supercomputer called Jetson for AI developers. Unlike the eye-watering price of an H100 or what will be the price of the GB200, this retails for a relatively cheap price of 250 bucks. And it’s this kind of handheld little board. And I guess Volkmar, maybe I’ll turn it to you. It’s like, you know, why is NVIDIA getting into this business, this kind of hobbyist GPU business? And do you think it actually matters for the overall AI market at all? Or is this kind of a little bit just like, Jensen wanted to do a fun thing and is releasing a little board just because it’s a fun end-of-year thing to do?

Volkmar Uhlig: Yeah, so this is a continuation of something NVIDIA invested for many, many years. And so before the ChatGPT craze, there was a massive investment of NVIDIA into autonomous vehicles. And so coming from that industry, we’ve been tracking that board pretty much—it came out, I think, 2015, ‘16 ballpark-ish—and has been continuously receiving updates from NVIDIA. So this is really a board which is for low-power robotics. There is a version which has lots and lots of cameras being able to be attached so that you can actually put it on a robot. And it’s cheap because it’s made for scale. It’s not running large language models, more like vision processing, planning, mapping. So it has a bunch of on-processors on it, and it has an NVIDIA GPU.

So I think the main benefit over all the other solutions on the market is that in many cases today, you’ll train your models on an NVIDIA card, so you’re living in an ecosystem, and then you can just move that ecosystem in production where you only run the inferencing part. And it’s powerful enough that you can actually do camera processing, the video encoders, etc., on that one chip. It’s a pretty low-power solution, so you don’t drain batteries. You don’t want to put an H100 on a robot simply because your robot drives like five meters and it’s out of power. And so it’s more of an embedded system-on-a-chip thing.

So I think the nice part of it is it’s now—it used to be around 500 bucks, now it’s down to 250 or so. It’s really affordable for hobbyists, but also if you want to build something at scale and you get a chip for 250 bucks, you effectively can build a robot now. So, at least from the electronics part—I mean, the robot’s missing, right?

Tim Hwang: Yeah. Right. You at least got the board. The robot is up to you. So the automotive part of that is a bit more involved, and it has like dual chips and has many more cameras and it has ADA compliance. And that’s one, you know, it’s kind of the baby chip of that.

Vagner, maybe I’ll turn to you on this is, you know, we’ve talked a lot about models getting smaller, but when we’ve talked about system on a chip and we’ve talked about AI at the edge, it’s often been in the context of a mobile phone, right? Like we’ve talked about Apple doing a new release. One of the things I think that’s pretty fun about this is that it’s marketed certainly as a tool for hobbyists, student groups that want to build robots or they want to do their own inference for their own experiments. That ecosystem I think is really interesting. And I guess I’ve got a question for you is like, how far do you think that’s going to go over time? Maybe the last thing I’ll throw in is I’ve been watching this project by George Hotz for some time, this Tiny Corp project, where he wants to basically offer everybody a petaflop. And I think one of the questions is just like, does it become cheap enough that everybody ends up having a little GPU rig at home? I’m kind of curious about how you think about that, because it starts to look quite different from how we do AI nowadays.

Vagner Santana: Yeah, this accessibility in terms of value, I think that it’s the most important aspect in my understanding. Even if we think about developing countries, for instance, in Brazil, 250 is like the minimum wage, a monthly salary, so it’s not that cheap in some developing countries. But even though if we consider that this is the cheapest right now, I think that opens possibilities, a lot of possibilities in those countries where people have a lot of creativity. And this kind of hardware may allow them to think about robots for agriculture or other interesting uses that the cost before this specific hardware was a blocker.

Tim Hwang: Yeah, that’s right. The kind of continual democratization is interesting. Definitely for international as well. I mean, I think that’s a really big component of this is who gets to be able to tinker with some of these tools seems really, really important. I guess, Abraham, do you, I don’t know, are you an AI hobbyist in your free time? I’m kind of curious if you would buy something like this, play around with these types of tools.

Abraham Daniels: I mean, it depends on who’s asking the question. Yeah, I definitely, in the old world, I definitely played a lot with it. I’m not as technically sound as some of the other team members here at IBM Research. But I find myself playing with the agentic framework quite a bit. When BabyAGI came out a couple years ago, I found myself really kind of diving into it. So, I don’t know if I’d buy this per se, but I think just like the AI development is being abstracted to a level where it’s a lot of plug-and-play nowadays, so you get a lot of... the skill set of a developer, although for core LLM development may still be sound, but in terms of just a hobbyist, I think that skill set can be a little bit lower. And I think this is kind of a pretty cool tool to help that side of the fence in terms of creating more models that you can run on the edge.

Tim Hwang: Yeah, absolutely. I think there’s going to be just a ton of activity there. Well, great. Well, thanks everybody for their time today. Abraham, welcome to the show. Hoping to have you on for a future episode. Vagner, Volkmar, always a pleasure to have you on the show as well. Thanks for joining us. If you enjoyed what you heard, you can get us on Apple Podcasts, Spotify, and podcast platforms everywhere. And listeners, we’ll see you next week.