Tim Hwang: What’s the most exciting announcement at this year’s IBM Tech Exchange? Kate Soule is a program director at IBM Research. Kate, welcome. What do you think?

Kate Soule: The Apache 2.0 license of Granite 3.0.

Tim Hwang: Kush Varshney, IBM Fellow, Granite Guardian.

Kush Varshney: Granite Guardian.

Tim Hwang: And joining us for the very first time, Petros Zerfos, who’s a principal research scientist at IBM Research.

Petros Zerfos: That’s an easy one. That’s the high-performance Granite 3.0.

Tim Hwang: Terrific! All that and more on today’s Mixture of Experts.

I’m Tim Hwang, and it’s Friday again, which means it’s time to take a whirlwind tour of the biggest stories moving artificial intelligence this week. We’ll talk about NVIDIA’s latest and greatest open-source model and Perplexity raising at a wild valuation, but first, we’re going to talk about IBM’s annual Tech Exchange conference. There’s a slew of announcements out of IBM, and we’ve got the ideal team to talk about what’s launching this week.

The first headline I want to address is that Granite 3.0 is out. Kate, I know you played a really big role in getting that together and being a big part of the launch. Tell us about what’s exciting and different here from previous generations of Granite.

Kate Soule: Thanks, Tim. We’re really excited about Granite 3.0. It launched at, like, 12:15 a.m. on Monday morning.

Tim Hwang: I know down to the minute!

Kate Soule: I know down to the minute! The reception has been really, really phenomenal. So, Granite 3.0 is IBM’s state-of-the-art large language model family. It’s a series of models that cover language, safety models called Granite Guardian, and we even have some models focused around efficiency, like a speculative decoder model that came out. They’re all available under the Apache 2.0 license, which is really exciting.

Tim Hwang: Yeah, and I would say, is there a deeper theme that IBM is pushing with this set of releases? It almost feels like every generation of Granite is getting broader and broader, with more things launching. I’m curious if the team had any particular emphasis this round.

Kate Soule: Well, with this round, our main goal was to consolidate all the different things into one model. Before, IBM had English language models, multilingual models, and code models in generations one and two. With Generation 3, we were able to bring all of that into one model while continuing to push the boundaries of how much performance you can pack into an eight billion parameter model.

Tim Hwang: Nice. I really want to get into the details here. We have an ideal configuration with Kate, Kush, and Petros, who were all involved in the Granite release. We’d love to dig more into the details. Petros, you named the most exciting thing at Tech Exchange this year as Granite, which you worked on. Do you want to tell us a little about your involvement with the release and what has you most excited about it?

Petros Zerfos: Yeah, absolutely. As I mentioned, it’s a very exciting release. My involvement is around the data engineering—essentially, the preparation of the huge amounts of data that go into training such large language models. This goes all the way from acquisition to the point where it’s converted into its vectorized form, called tokens, which is what’s used for training the Granite models. We’re talking billions of documents, lots of terabytes and petabytes worth of data, with massive infrastructures thrown behind it. It’s very exciting.

Tim Hwang: Yeah, for sure. I really want to get into that because so often, particularly at tech conferences, people see the end result—these cool new models they can use. As a consumer, I’m excited, but what’s exciting about your work is talking about what goes on behind the scenes. That data curation... tell us what’s hard about it. What makes it a really challenging task?

Petros Zerfos: That’s a very good question. What makes it a very hard challenge is a multitude of things. First of all, the sheer volume of data needed to be curated and fed into the training process is breathtaking. We start with literally petabytes of raw data collected from a number of sources, including the whole internet itself. The curation process and subsequent steps of annotation and filtering to find the golden nuggets of very high-quality data that will go into the training is a massively challenging process. Lots and lots of machines, clusters, and data centers are needed to go through such cleansing and filtering.

Tim Hwang: Yeah, were there any particular documents that surprised you? I’m kind of curious if there are any surprises in the process, like, “Oh, it’s really funny that the most high-quality piece of data is ABC or XYZ.”

Petros Zerfos: Having processed through pretty much most of the data out there on the internet, you can definitely find some things that make you wonder about humanity itself and what it puts out there. But of course, there are the golden nuggets of knowledge in the form of textbooks, scientific papers, medical studies, and legal studies written by scholars and people with high expertise. It’s a pleasure to have those high-quality aspects included in the training of Granite.

Tim Hwang: It’s an aspect we’ve talked about on the show before. Hearing you talk more about it is great because it’s not just dumping huge amounts of data into the model. There’s a lot of work that goes into selecting the right tokens. It’s almost artisanal in nature—getting the right blend for the best results.

Kate Soule: Well, Tim, while it’s artisanal, I also want to highlight something the team did that I think is really cool: the degree of experimenting and searching over different data mixtures. Training a two billion parameter model requires training... Petros, I don’t know, how many small models do you think you trained?

Petros Zerfos: Oh, we trained hundreds of those very easily. We trained thousands of even smaller models to get down to the proper mixtures, as well as bigger models in the order of one to two billion parameters. We trained literally hundreds of those to figure out the best type of cleansing and mixing. Definitely lots of effort by very large teams in IBM Research and lots of infrastructure behind it, both GPUs and general clusters.

Tim Hwang: Yeah, for sure. To underscore that it’s not always black and white, some of the decisions and processes the team went through... a fun example is thinking about how many emojis to include in the Granite training data. What is the appropriate level of emojis for a model to understand?

Kate Soule: That’s a hard question. What’s the risk of having too many emojis? Well, then the model has a predilection to give a lot of emojis in its response. Depending on your use case, maybe you care about that, but in an enterprise setting, you probably don’t want it to skew towards emojis. But if you remove emojis altogether, the model doesn’t understand the concept of emojis or how to interpret them, which is critical for basic tasks and use cases. So, there was a whole effort—I’m not kidding—just figuring out the right level of emojis the model should be trained on to understand them.

Tim Hwang: That’s fascinating. Well, Kush, I don’t want to let you off the hook here. I understand you were also involved in this release. Do you want to talk a little about your part of this launch?

Kush Varshney: Yeah, I was involved in a few different parts. On the Granite 3.0 language models, as Kate said, it’s really language and code and a lot of things all together. I was involved in a lot of the safety alignment. After Petros does his work, we have the pre-training data, then there’s the training process, and then further alignment after that. Part of that is taking the model from the base model to an “instruct” model, and then doing further tuning to make it safe in various ways. I was working with a big team coming up with seed examples to generate synthetic data across many different types of harms and risks, figuring out how to get the model not to engage in those topics.

The way we evaluate that level of safety is through a variety of benchmarks. One was developed in our Haifa research lab; it’s called AttackAQ. We compared the Granite 3.0 models—the 8 billion instruct, as an example—and it outperforms all the other competitors out there in this benchmark. It really is the safest in many ways.

That’s one half. The other half of the work was on the Granite Guardian models. The way to think about it is, when thinking about safety and preventing harms, you want to do the best you can on the main model, but it’s never going to be perfect. So, there should also be a second, independent model checking the first model to ensure it’s not putting out bad stuff. Granite Guardian is that second model, built on top of the Granite 3.0 language models, but constrained to give a yes or no answer. It will look at an input prompt, a model response, or the combination and say yes or no: Is this harmful? Is it a jailbreaking attack? Is there hallucination? Is there a problem with context relevance or answer relevance in a RAG setting? This model is meant to act in that capacity. It’s important to understand it’s not limited to just working with Granite models; you can apply it with any model out there.

Tim Hwang: Yeah, for sure. There’s a lot there. One question to push you a little further, Kush: Safety is so broad. There are infinite things a model could do wrong. How do you and your team manage those risks? It’s an infinite attack space, but Tech Exchange is this week; you have to get something launched. How do you reconcile those two? How does the team think about broadening its risk set over time? It’s an interesting process people don’t usually hear about.

Kush Varshney: Yeah, and it is always about broadening. As you said, the attack surface area is pretty much infinite, so we can only pick and choose to touch on some parts. We understand that. We created this Attack Atlas (it’s a paper to be presented in Europe). There are so many different ways, strategies, and topics of harm. We just do our best. You keep making progress, keep adding things, using taxonomies to categorize different types of risks and harms, building that up, trying to get as broad coverage as you can, looking at different attack strategies. It’s a cat-and-mouse game—red teaming, blue teaming, going back and forth, seeing what the problems are, figuring out how to address them, and cycling through it. Nothing’s ever going to be perfect; it’s a process.

Tim Hwang: Yeah, for sure. Kate, you’ll have to indulge me. As the person who said the most exciting thing is Apache [2.0], I was like, “Oh my God, yes, let’s talk about Apache 2.0!” Why should our listeners be excited about that if they’re not huge licensing nerds?

Kate Soule: So, Apache 2.0 is an incredibly permissive license. It basically says anyone can take and use our Granite models, customize them however they like, and use any outputs however they like, and IBM will make no claims to that IP. You have full rights. That’s really important, especially for enterprises looking to customize LLMs with their own data and IP. You want to ensure no further restrictions on what is now your IP encoded inside an LLM. We’re excited to offer these models under those terms, reducing barriers for the broader community to use and customize them.

It’s something of a dying breed. If we look at models released in the open, more are being released with custom licenses. We’re trying to keep it simple: Apache 2. Please take our models, customize them, and use them out in the world.

Tim Hwang: Yeah, I was confronted with this recently, importing a model from Hugging Face. It was gated with a completely custom license, and I thought, “This will take forever to see if I can use it.” Can I ask why IBM is taking the most open perspective? It sounds like a conscious strategy to be the most open of all the open providers.

Kate Soule: Well, again, I think it comes down to the enterprise use case. We believe the future of LLMs and generative AI in the enterprise is being able to customize models with enterprise and proprietary data. We’re trying to create the tools—both base models like the Granite series, which can be customized without restrictions, and tools like InstructLab through our Red Hat product offering—so you can take those models and customize and build on them without concerns. Then, wrapping that all under, for example, Watsonx.ai with indemnification and other protections and support. We’re trying to create an open market and ecosystem our customers can build on with confidence.

Tim Hwang: Yeah, that’s a theme I wanted to build on. A big part seems to be unleashing developers to do what they need with these models without controls. One unique thing, Petros, maybe you’re the person for this: IBM is also open-sourcing the data prep kit around these models, which is unique. There’s hype around open-source models, but here, there’s also openness around constructing the model. Two questions: Why is IBM doing that? Let’s start there.

Petros Zerfos: Sure. That goes along the general theme of openness. We’re open-sourcing the models and their weights under the most permissive license, Apache 2.0. Similarly, we’re doing the same with the software assets we developed, open-sourcing them under Apache 2.0. This enables the community to build upon that, reproduce it, and use the same facilities we developed for training Granite. We believe this benefits the overall community and ecosystem and enables more developers to follow the same best practices we learned through hard lessons.

Tim Hwang: You guys have solved the emoji question so developers don’t have to!

Petros Zerfos: We debated a lot around the emoji question. I can only imagine the meetings!

Tim Hwang: It’s incredible. A follow-up question about the data prep kit and open-sourcing it: Do you think this is also a way to encourage other providers to release their data openly? This is an important aspect of the ecosystem. I’ve been frustrated when a new model behaves differently and breaks tooling, and I’d love to delve further. Is the hope that what IBM is doing becomes a more general practice? Do you think it will become the norm?

Petros Zerfos: Yeah, that’s a very good question. There’s an interesting adage: “Every conversation in AI starts with models but ends with data.” Everyone recognizes data is the oil or fuel that powers the models. Open-sourcing the data prep kit is good practice, bringing more people and developers into doing the same. There’s a trend developing around providing data assets for preparing models. NVIDIA, for example, has its NeMo Curator. Other big names are going in that direction. Along with IBM’s general theme of openness, data is the natural thing that follows.

Tim Hwang: Yeah, for sure. I want to throw it open before we move on. Obviously, many things were announced at Tech Exchange. We’ve been talking a lot about Granite because you all worked on it. Are there other things you’d point people towards they should check out? I know there’s a Code Assistant announced, but the list is long. Any specific highlights?

Kate Soule: I might give a couple of shout-outs. One, please go try the models, especially on platforms like Hugging Face. You can run these models locally; they’re blazing fast. We’re excited to make them available across partners. Second, there was a big focus on agents and assistants. We’re excited to see how the watsonx software portfolio evolves, creating different agent orchestrators and managing agentic systems. You’ll see a lot of exciting work from IBM in that space.

Tim Hwang: Yeah, for sure. Maybe we’ll end with you, Kush. Where is Granite going next? Sitting here in 2025, talking about Tech Exchange, I’m curious what the team is piling towards, particularly in safety. What’s the next frontier in safety for people to hear about?

Kush Varshney: Yeah, building on what Petros said, having the data prep kit out there is a boon for value creation and contributes to safety. When you can inspect these things, you can know why something is happening and identify potential concerns. The movement towards openness will be a big aspect of the safety world.

At Tech Exchange, we announced new features in what’s next on governance—our platform play on governance and safety. Where Granite goes next, especially with Granite Guardian, as Kate said, is agentic workflows. Our next release of Granite Guardian will have function calling hallucination detection, which isn’t out there from anyone else. That will bridge the gap when you talk to a model in natural language, and it translates that into an API call. We want to ensure nothing wrong happens in that step—the parameters, function names, or parameter values should all come out cleanly. That’s one of the more exciting things we have lined up.

Tim Hwang: Well, awesome. A lot more to talk about, but this is a great overview. I’m glad we got behind the scenes on how these launches happen. You just see the model at the end of the day, but it turns out lots of humans spend a lot of time getting it right. Thanks for giving our listeners a lead-in.

Our next story is news about the company Perplexity. If you’re not familiar, Perplexity is AI-driven search. It’s different from traditional Google; you ask a query and have an interactive conversation, and it pulls results from the internet. The big news is rumors it’s about to raise USD 500 million at a USD 6 billion valuation, which I believe is twice what it was before. That’s wild. We’re in an era of AI excitement, and valuations are through the roof, but even I saw this and thought, “Wow, this is intense.”

Kate, maybe I’ll kick it to you. Is this valuation justified? What is the future of search? Is this the new Google? How do you size up this news?

Kate Soule: Yeah, there’s a lot going on, certainly a lot of hype. But yes, I think chat is the future of search. It’s a much more natural way to find information and inquire about something. But I also wonder how differentiated or competitive Perplexity can stay. What’s their moat to prevent others from doing the same with a different API call? I worry we’re seeing some inflation of expectations in this valuation.

Tim Hwang: Yeah, for sure. Not for nothing, out of all my monthly subscriptions, Perplexity is one I use regularly. But you’re touching on an important question: What is the moat? Is there one? Petros, any views? It seems like other search companies might be good at this eventually. Clearly, someone sees a good bet. What’s the bull case? Is there a moat?

Petros Zerfos: I have to admit I agree with Kate. I’m also struggling to figure out the moat. Search has been an area many companies, startups, and mega-corporations have tried to tackle and attack the incumbents over the years. It never panned out well because the existing ones were good enough. Now, it’s a brand-new interface and way of interacting with a search engine. More importantly, getting a good summary instead of having to read yourself—everyone appreciates a nice, executive, bullet-point type of interface.

That being said, the valuation and expectations probably make sense. The moat? I’m also struggling to figure it out, especially as a few big names we’re all aware of are essentially on it.

Tim Hwang: Yeah, for sure. Kush, one question I wanted to bring up around safety. I agree; interacting with Perplexity, I think, “Oh, chat is giving me a lot of action on search.” Unfortunately, it’s caused me to buy too many books because I ask for recommendations, and it gives me 10 fascinating books.

Let me play skeptic for a moment. One funny thing about LLMs is everyone has rushed to use them as a search interface, but out-of-the-box LLMs aren’t concerned with information retrieval, facts, validation, or verification. There’s no notion of PageRank to give a sense of credibility. So, playing skeptic, are LLMs the future of search? They do something fundamentally different from what you want in search. We’re bolting search features onto this tech. Isn’t that getting the car before the horse?

Kush Varshney: Yeah, I think you’re absolutely right. The fact is, it’s the RAG doing the search, and the language model is on top, creating the bullet points. I’ll disagree with Kate a little; I’m not convinced chat is the best method or interface for search. When I’m searching for stuff, like a research assistant going to the library, it’s not a linear process. You go down one rabbit hole, come back, look for this, go over here. Chat insists on a linear process. Intent-based interaction is part of it; chat is the simplest version of intent communication. But I think more helpful ways will emerge where the LLM’s strength will be organizing the work, organizing different threads, and putting them together. The search itself, the retrieval, is not part of how the LLM does it. That’s where we might end up.

Tim Hwang: Yeah, that’s super helpful. It’s almost two innovations: the retrieval and the LLM as the spice on top making it digestible. Kate, any response?

Kate Soule: No, I don’t disagree. I think chat is a huge improvement over shouting into the void of a search box, but is it the final frontier? Kush brings up good points; it’s not a linear flow. In many ways, we’ve worked on making a faster horse when we need a car. We’ve made chat a really fast horse, but what does a car invention look like in search? There are certainly opportunities ahead.

Tim Hwang: Yeah, it’d be cool if it’s an entirely different paradigm. I think about the anchoring effect of ChatGPT. We see everything through this lens because it was so successful, but it’s almost a historical accident. It’s interesting to think about different users. Kush, as a researcher, you have a well-honed art for rigorous investigation. Someone trying to find the closest grocery store or doing casual investigation is a different mode. There’s tremendous diversity in potential interfaces; there probably won’t be a one-size-fits-all.

Kate Soule: Yeah, there’s probably not going to be one size fits all for search interfaces.

Kush Varshney: I would definitely agree. When my kids look for information, like “How many goals did Alex Morgan score?” they don’t need the same rigor as my research. So, absolutely.

Tim Hwang: Yeah, my long-term theory is that, like Googleese—where people don’t speak in English but a string of words optimizing search results—we’ll end up with our own “Perplexityese,” not quite a conversation but how we’ve learned to get the best results.

Kate Soule: Well, ironically, as we do that, model providers will figure out how to take that Googleese version, translate it into what’s optimal for the model, and feed that. There will be layers of trying to find the right way to frame a question.

Kush Varshney: And that’s what agentic workflows are—multiple layers of agents translating from one thing to another. That’s where we’re headed.

Tim Hwang: All right, for our final story, it’s another open-source model story to compare and contrast trends. NVIDIA, maker of fine GPUs, recently released a fine-tune of Llama called Nemotron, specifically Nemotron 70B Instruct. It was widely touted; they showed it beating state-of-the-art benchmarks across other proprietary models. That’s interesting, but I want to ask the panel: Why? NVIDIA is known for GPUs and hardware. Why are they getting into model training, and why open-source models? Petros, maybe we’ll throw it to you.

Petros Zerfos: Yeah, that’s a very interesting question. Everyone knows NVIDIA for its GPUs. I’m not sure how many people know that NVIDIA’s moat is actually software—the CUDA interfaces and drivers that managed to attract developers over the last 10 years onto NVIDIA hardware. That’s why everyone uses and still uses GPUs from NVIDIA. So, it’s natural to expect them to expand on that, both in the software ecosystem around their hardware and by showcasing it through models they can train themselves. My last thought is that NVIDIA is also developing its NVIDIA Cloud, which contributes to the ecosystem of AI models NVIDIA is driving.

Tim Hwang: For sure. From your side, Kate, it sounds like, from Petros’s interpretation, it’s a show of strength—NVIDIA saying they can do models like this. But part of this is trying to attract people to their cloud, right? When I think of cloud, I think of Google and Amazon, not NVIDIA. Is that how you read it—promoting that aspect of their business?

Kate Soule: Yeah, I think it’s a powerful demonstration. They’re saying they can take a model, customize it, continue training it, and boost performance beyond the original instruct version of the 70 billion Llama model. In doing so, they’re demonstrating their capabilities and inviting customers to join and customize their own models on NVIDIA’s platform. It makes sense as a pure marketing point to showcase their capabilities.

Tim Hwang: Yeah, definitely. Kush, do you think this makes other companies nervous? NVIDIA has always been in the background as the chip people, the infrastructure. Now, they’re releasing something that competes with o1 or Opus. Should companies be nervous? Is NVIDIA playing in a new playground?

Kush Varshney: Yeah, I think so. In the future, what will happen is, just like with traditional machine learning where you’d look for a dataset and build a model, in a couple of years, you’ll have a problem, look for an appropriate model, maybe some fine-tuning data, and work with those. Models will be artifacts in the world of possibilities for solving your problem. NVIDIA can position themselves by saying, “All these models are out there; you, the customer or company, don’t have to invent those pre-trained models. What you need is customization. Come to us; we’ll help you.” Having that mindset available—that you don’t have to worry about all the different models, just the customization—will be their strength.

Another thing: There’s this overused trope that in the Gold Rush, the people who made money were the ones who provided the shovels or the blue jeans. With the blue jeans, they crossed over from being for miners to a high-fashion item people would customize. So, somehow moving from that commodity to fashion as well.

Tim Hwang: Yeah, that’s right. It’s the high prestige. That final comment is interesting because it suggests some visions are aligned between IBM and NVIDIA. NVIDIA is excited because more demand for models means more chips. IBM wants to release models to unleash developers but believes there will be enterprise services on top. I understand Granite will be available on NVIDIA as well?

Kate Soule: It is. They were a launch partner. You can check out Granite 3.0 models on NVIDIA today. Even Granite Guardian—within 12 hours, they had a full working demo. You can try it there too. Yeah, it’s so fast.

Tim Hwang: Well, great. That’s all the time we have for today. Kate, Kush, always great to see you. Thanks for talking about Granite. And Petros, hope we have you on the show again.

Petros Zerfos: Thank you very much for having me. Hope so too.

Tim Hwang: You can get us on Apple Podcasts, Spotify, and podcast platforms everywhere. We’ll see you next week for another action-packed week of Mixture of Experts.