Tim Hong: Hello and welcome to Mixture of Experts. I’m your host Tim Hong. Each week we bring together a panel of researchers, product leaders, engineers, policy experts, and more to discuss, debate, and distill down the week’s biggest news and trends in AI. So today on the show, three stories. First one, the state of the open source – what are the biggest trends in open source models and how will they shape the business of AI? Second, the future of retrieval augmented or RAG – they’ve come so far, where are they going to go next? And then finally, Kolmogorov-Arnold networks or KAN – what the hell are they, why are all the nerds suddenly talking about it, and should we buy the hype? So today on the show, I’m ably supported by an incredible panel of experts. So, first off, Marina Deniker, senior research scientist at IBM. Marina, thanks for joining. Good to be here. Yeah, and particularly thanks to you for joining us so early Pacific Time. David Cox, VP Models and director of the MIT IBM Watson Lab. David, thanks for joining the show.

David Cox: Pleasure to be here. And returning for the second episode, we were joking that we just made, make this a Kush Varshney show going forwards. He’s unfortunately declined that, but Kush Varshney, IBM Fellow working on issues surrounding AI governance. Kush, welcome back.

Kush Varshney: It’s great to be here. And yeah, I’m the Varshney with the hair. So yeah, we’ll use that as a little mnemonic. So.

Tim Hong: Well, great. So let’s start with the first story that we want to cover today on Mixture of Experts. So I think from where I’m sitting, you know, there has been just so much happening in the world of open source, right? So Meta, of course, released LLaMA 3 a few weeks back. Apple, in a very, you know, big move for them, I think, released the Open ELM on-device models. And then IBM just recently released its Granite family of models. And so David, I kind of want to give you a chance to kind of first plug Granite, tell us what it is and what you guys have been working on. And then I kind of want you to kind of go into, you know.

David Cox: Why it is that IBM decided to release Granite open source and why it thinks that doing this matters. And I think from there, I think we can talk more broadly about what’s happening in open source, but I wanted to give you a shot to talk a little bit about the work that you and the team have been doing. Sure, yeah, happy to. We actually had two major open source announcements. This was a big week for us across IBM and Red Hat. The first first was that we open-sourced the Granite code family of models. So these are models in a variety of sizes: 3, 8, 20, and 34 billion parameters, trained on 116 programming languages. These are, you know, state-of-the-art models competitive with, you know, the best in in the field. And one of the areas that we really optimized for for Enterprise users, because ultimately IBM is interested in supporting Enterprise, is that all-around capability, you know, not just Python and generic code generation, which is often the focus for the academic community, but also Java. And Rust and all kinds of other languages, and also things like code fixing and explaining. So there’s a lot of things you can do with code models, and it’s really being integrated into the software development, you know, fabric of how we do software, and software is integrated into the fabric of everything we do in society. And we we wanted to release these because we, you know, ultimately our position is that that open wins. You know, like we’re communities will build around these models, people will build things that, you know, that we wouldn’t expect. They’ll they’ll be able to extend the models, and that’s that’s super powerful. And that leads a little bit to the second announcement, which which happened through Red Hat. So we have a technology that we developed for doing alignment of models, uh, we call large-scale alignment for chatbots, and that gave rise to a project called InstructLab. And what InstructLab is, is a way to actually aggregate community contributions to the instruction. Tuning of a model. So now any developer anywhere in the world can submit new skills and new knowledge to a model, and and then that actually gets integrated, and then we do a weekly build of that model. So it’s a different cycle of development, a different kind of community forming where we’re not just forming around a model and you know, building inference tools and things like that, but we’re actually able to merge all those contributions and then update the model every week. So we’re really excited about this. It’s been a fantastic partnership with Red Hat building this out. They know open source better than than anyone, and we’re really excited this got announced at at Summit on on Tuesday by Matt Hicks, the CEO of Red Hat.

Tim Hong: Yeah, that’s awesome. And I think I don’t know if you’d agree with this, is like I see what IBM has done with Granite and with InstructLab, and it’s kind of like you, I was joking with a friend the other day, I was like, it’s open source putting its big boy pants on, right? Like they’re kind of like moving into like open source being something that like Enterprises will actually use. And I think that’s really changing what we mean by open source, right? Like I think like the big trend even a few months ago was like, oh my god, these open source models are just getting so big, right? Like they’re huge parameter models, and like, isn’t that the exciting thing is that open source models will be, you know, on par as sophisticated as like state-of-the-art? But what’s kind of interesting here, I think, is really sort of like twofold, right? Like I think like what’s interesting with Granite is you guys are releasing a class of models of different sizes sort of on the idea that like not everybody’s going to need like the the chunkiest model in the whole world, which I think is really, really interesting. And then I think again, it’s also kind of like on par with what we see out of the Apple announcement, the OpenELM announcement, right? Which is like these are not the biggest. Models, but they’re on-device models, right? And it kind of feels like I don’t know if you’d agree with this, David, is like, it feels like open source is finally now kind of like responding to market need, like then in some ways, like Enterprises are like, how do we actually apply this stuff? And now like essentially open source community is like trying to now, you know, adapt to actually provide solutions to that. But I don’t know if that’s like a characterization you guys would agree with.

David Cox: Yeah, yeah, no, and and I think I think you’re spot on. You know, there isn’t just one thing that people want to do with LLMs, so there’s not going to be just one LLM that wins the day. You know, we have our models running on laptops, and there are that’s they’re really interesting, you know, advantages to doing that. Like if you if you want to be on-prem, you don’t want to be data over the network, it’s it’s, you know, there’s, you know, it’s proprietary and you’re worried about IP, you can run these models in many cases on on your laptop. For other applications, you want the very best performance, you know, say you’re doing an application modernization that’s just going to happen once, and it needs to be, you know, the highest quality, then you can move to one of the larger models. So we we really are trying to be responsive across the spectrum of of different needs. And yeah, at IBM, we we are trying to be sort of, I think you said, the big boy pants, you know, like we’re very transparent about what data we put in the models, which is which is not always true, but which is very important if you’re an Enterprise want to use these. The other point of differentiation for our models is we release them under Apache 2 license, just a clean Apache 2, no additional restrictions. And this can be really important for for adoption. We knew this is something that our customers would ultimately need and want. So that that that’s, you know, how we’re evolving sort of the our approach to open source and. Again, yeah, like you said, meeting the customer needs.

Tim Hong: Yeah, definitely. And Kush, Marina, I’m not sure if you’ve got views on this, is like I think, you know, I think we can use this as a springboard to kind of talk about like how this is going to sort of shape the the market as a whole, right? Because I think, you know, if I’m now you like an Enterprise sort of thinking about how to integrate LLMs, right? It feels like there’s increasingly options, right? Well, we can, you know, go work with, we can try to do it at all ourselves in-house, right? Like we can try to go with like the big proprietary models, right? And it kind of also feels like there’s going to be a range of new businesses that emerge here as well. Like just like the whole business of like you come to us with a problem and we fine-tune open source models for you seems like it’ll increasingly become a big part of the ecosystem. But, um, yeah, I’m kind of curious as from from your point of view, kind of like in, you know, the the kind of research space.

Marina Deniker: And even thinking about like, you know, where this all goes, just if you’ve got views on on how this will kind of impact the ecosystem as a whole. Yeah, I mean, one great thing that, uh, I mean, InstructLab enables is really, I mean, shifting power to value creators. So it really allows, I mean, as David said, I mean, this whole community to to really congeal around this thing, um, and make these models authentic for themselves. It’s some sort of commitment to locality as well. I mean, for whatever you need for your Enterprise, for your organization, you can, uh, really make things make things yours. So I think it’s, uh, it’s an awesome, awesome thing. Yeah, I really appreciate being able to add the skills as you find them needed for your own use case. So the thing with all of these models is that it’s very hard to predict when you put them out what are they actually going to be used for. So being able to have the flexibility to say, oh, I’ve realized I have a use case, I need to adapt quickly, I need to make the model adapt quickly. Sometimes with something that’s proprietary or somewhere else, you just don’t have the ability to move that quickly or even to stress test or check and is this going somewhere or is this not going to be helpful at all? So from that perspective, actually, the way that we’ve released InstructLab is very, is very good. It’s very effective for checking these cases. It it’s very rare that, um, any given company or Enterprise’s needs would be represented, would be a top of mind for the developer of of any given base foundation model. Like does does Meta, you know, care about an insurance company? Well, you know, they probably do, but not it’s not their not their primary, uh, con thinking about just being like, I wonder what AIG thinks about this. Exactly, exactly. So so having a base that’s built for Enterprise but then giving the ability to customize and really focus and and bring in, you know, knowledge and and and particular things you want to do that are specific to. That industry can be really powerful.

Tim Hong: Can I, so we have a few more minutes on this topic. Can I play jerk for a second, right? Because I do think that like, you know, one of the most interesting things about open source is that early on, you know, if you were if you were a government, right, or someone worried about AI ethics or AI safety, right, you basically say, well, the rise of these few leading companies with proprietary models is like really good for us, right? Because we only have to go to a few companies and change their policies in order to sort of secure the ecosystem, right? And I think you might say, well, one of the issues of these increasing proliferation of open source models, right, and the fact that everybody’s kind of going to be running their models on premises, right, is that there’s a lot more room for people to misuse these models. And also like, you might think that also they create all of these supply chain security issues as well. Like I’m kind of thinking about how like, uh, NPM, right. Like other instances in which open source is really taken off, you know, security ends up being this really big problem because like the provenance of any particular component is really difficult and your stack might rely on, you know, hundreds of open source components. And I guess I’m kind of curious, I mean, I don’t think anyone’s got a good solution to this, and and look, I came up as like a free software advocate, so like I I’m on I’m on the side of what’s going on here, but I’d love the kind of panelist of like, you know, offer an opinion about that. Like do you buy that those are risks? I don’t know if there’s kind of smart solutions you guys are thinking about just to kind of wrestle with that a little bit. I think it’s one of the most interesting parts of this development.

David Cox: Yeah, one thing just to start off, um, on the security issue, um, history has proven in open source software that open source ultimately ends up being safer, not less safe. There efforts for instance, to create, you know, private versions of the Linux kernel, and it it turns out it’s just hard to keep those safe because more eyes mean, uh, you know, more more sort of, you know, uh, people who can find, uh, you know, problems, understand problems, and and fix them. So I think having that transparency, enabling the academic community to get involved, to build solutions, uh, for many problems that we may face, I think is super important. I will also say we’re very careful about what we, um, what we release. I mean, we’re we’re we’re very careful about what data goes into these models, uh, before we release them, ensuring that they’re, you know, minimizing the risks, uh, any possible risks around, you know, you know, potentially dangerous, you know, activities. We’re we’re not releasing models that we think are could be used for for for ill intent, of course not. Yeah.

Tim Hong: And I think, uh, I mean, I do think that there’s going to be a need almost for like a Consumer Reports or a Wirecutter for these models at some point where it’s basically like, there’s going to.

Kush Varshney: Be so many models out there that it’s going to literally be like, well, we had a couple experts spend like a few hours really testing this thing, you know, and this is like an important part of the the ecosystem. Kush, it looks like you might want to get in. Yeah, I mean, we actually do work on exactly that, the Consumer Reports sort of idea. So we call it AI Fact Sheets, and Model Risk Assessment. And it is exactly a way to to analyze these different models that are out there, give them different scores along different dimensions. And as a consumer, you can, I mean, really look at different vendors, different sort of options, and get a good sense of of what’s available. So this is actually something already available through through Watson X. Governance, one of our flagship products.

Tim Hong: Yeah, I imagine it’s come some kind of future when I quit my job as a podcast host to be a like a model sommelier. It’s just like, have you considered like this this model for your use case? Fine.

Kush Varshney: Vintage. That’s awesome.

Tim Hong: Yeah, a fine vintage, yeah, exactly, right? Yeah, really good oaky overtones on this 2024 was a good year for LLMs. Yeah, exactly. Um, Marina, any final thoughts before we move to the next topic here.

Marina Deniker: Yeah, I would say that it’s still very early days also with this technology and everything that we’re going into. So especially as scientists, we would like to try not to have the hubris of thinking, yeah, we’ve got this, you know, leave it with us, we’ve we’ve sorted out the rest of this. There’s been so many interesting developments and surprises in this technology in the last few years, and we we think that will continue to be for sure. In that sense, open source is actually going to be more efficient, even from a market standpoint. More eyes means more ideas means more places that this is going to develop in unexpected and interesting ways. So it’s actually even I think more efficient, besides whatever thoughts we may have about the morality of it as well.

Tim Hong: Yeah, no, for sure. And again, I’m kind of arguing against myself because like I’m very pro open source. Um, I think it’s just like a very interesting kind of set of considerations as like the whole architecture of the industry sort of shapes and changes. Well, this is great. So let’s move to the second topic today. I really want to talk about retrieval augmented generation or RAG. So if you’re not familiar with this, RAG is basically one of the hotness in in in AI. Um, if you look at the papers that I clear this year or ACL, um, there are a lot of papers using RAG methods. Um, and you know, I guess Marina, I you keep me honest here. I mean, I think one of the reasons that it has been so prolific and of so much interest is that RAG seems to kind of open a window for solving a lot of the models, the problems that we have with language models, right? Like well, we can’t train these models, pre-train these models all the time, but if they’re really good at pulling data from elsewhere, um, you know, this is a good way of keeping their responses. Up to date. Um, it’s a good way of ensuring that they’re, you know, more factual potentially. Um, and, um, and so I I’m curious because I know your group recently released a paper, um, thinking about and using RAG. And so maybe as a springboard for the conversation, I don’t know if you want to quickly talk about that and then we can kind of more generally talk about, you know, I guess from your point of view, what you see as sort of the existing limitations of RAG and what are the big technical problems that need to be solved.

Marina Deniker: Sure, that sounds great. So, um, the paper that you refer to, it’s a description of a methodology and a system for trying to evaluate it more deeply. Again, the point of RAG is it’s one to be able to have a conversation with an LLM in which you ask it to write a haiku about frogs. They’re great at that, no problem. We he, we live at business use cases, and so it’s very important that when you have business use cases that rely on factual information and it’s really a problem if you get things wrong, this is where you get into RAG. Like you said, being able to point to a reference of, all right, the reason I’m giving you this answer is because this is the content that I am relying on, whether it’s informational or it comes from a knowledge base, whatever. Then you want to actually go and double check, is this going to act the way that I expect it to act? And it’s one thing again to, uh, test these LLM models against large benchmarks. There was some good comments last week about benchmarks and the use, you know, usefulness of them as time goes on. It’s another thing to actually see what happens in a customer’s use case. This is an old data analysis, uh, necessity. You have to go into, okay, what when to the test cases that you’ve created your testing, where did your data come from? What are the documents? How have you managed to without knowing it introduce biases into the evaluation that you’re doing because of the way your annotations are done, because of the way you defined your metrics, because. People have different understandings of what is acceptable, what is not. You have over corrected for a particular query type. You have cor over corrected for a particular way of responding. This is all analysis that you need to do to have confidence in the solution you put out that includes an LLM but is not just the LLM by itself. It’s the LLM as a part of a solution. And so that’s something that my group is does a lot. I know Kush’s group does that a lot as well. Is diving into the details of that, especially how we take our our customers through getting confidence and what does it mean to to deploy their LLM. And our system has a fun for those of us from the 90s, we called Inspector RAGget. Yeah, Inspector Gadget. Um, and it really is a way to to make sure that you can take yourself through that analysis and and feel confidence in what you’re getting, not just the A.G. rate number.

Tim Hong: Yeah, it’s funny about the 90s. I was in a class that a friend was teaching today or earlier this week, and one of the kids was like, I hear back in the day, there’s this thing called GeoCities. I hear it was really cool or something like that. I was like, oh my god, I gotta get out of here. Um, yeah, so there there’s there’s so much to go into there. And I think there’s kind of like maybe two topics we could dive into. You know, I think the first submarine, I’d love to get your thoughts on, is I think one really great theme I think that came up from last week’s episode, episode was kind of the idea that almost AI is in this kind of weird period of like benchmark bankruptcy where like essentially there’s like all of these B benchmarks that no one cares about, and then the benchmarks that do people do care about are like so thoroughly gamed that they basically provide no valid information anymore. And like one outcome that I think Schmit was saying on on the uh on the episode was like, well, that’s one of the reasons why like the solution now is like just talk to the model for 15 minutes and then you figure out whether. Or not it’s good or not. And it strikes me that like I don’t know if you put Inspector RAGget in kind of this context, is like it seems like there’s also a switch from like from benchmarks to like monitoring as the way that we really assess whether or not models are high quality. I don’t know if you’d buy that, because I as I take kind of your group’s work is an attempt to say, okay, well, we’re not going to really, you know, benchmarks are a useful guide, but really in practice, what most people want is to see like lots and lots of telemetry about their models and like that’s how we approach this problem. Um, but kind of curious to get your response on that. Like do you buy the idea that AI is is in a benchmark bankruptcy and do you see kind of RAGget as sort of a solution to that or an answer to that.

Marina Deniker: Yeah, I think you should think of benchmarks is something that you should iterate on rapidly and evolve. Now, the problem with talk to the model for 15 minutes and just get it vibes, uh, kind of feel of it is, uh, people are not very good at coming up with what is the right thing to talk about it to for 15 minutes consistently. They are not very good, they themselves will only think of whatever came into their head, whatever they were talking about to their customer last week, and they will introduce, like I said, a really a lot of biases and what they thought of. Then you end up being very nastily surprised when you actually go ahead and deploy your model and they’re like, well, that didn’t work, but I talked to it for 15 minutes, it seemed fine. You wouldn’t also, um, you know, deploy a representative to a customer after talking to them 50 minutes and thinking that seems fine. So realistically, what you actually want and what I hope the point is of approaches like Inspector RAGget is constant evolving benchmarks. Yeah, talk to it for 15 minutes and then go check yourself. Hey, what data did you end up actually putting in? What kind of questions did you end up putting in? Do you realize that you didn’t do the. Right vibe check. Do that a few times, your vibe check becomes then into something systematic, but it’s something that is that is iterative, that is interactive, rather than some academic somewhere put out a benchmark, I don’t know what this has to do with my data. Make your own, make it iterative and constantly, you know, check yourself for what you’re doing is actually proper quality. That ends up being really the right thing to do. So move yourself from that, um, you know, shout out to Daniel Kahneman, from that system one thinking to the system two thinking, then you’re going to have confidence in what you’re actually deploying.

Tim Hong: Yeah, this is a, I think it’s so interesting because I think this is what you’re describing is going to be an enormous need across like every company that attempts to adopt this stuff. And, you know, I was joking earlier about being a model sommelier, like I think my other business proposal is like, you’re an eval atelier where you’re basically like, we help to craft finely crafted.

Marina Deniker: Evals for like what you need and kind of what you’re talking about because like the art of creating a good benchmark and evolving that benchmark is describing a lot of our jobs actually here at IBM. Is what that’s literally what we’re doing for our cellier.

Tim Hong: So, um, David, Kush, I don’t know if you got responses you want to jump in.

Kush Varshney: Um, maybe I can be a little bit controversial. Um, so yeah, um, I mean, people talk about RAG being a solution for hallucination, for lack of factuality, lack of faithfulness, lack of groundedness, these sort of things. But, um, to me, I mean, it’s part of the solution, but, uh, I don’t think it’s the full solution, because even when you get the retrieve documents, there’s a model in between, and it can ignore those documents, it can get confused by them, it can, uh, I mean, just hallucinate anyways, I mean, all sorts of things. So, um, uh, to me, I mean, what Marina is talking about is very important, not just, I mean, over time, but, uh, like as part of the the the process initially as well, or in runtime, in. In FR time. So mean checking for hallucination separately, um, uh, thinking about, can we trace back the information where did it come from in those documents? Uh, can we even come up with new architectures that, uh, uh, don’t hallucinate, uh, by design in some ways? So I think there’s RAG gets a lot of play right now, but, uh, I think it’s a stepping stone. Um, I don’t think it’s the the end of the journey.

Marina Deniker: I actually completely agree with you. Fully agree with you. RAG has not fixed it. It’s just an additional step in the direction. I completely agree with you.

Tim Hong: Interesting. So do you think in like, I don’t know, it’s always tough to predict on these things. Like in two years we’ll be talking about RAG.

Kush Varshney: I think we will because, um, RAG is like, I mean, it’s search, right? And a lot of the companies who are in the cell LLM game are search companies at the end of the day. So, um, I think it’ll stick around, it’ll have a lot to to to do. But, uh, yeah, I mean, I think for Enterprise use cases, um, maybe it’ll get a little bit. Less emphasis, maybe not, I don’t know.

David Cox: Well, I mean, for freshness of data, some kind of retrieval can be helpful. Like you just added it to the database, you can retrieve it immediately. So there there are more than one problem that RAG solves, and and I agree with, uh, with Christian and Marina that hallucination, that’s that’s a it, you know, it helps a little, but like it’s a separate problem that we need to address lots of different ways. Um, but but the ability to access new information, the ability to customize quickly, I mean, we’re starting to get, uh, I think layers of technology that allow us to to address that. InstructLab is one of them. If you wanted to ingest knowledge into the LLM and build it into your sort of your into the LLM itself, you can do that. But you probably still want to be retrieving things and there’s going to be a balance, and and we’re going to figure that balance out, I think, over the next next couple years.

Tim Hong: Yeah, as it all. Yeah, I think it’s definitely like the much more realistic pathway that I’ve heard, right? Like I think like the other alternatives I’ve heard are like, well, at some point the model will become so big and know everything, and then we’ll be able to pre-train it frequently enough. I’m just like, I really, how many H100s are you going to buy to pull this up? You know, it’s just like, is not within the realm of possibility.

David Cox: So yeah. And it’s not a concept, go ahead, David, sorry. I was just going to say, and not every company’s going to give their data over to OpenAI to let them, you know, their proprietary data. It’s it’s a real problem.

Kush Varshney: Yeah, all I was going to say is I mean, like these sort of ideas, like having multiple levels and layers, I mean, it’s part of computer engineering. I mean, cash and like different types of locality. I mean, this is all like, uh, very much the sort of thinking that, uh, computer people have had. So it just needs to come into to this too.

Marina Deniker: Yeah, orchestration, right? And making sure that there’s routing involved, there’s decisions that evolves, there’s different checks, there’s different guards. That’s not going to go away. I don’t care how long you’ve trained the LLM, that’s not going to be fixed.

Tim Hong: Yeah, for sure. So we have a few minutes left on this topic. I think the last area that I want to kind of push us into is I think Marina, you had kind of a really sort of interesting comment when you’re explaining Inspector RAGget, which is basically this kind of feature of trust, right? Essentially like what does a user need to be shown to trust the model? Um, and I think what I love about that topic is that in some ways it’s it’s pushes you into the realm away from like like it turns out people trust models regardless whether or not they’re a huge parameter model or tiny model. And like, you know, I heard this great anecdote where, um, this MLE was telling me this story like we were doing an eval where there’s these side by sides and what we discovered is that the users that we were testing against just felt that longer outputs were more credible and trustworthy. Regardless of any content they included, right? And I was like, oh, that makes sense because like you’re saying like, do 500 of these tasks in the next hour, and so they’re just using these visual heuristics to evaluate text. And one visual heuristic that we use to tell whether or not something is more substantive is like, is it long and look dense? Um, and I think this is like such an interesting thing because if you go down that route, I mean, I guess it’s a prescription for madness because you’re basically like, well, does font choice influence how people think like how trustworthy their models are? Yeah, exactly. And so I guess Marina, I’d love for you to kind of like riff on that a little bit, is like, you know, how far does this rabbit hole go? Like once you move away from benchmarks and you say we’re going to give you a dashboard of different things, you know, you’re now kind of in almost like the theater of trust. Like what do we need to show you and what is the metric that drives the most trust with. The user, and is that trust justified or not? And just would love to get your thoughts on that as someone who’s working in the space. There’s some interesting psychology here.

Marina Deniker: So we know that people get extremely mad at computers when they make one mistake, but they’re much more okay if you were told that it was a human. So that’s an interesting psychology. There’s another fact that these models are fabulous snake oil salesmen. They will tell you something that you will read it and you’ll believe it, even if in the back of your mind you’re like, wait, isn’t that not what I thought that was? But it will sound so convincing and so accurate that you like, oh yeah, that’s that that’s the right answer. I have no further questions. They’re very good at that. So in that sense, actually, even human evaluation is very challenging. People are bad at catching these kinds of things. On the other hand, you find yourself in Enterprise situation and that’s risky if you really did give incorrect information, that is very risky. It’s again, it’s a good reason that you can’t really deploy these models by themselves with with no support. But I think there’s a lot of psychology actually in setting expectation, just in the same way that when we first had Wikipedia in the world, when we first had Google search, and people thought, oh, well, if it’s on the internet, it’s true. And then people learned. And I think that over time, it’s going to be the same thing where you’re going to learn what kind of things really need to be, what is the right way to interact with these models, what is the safe way, what is the Consumer Reports sort of appropriate way. So some of this is technology, a lot of this is people, a lot of this is people psychology. I cannot give you enough data points and tell you I will never ever ever make a mistake in this model. It’s not possible. So we’re going to have to figure out how it is to set people’s expectations if people are allowed to sometimes make mistakes and you ask for for a clarification. How do we get. To that state of the world also with the use of the technology.

Tim Hong: Yeah, for sure. Yeah, I think and and this sort of pushes into I think a sort of interesting direction is like under certain conditions, I’ll just throw out the hot take, is like under certain conditions, basically optimally safe performance of the model is not necessarily optimally easy to use, I guess, right? So like that is to say like a perfectly articulate model may actually signal more trustworthiness than is warranted. So there actually may be weird situations where there’s kind of this trade-off, which is like, we actually wanted to perform worse because it inspires a an optimal level of doubt in the user, right? Would be sort of the theory that I’m I guess I’m arguing.

Marina Deniker: Well, short of you don’t have to have a perform, if it just could just express its own uncertainty, that would already be a big a big improvement because people really aren’t great at dissociating things like fluency and, you know, convincingness of of of sort. Of discourse with actual truth and particularly in contexts that enterprises are working in with RAG where you’re, you know, taking Enterprise documents, HR documents and policies, and you have to be correct about them unless the person who’s vibe checking that model really understands those policies in perfect detail, it’s it’s very tricky to evaluate and people tend to be fooled very easily. Chris, you want to jump in? I see you kind of.

Kush Varshney: Um, no, I mean, I think the word Marina used earlier, humility, um, is the key. I mean, the AI needs to be humble, um, and we need to be humble as well. So I think that that combination is is the the right way to go.

Tim Hong: Yeah, for sure. Um, yeah, and I think it’s part of the problem too. You know, I was talking with my friend who kind of relayed this story as well, is basically that, um, people have all these expectations that are built up around computers, which makes this particularly difficult. And like the language model behaves in such a fundamentally different way that it’s like. Violating our expectations where you’re like, it’s good at poetry but bad at math. Like that literally flips everything we have built up in terms of intuitions about computers for like the last 20 years. And so like the adage I’ve been using is like everything you think computers are good at, like LLMs are bad at. Everything that LLMs are bad at, like, you know, you know, computers are good at. And there’s kind of this weird mismatch that we’re sort of navigating at the moment. All right, so Kush, uh, you and I are going to kick this topic off. Um, this is going to be the big challenge of the episode. Um, which is if you’ve been watching the more nerdy channels of the AI discourse online, people have been very recently excited by something called, uh, the Kolmogorov-Arnold representation theorem, which is giving rise to a paper that proposes a kind of Kolmogorov-Arnold network or KAN for short. And, um, it’s very difficult to tell from the outside, I think, if you’re not a technical person as to like what it is and why. It’s exciting, which worries me because it has all of the indications of being like, Oh, do you do you use blockchain because like blockchain will solve this, right? Like I think we like rapidly go down that direction. So what I kind of want to do over the next, um, few minutes as we close out this episode is to basically give the clearest, easiest to understand explanation of KANs that anyone has yet articulated on the internet, and we’re going to do this together, right? Okay. Um, so Kush, no pressure on this, no pressure. Um, so I think Kush, maybe the best place to start reading some of the papers is can you give kind of like a quick explanation of why models, large language models, approximate functions? What does that mean exactly.

Kush Varshney: Yeah, that’s precisely the right place to start. So, um, yeah, uh, I mean, so a mathematical function, uh, it’s looking in some space, right? Um, so in middle school or high school, we saw these one-dimensional functions. Um, uh, if our data was just one-dimensional, what that, uh, function is trying. To do is fit that data, and by that, uh, use the function rather than the data to predict the next thing. Um, so by doing so, we’re actually able to, uh, to so-called generalize. So data tells us the pattern, the function describes the pattern, so that the next time we want to, um, make a prediction, we use the function instead of the the past data. So I think that’s a very key point, and then we can go into what those functions are, how to represent those functions, and how to compute those functions. But, uh, yeah, that’s the the starting point, right.

Tim Hong: And a prediction here, just to make it very simple, is like, um, uh, tall people, that’s one variable, tend to be heavier, is that right? Like, for example, that’s that’s a prediction that you could build a function around.

Kush Varshney: Yeah, exactly. And you can be very quantitative about that. So if, uh, I’m 6 feet tall, maybe that predicts that I’m 180 lbs or something like that. So, um, yeah, so then I think the next step, we’re going to take the step by step, we think through this problem step by step, right? Uh, is basically, um, so as I take it, right, like one of the things that machine learning has really done better than kind of traditional models of AI or traditional models of, you know, computer programming is that we’ve we frequently tried to kind of like hand draft all of these rules, right? So like you want to write an algorithm to divide, you know, pictures of cats from pictures of dogs, you would do feature engineering, right? You get a bunch of people together to kind of like write these equations, these functions out, right? And I guess, is it right to say that machine learning what it’s been really good at doing is like coming up with these functions on its own, right? Like to basically come up with those rules on its own to do this prediction.

Kush Varshney: Yeah, I think that’s a good way to put it. I’ll be a little bit more specific though. Um, so we as humans, um, the algorithm designers, et cetera, um, have been the ones who come up with what the functions are. Um, the functions have parameters and it’s. The learning algorithm that’s figuring out the parameters to best, uh, best fit the data. But at some level, um, we the the computer scientists, the folks are the ones who decided what were the functions, uh, in this library or this universe of of possible functions, and then let the, uh, the algorithm figure out the, uh, uh, the nuance, the the parameters and so forth.

Tim Hong: Yeah, for sure. And so what we what we do here, I guess, right now in the world of AI is multi-layer perceptrons, right? Which is like this very particular way of implementing AI that is doing the approximation of these complex functions on its own and can solve all of these magical things, right? Like it can, you know, have conversations with you, it can sort pictures of cats from dogs, whatever, whatever you want. Do you talk a little bit about maybe like the trade-offs of that? Like what what do we need to do in order to achieve that magic, right? Like, um, yeah, yeah.

Kush Varshney: So, um, there used to be this, uh, car dealership that had a commercial one in the 90s, again.

Tim Hong: Um, so they used to dating all of ourselves here. I’m yeah, so they used to say, um.

Kush Varshney: Stack them deep, sell them cheap, uh, in terms of their cars. Um, so, uh, so with these, uh, the multi-layer perceptrons or the feed-forward neural networks, um, uh, the trend has been just, uh, there’s this one kind of these layers, what they do is, uh, they multiply some inputs by some weights, um, add them up and then apply a nonlinearity, um, something often which is called a ReLU function, a a rectified linear unit, which kind of, um, changes the output, right? So you have those, you layer them on top of each other, you stack them really deep, um, you keep doing this, keep doing this, keep doing this, um, and that way, uh, you actually end up with the ability to, um, uh, end up with almost any, uh, sort of nonlinear function, uh, to describe the data. So, uh, you talked about, uh, this universal representation or approximation theorems and stuff. So, um, uh, you can actually prove that through even not very deep neural networks, um, uh, you’re able to represent. Any function, uh, that that you have in front of you. Yeah, so that has been.

Tim Hong: Seems like we’re at the magic, right? Like this is this is how it works. Um, I guess one result of that, right, is that these models have been like they’re really expensive to make, like you need like a lot of energy and a lot of chips and a lot of data and a lot of computing time. Um, and so along comes Kolmogorov and Arnold, actually don’t know who those people are, I just know them from their representation theorem.

Kush Varshney: You should know Kolmogorov from a lot of other things, yeah.

Tim Hong: Oh, he’s that one. Okay, same guy. Okay, all right, yeah, I know that guy. Uh, it’s the Big Lebowski, yeah, no. Um, so, uh, tell me about the representation theorem, right? What does it, what does it tell us? What’s what’s the big deal.

Kush Varshney: Yeah, so, um, like we just were talking about, so when you have some function that you’re trying to represent, um, mathematicians have come up with all sorts of ways to, um, be able to decompose that goal of representing this function I have in front of. Me in terms of more primitive, uh, other functions, right? So, uh, this is, I mean, we see it, uh, as an electrical engineer, I see it, um, like Fourier transforms or Fourier series are ways of decomposing a function into, uh, sines and cosine functions. Um, the same way, um, uh, with the, uh, uh, the multi-layer perceptrons, it’s into that particular structure. You’re decomposing into these weights and, um, these nonlinearities. So what, uh, the Kolmogorov-Arnold representation theorem is about is decomposing again any function into, um, some other functions, in this case, they happen to be one-dimensional. Um, so, uh, they could be splines or some other smooth one-dimensional function, and by combining them, uh, in a particular, uh, summation, uh, you can again, uh, represent any multi-dimensional nonlinear function. And that’s the proof, um, is what, uh, what Kolmogorov tells us, that, uh, in this way of taking 1D functions, you can come up and represent any, uh, multi-dimensional function.

Tim Hong: Yeah, which is which is pretty wild, right? Because what you’re sort of telling me is basically that like the machine learning models that we have now can work this magic, right? They basically come up with a magic mathematical formula that can help you tell pictures of cats from dogs. Let’s just take that example. And then kind of what you’re saying is that we can take that magic formula and like represent it, like we can break it down to these tiny, tiny, tiny Lego blocks, right? Like all the way down to what we were just talking about, like tall people or heavier, like single variable stuff, right? And and kind of the theory and course you keep me honest, you’re the one who actually understands this stuff, is like like any, I don’t know, or like most very complex, uh, formulas can can be reducible in this way. I think is what the theorem is saying, is that right.

Kush Varshney: Exactly. Okay, so we’re there. Uh, the KAN Network then is a network that attempts to do that. Yeah, exactly. So, um, uh, so when in the regular neural network we had, um, these weights, um, on these edges that are multiplying the. Inputs, here we’re applying that function, the splines or the other sort of winding, um, uh, sort of nonlinear function, and then instead of, uh, I mean, you add in the same way. So the KAN is also adding up the inputs, but then there’s no additional, um, nonlinearity afterwards, um, like the ReLU that we see, um, in in the normal neural network. So all the nonlinearity is done, um, before you add things up. Uh, so it’s just more complication in one place rather than in a in a different place. And so, uh, just by doing that change, you can, um, uh, reduce the number of parameters, um, because, uh, this more complicated thing on the edges, uh, is actually, uh, more able to represent the these weird various sort of behaviors. So so that’s kind of what it’s, uh, what it’s trying to do.

Tim Hong: Nice, so we’re there. Deep breath. Yes. Two last questions. Yes. Does it matter what what would what would KAN models, what’s what’s the promise of KAN models if that you can do this? It seems like you’ve just taken this complex thing and turn it into a bunch. Of Lego blocks. So from my one brain cell standpoint, I’m like, isn’t that kind of the same thing.

Kush Varshney: Yeah, um, so it it’s true. I mean, you’re just shifting the nonlinearities from one place to a different place. Um, one thing that the KANs, uh, are able to do better a little bit is, um, more interpretability. So, um, when you look at those splines, uh, they actually make sense to us. Um, so that, uh, height and weight sort of relationship or any of those other things, um, uh, we can understand better. Uh, so there’s this interpretability method called SHAP, which, um, people have been using for a while. Um, this is like automatic SHAP without having to do SHAP in a sense, um.

Tim Hong: Inability there for some folks who are not maybe as into the papers is just simply understanding what the models, why the model is doing what it’s doing.

Kush Varshney: Yeah, exactly, exactly, yep. Um, and so I mean, that’s one advantage. Um, the disadvantage though is that, uh, uh, our hardware infrastructure has not been optimized, um, for, uh, uh, for these sort of things, for these splines and and so forth. Um, whereas the matrix vector computations for neural networks, um, are, um, uh, kind of, uh, like very highly optimized through those, uh, H100s and so forth. So, uh, so that’s, I think, the difference. We might, if this catches on, I mean, develop some hardware for this type of thing as well. Um, and, uh, I mean, the last point that I’ll make is these are not new ideas. I mean, this is something that’s been around, uh, uh, even our team, uh, like a couple years ago, um, we developed something called CofferNets. It uses continued fractions, which are, um, a third way of representing functions. Also, it has a approximation or, um, universal approximation theorem associated with it. Um, uses continued fractions that have been known since antiquity. Like the ancient Indians and ancient Greeks knew about all this stuff. So, I mean, like all this fancy math is great, and, um, uh, it’s just different ways of putting together, I mean, different, uh, of these functions together. And then at the end of the day, um, uh, they all. Let you, I mean, kind of represent, uh, these different nonlinear functions. How you train them, uh, might be more or less costly, where the interpretability is might be more or less easy to hard. So, uh, so I mean, it could turn into something, uh, it might just be another option. Um, so we we’ll see.

Tim Hong: Yeah, for sure. Yeah, it’s fascinating. And that I think opens up definitely a direction that I hadn’t really thought of because I think the main thing I had heard is, well, you can make much more energy-efficient models, right? But, um, it seems like two things you’re pointing out. One of them is like we might be able to understand why these models are making the decisions they do at a much like closer level of depth than we have in the past, which seems which seems huge. Um, and then I think the second point is actually that this is like not new stuff, right? Like I mean, much like neural nets themselves, this is like we’re just like pulling all this old stuff back again and being like, Oh, I guess it works now.

David Cox: You know, ultimately, well, I think that resonates really well with Chris’s point about the hardware match. I mean, often times, uh, you know, we get success in the field moves not because something is the mathematically optimal thing, but it’s something that can be done sort of irrational scale with irrational speed. And and as you say, deep, you know, deep learning was, you know, kind of a rebrand of artificial neural networks. It was around for decades before it caught on. Why did it catch on? It’s not because there was some mathematical breakthroughs, because the hardware like GPUs by accident were really good at doing this. And then that just sort of set us on a path. So, um, obviously all these new developments are really exciting, and we could build different hardware potentially. Um, but you know, any new idea like this is going to compete against how wonderfully good GPUs are at doing the basic computations needed for for deep learning. Um, so, you know, that there’s it’s an interesting battle, you know. An interesting set of trade-offs there.

Tim Hong: Yeah, that I think relationship between sort of like hardware and what’s happening on the model side, I think is like one of the most interesting aspects of this, and like how long does it take for a model to influence hardware? You know, are we just locked into CUDA for the rest of our lives? It’s like all these things are like very, very interesting questions. Um, so, uh, Marina, any last thoughts, uh, before we we close up today.

Marina Deniker: Yeah, um, even more general comment, continuing what Christian and David said, is representations of data are not created equal. So yes, it’s the same information, but when you change the way that you represent it, you’re able to do things with it that you weren’t able to before. So even with something like a large language model, you’re representing data that exists, let’s say, on the internet, but you’re representing it in such a way that you can access it in a way that you couldn’t before. Same thing with, for example, KAN versus MLP. The representation changes. There’s going to be trade-offs, but it’s always very interesting to try this. The fact that we now have more of these options open to us because the hardware has caught up to the math that has been around for years or decades or centuries. Yeah, that means try again, try again, try again, see what what new things will come up. Data representation is really one of the things underlying driving this current era of AI. So more work in this direction is just going to continue to drive things an interesting place.

Tim Hong: That’s great. Yeah, well, I can’t think of a better not to end on. Um, Kush, MVP, thank you for coming on the show again. Um, and Marina, David, hope to have you on the show again. And, uh, I hope all of you listeners out there join us next week for another episode of Mixture of Experts. Thanks, everyone.

David Cox: Thanks.

Kush Varshney: Thank you, appreciate it.