Tim Hwang: So it turns out that Apple is starting with AI pretty modestly. It’s not going to get me to buy a phone. Alright, so level with us: how hard is it to make agents actually work? I think control is going to be the key thing that makes or breaks autonomous agents. I think there’s going to be a lot more fraud coming. Benchmarks are necessary; we need to all agree on a particular thing we’re looking at. They are not sufficient. All that and more on today’s episode of Mixture of Experts.

Tim Hwang: Hello everybody, I’m Tim Hwang, and I’m joined today, as I am every Friday, by a world-class panel of researchers, product leaders, and more to hash out the week’s news in AI. Kate Soule is a Program Director for Generative AI Research. Hello, Kate. Marina Danilevsky is a Senior Research Scientist. And Maya Murad is a Product Manager for AI Incubation.

Okay, so as we usually do, we’re going to start with a quick round-the-horn question. The question is: if you have an iPhone, will Siri ever be any good? Maya, what do you think, yes or no?

Maya Murad: I think so. I think Apple has a great track record of amazing user experiences, and I know they took their time with their AI, but I know it’s for the benefit of how I interact with my phone. Definitely.

Tim Hwang: Marina?

Marina Danilevsky: Maybe. I still find myself fighting with Siri a whole lot and giving up on it most of the time. It’s really great for reminders, for sure.

Tim Hwang: And Kate, what do you think?

Kate Soule: Yes, assuming they can get to the user customization.

Tim Hwang: Well, let’s just get into it. Our first story of the day is to talk about the Apple Intelligence updates of this week. The background is that there was a big WWDC announcement earlier in the year announcing Apple’s long-awaited drive into artificial intelligence. This week, we saw a slew of announcements continuing to hype Apple Intelligence in the context of the new iPhone 16 release. They announced a whole slew of things: LLM assistance on text, image search, image generation, and what we were just talking about, a Siri update.

I kind of want to ask you, just as users of these products—we were talking before the show, and all the panelists, myself included, have an iPhone—Maya, are you excited about any of the features on the way? And if so, why?

Maya Murad: It’s not really enough for me right now to go out and buy a new iPhone and try to get those features. The features that are there—while it’s nice to have the LLM locally, and I always like Apple’s stance on privacy—what can they do? Alright, we can rephrase text, we can summarize email, we can generate emojis, which would be really fun when texting my kids. None of it is still in the “I’m going to pay $1,000 now for a phone” category. I’m not going to pay $1,000 for custom emojis.

They’re helpers, and the helpers are nice, and I’ll appreciate it when I get it, but it’s not enough for me to go, “Wow, this was a game-changer by Apple.” At least that was my feeling.

Tim Hwang: Yeah, that’s kind of the most interesting thing. There are almost two points of view: one is that Apple’s getting it all wrong, that AI is the killer feature that will sell phones. The other is that they have it exactly right—none of the current AI features are good enough to motivate someone to buy a phone; the software isn’t pushing the hardware here.

Marina Danilevsky: Well, from my perspective, it’s not going to get me to buy a phone; it’s not something that’s going to push the boundaries significantly. But at least it gives Apple an entry into what they’ve largely been standing away from. So I think it’s definitely helping them move in the right direction. Hopefully... I just don’t even use Siri today because it’s always like a 50/50 shot. Is Siri gonna...?

Tim Hwang: Yeah, I don’t even use it either. They’re pushing it with the button and stuff, but I never touch it.

Marina Danilevsky: We need to get to the point of basic Siri usability, and I don’t think you can get there without an LLM. So I think they’re making the right move from that call, and later on, they have more opportunity, once they lock that in, to actually differentiate both hardware and software with AI.

Tim Hwang: Yeah, for sure. Maya, what do you think? One way of looking at this is that Apple is a hardware company, so they’re very careful by nature because if you mess up hardware, you really mess it up. But are they maybe too slow? By the time Apple gets this stuff done, it’s going to be OpenAI on the phone, Anthropic on the phone, Perplexity on the phone.

Maya Murad: I think that’s a good question. Part of Apple’s ethos and appeal, especially when they started decades ago, is this focus on design and user experience, and doing less. They always did less compared to competitors. I think in the space where user experience matters, we’re overloaded cognitively with too many things on our phones.

I’m really excited about Siri being able to navigate different apps. If they nail that, for me that delivers tremendous value. I don’t see them being in the hot spot to be able to respond to generative AI in the same way. They’re not search; their core is not search. They’re not a chatbot in itself. They’re a hardware way for me to communicate, a series of add-ons. So I don’t think they’re feeling the same pressure as some of the other tech players are.

Tim Hwang: Yeah, for sure. It’s going to be an interesting situation. The other week, I tried out all these AI services, signed up for a bunch of subscriptions, and the first month’s bill just came in, and I’m like, “This is really bad.” But it kind of feels like if Apple can release some of these features for free, it totally changes the economics. Kate, I see you nodding.

Maya, does Apple have an advantage, given that they are the integrator between all these apps, beyond what any one app like a Perplexity or Anthropic app that you could install might have?

Maya Murad: Yeah, I think absolutely. They own the ecosystem of apps; they own the App Store. I think if they could lower the barrier in the interface of connecting between different apps, that would be really interesting. I wonder if in a year from now, Apple will be the AI killer in the same way that when OpenAI launches something, it kills a bunch of startups.

Tim Hwang: I mean, one thing I think about is there have been all these demos floating around that are often more impressive than they are in practice—like, “type what you want and a new app emerges.” It feels like that might eventually happen on Apple, but it’s also an enormous threat to the whole edifice of the App Store they’ve constructed. Navigating that is going to be really complex and challenging.

Kate Soule: I think Maya brings up a really good point. They’re more likely to gatekeep the App Store until they’ve got their own integrations working, and then it’ll be all about how seamless it is. Otherwise, if you have different services trying to talk to different apps, there’s so much under-the-hood middleware that you’ll have so many points of failure. People will be like, “Alright, well, the Apple version can’t do as much, but at least it works.” There seems to be a real opportunity there.

Tim Hwang: Yeah, for sure. It reminds me of the debate years ago about self-driving cars. It was always “five years away.” One interesting debate was whether we needed to reconstruct the environment to make it simpler for the robot cars or just let the robot car roam and train it in all environments. There’s a similar thing here for agents with Apple: they confront a very heterogeneous situation with the App Store, which prevents them from enforcing clean agent experiences. But if anyone can do it, it’s Apple, because they have the most control over the space.

Well, that’s a really good segue to our second topic. One of the reasons we were excited to have you on the show, Maya, is that you’ve been working on agents. Literally, in the last 10 episodes of Mixture of Experts, people have been saying, “Agents are on the way, agents are the new big thing.” We’ve debated it, but you’ve actually been working on them. The circle of people talking about agents and the people actually working on agents is often very different, and you’re rare in that respect.

Do you want to tell us a little about that work? I’m curious to learn about it and what you’re learning.

Maya Murad: Yeah, of course. So I sit on a really interesting team in research; my team focuses on incubating new technologies and opening up market opportunities for IBM, and we’ve been focusing on agents for several months now. One of the first things we did this month is we open-sourced a framework for building agentic applications. It’s still very early days; we did a silent drop, but we think we have some interesting features that reflect our learnings.

There’s a lot we learned along the way. It’s very hard to bring agentic applications into production, and it’s easy to take it for granted. In terms of operational complexity, this is a step-change from fixed flows. It’s not incremental; it’s another paradigm. It’s much harder than fixed flows in terms of implementation.

Tim Hwang: We’d love to talk about learnings; we’ve got time for it. This is where experts can shine. Maybe one way to cut through it is: is there something you found surprisingly hard? Where you thought, “Ah, we can nail that,” but it turned out to be really difficult.

Maya Murad: So there were two things that were a blind spot for us; we didn’t expect how hard they would be. And then one thing made it very clear how to bring this into production.

First, an agent is underpinned by a prompt—a set of instructions telling it how to behave. Let’s say you built an agent around Model A and optimized it. If you want to bring it to Model B, the whole thing breaks. We had this experience firsthand. We started with LLaMA 3, moved to LLaMA 3.1, which we expected to be an incremental change. Nothing much changed, but the whole thing broke. It took us three weeks to re-optimize everything under the hood, and we’re still not fully there.

This is critical because if you want to stay on top of the latest models, you have all this cost related to changing models, which is prohibitive. This is pushing us in a direction where once you pick your model and build something in production, it will be really hard to change model providers. I’m not very happy with that being the status quo, so that’s one part, and we have ideas on how to overcome it.

The other part, which we discussed with Kate this morning, is we take for granted how to build with AI. In traditional software engineering, I specify my features, code them, test them, and I’m done. With LLMs, I have features I didn’t sign up for—like outputting hate speech—or useful features like summarization that come out of the box. But did I test for them? No, we took them for granted and didn’t follow a test-driven approach. I want to pass it to Kate because she works a lot in prompt optimization and I’m sure has had struggles with this.

Kate Soule: Yeah, thanks, Maya. One thing I found interesting is how to think about the hierarchy of how a model or agent is aligned. There’s the work the model provider does to train it to be safe and good at tasks like summarization; they enforce a perspective on how the model should behave. Then you have the alignment preferences set by the agent builder—how this model, with this system prompt, will behave in this agentic environment. Then there’s a third level: when a user interacts with the agent, they have their own preferences.

This gets back to Siri: does Apple have the ability to personalize to an individual user? A user might want things short and in bullets, or in markdown with tables. There are different tiers to account for when building. You have control over some parts, like your system prompt design, and you can create tools for user personalization. But then there are things you don’t control, defined by the model builder. That’s where interesting challenges come out—navigating these different levels of control, especially if you envision needing to switch models, where a different provider might have a different process.

Tim Hwang: It almost presages a kind of “legacy code” or “legacy models” situation. The intention is always to move to the next great capability model, but your story suggests it changes the agent’s behavior so much that you may not want to due to uncertainty or the evaluation burden.

Maya Murad: Well, one thought is the notion of backward compatibility doesn’t really exist in LLMs; we haven’t thought about it much. But we know it’s a big deal in software. The notion of “did everything immediately break?” is not all that useful. If we’re going to do this seriously, we’ll have to take that into account, creating a whole new slew of benchmarks, functionalities, and tests. “It was working this way before; how will it work if you plug it into something old?”

Another thing I agree with Kate on is the notion of control. Think about generative AI with art: you can give a prompt to make a picture, but you can’t say, “I love it, but just change that little thing over there.” That’s not how these models work. In software, you might say, “You did a lot right, but I need you to change this little bit.” It’s not going to work like that at a basic level.

Both of these things are a very different way of looking at software building and need to be thought of carefully to make sure it’s practical over time.

Tim Hwang: Marina, I was wondering if you had a perspective. Does getting towards GPT-style structured outputs start to solve some of the backward compatibility? If we have greater structure on prompts and outputs, does that solve the problem?

Marina Danilevsky: It’s a step. Part of it is the structure of the output, but another large part is the acceptable states and constraints on what makes sense. Even with structure, the content could still be anything—we’re still talking strings or primitives. There’s still a notion of what states in your application are okay or not okay.

In a deterministic flow, you write it all up, you know what will happen, you have tests and catchers. How does that look here? That’s the next thing on my mind.

Maya Murad: Yeah, I think both of you raised a great point. I loved how much you spoke about control. I think control is going to be the key thing that makes or breaks autonomous agents. These things can run wild and have costly consequences. We’ve seen firsthand how data you thought was proprietary gets sent to an external tool and goes to a third party unintentionally.

One of my thoughts is I don’t know if fully autonomous agents will go into production this year. It might be more of a hybrid, compound AI system where some parts are agentic—giving the LLM degrees of freedom—and other parts are more prescriptive, with verifiers that allow us to get the level of control we want. That’s what we’re starting to explore because, with the underlying models we have, I don’t think we can have fully autonomous agents safe in production. That’s where I’d put my money right now.

Tim Hwang: Yeah, for sure. It seems we’re going to enter an era of pseudo-agents, with agent-like elements but actually quite deterministic in ways, and that could persist for a long time. My ultimate leer dream is the agent unshackled, but the issues you’re pointing out are deep and categorical. Do you agree?

Maya Murad: Yeah, definitely not in the camp of “agents and AI unshackled.” I think AI has to serve us and fit our needs. We need to understand how it works and ensure it works in accordance with our values. That’s the type of AI I’d prescribe and aligns with my worldview.

Tim Hwang: Yeah, for sure. Before we move on, Maya, you said you soft-launched this framework. Do you want to direct our listeners to it, or is it still a teaser?

Maya Murad: More than a teaser. We launched it with a silent drop; we haven’t shared it broadly.

Tim Hwang: If this is the first moment you’re sharing it...

Maya Murad: Yeah, first moment I’m sharing this. If you’re listening, it’s called the BStack, specifically the B Agent Framework. You can do cool things out of the box: create an agent that can plan, use tools, and correct itself. We have use cases, but we have exciting updates coming. Along the lines of solving the cost of switching models, we want to reduce that friction. We’re working on bringing Enterprise controls. If people are interested in joining this journey, I’m very open to it. It’s the beginning steps, but we’re excited about what we can learn. My team... it’s B, as in bees. Our team likes naming things with puns. So bees, like worker bees, and maybe there are hives and all that.

Tim Hwang: So to move us to our next story: this week, a New York City-based startup called HyperWrite released a model called Reflection 70B. It was widely touted; the company leader said this model integrated a new method called “reflection tuning,” the secret sauce that allowed it to hit crazy good metrics on all major benchmarks. There was a lot of hype, as most models rising on leaderboards get nowadays.

Then immediately, there was a turn. People said, “Wait, we tried to reproduce these results, and it seems nowhere near what you’re claiming.” Furthermore, digging in, it seemed like they just did some bargain-bin fine-tuning on open-source models. In true Twitter-driven media cycle fashion, there’s been mutual recrimination and dispute. The end result seems to be a startup that went on publicly available benchmarks—which everybody uses to evaluate model quality—and may have engaged in shading the numbers to make their model look better.

This is interesting because these leaderboards have become the benchmark for telling who’s advancing the state-of-the-art. Marina, maybe we’ll toss it to you first: should we be worried we’ll see more of this? The value of gaming these metrics is always rising. Regardless of intentional fraud, are there bad incentives? Is this something we should care about, or is it just what happens?

Marina Danilevsky: To some extent, this is what happens. But I was really happy to see so many folks jump on right away to say, “No, I’m going to try to reproduce the results,” “You need to upload your weights,” “What about this?” That is science acting correctly. Good science is supposed to be reproducible. While it’s possible to have a hype cycle, it was nice to see immediate checking from third parties.

In previous years, decades, centuries, the cycles for checking work were very slow. In our field, it’s actually easy to quickly start checking. That was a nice thing.

Another thing about benchmarks: they are a temporal proxy of a specific slice of the world with many things held constant. We’re trying to check performance on a particular thing under artificial controls. Are they useful? Absolutely. Are they sufficient? No. In science, we talk about necessary and insufficient. Benchmarks are necessary; we need to agree on what we’re looking at. They are not sufficient, nor should they be. The point is to have a benchmark, then another, then another, to check each other, stay honest, and motivate exploring the holes. What’s the next thing to look at? I found this very satisfying; the system is working, basically.

Tim Hwang: Yeah, and this is maybe another way at the problem. I remember at NeurIPS years ago, there was a push to say machine learning has a big reproducibility crisis. If anything, this story is the other direction—we’re seeing emergent reproducibility in the rough-and-tumble of Twitter or X. Kate, you’re nodding. Do you think the problem is solved for reproducibility, or is it still a persistent worry?

Kate Soule: I think it’s encouraging that we have such an active community focused, as Marina said, on good science, checking and validating. From my perspective, a bigger issue isn’t just reproducibility but transparency. How was this model trained? How is it communicated? We need to move as a field—and there are good actors—but clearly, in some cases, we haven’t gotten there yet. It’s not just dropping a benchmark; you have to be transparent and open about your methods, approach, training data.

Right now, the norm is to train behind a black box, put an API out, and say, “Trust us, it works.” Can you imagine that in other industries? “Here’s an airplane; trust us, we tested it, it’s fine.” We need more openness in how these are trained. There will always be misaligned incentives with benchmarks because they get so much attraction. We need it paired with open discourse on what was actually done and the ability to inspect it.

Tim Hwang: Yeah, this seems to be the crux. Benchmarks have represented a compromise: companies have trade secrets but show performance on benchmarks as transparency. Kate, you’re saying we should expect more than just benchmarks.

Kate Soule: Yeah, there’s a lot of “transparency in name only.” A company says, “We’ll publish a paper to follow,” or gives a high-level overview. But if you don’t share the weights or get into the weeds of what was done, it can be surface-level. Did you deliver details that help scientists reproduce it? That’s the level of transparency we need.

Maya Murad: I’m in the business of building AI applications for production. For me, seeing a model’s performance on a benchmark isn’t useful because a number of things could be happening: the model might have seen the data before, or it might not generalize to my use cases. My ethos around benchmarks is: if there’s a test dataset that fits a feature I’m developing—like improving reasoning or tool-calling—that helps me identify blind spots, I’ll go all in. But it’s so important that we’re investing in building our own test cases and evaluation criteria because we have a specific goal. Nothing beats doing that. Even when selecting LLMs, benchmarks are nice to have a sub-selection of models to look at, but as Marina said, it’s helpful but not a complete signal.

Tim Hwang: Yeah, there’s an interesting phenomenon I’ve been chasing: it used to be that the limiting reagent was getting new models out, but now models are everywhere. The new limiting reagent is a well-crafted eval or benchmark set; that’s increasingly becoming the bottleneck in workflows.

Kate Soule: Well, and I wonder, as models across the board get better—if you look at the state of the field, models are doing what last year required a model 10 times bigger—performances are improving. We’re starting to get into the zone of commoditization, leading to a race to the bottom, trying to inch up a metric by 0.01% that might not be informative for your use case, as everyone tries to show differentiation. It’s becoming more difficult as performance improves on the bread-and-butter, low-hanging fruit tasks that any model can do now.

Tim Hwang: Yeah, for sure. Sometimes I look at these benchmarks and think, “What are we doing here? What are we spending time on?”

So, at Mixture of Experts, we always try to do a paper as one of our stories. I want to end with a paper that just came out, titled “Can LLMs Generate Novel Research Ideas?” This is a continuation of a paper we talked about last week about using AI for science. The big debate is the interesting question of whether LLMs can be creative—can they become a partner that pushes us in new research directions we wouldn’t have gone before?

This is particularly interesting because there’s a parallel discussion. Some of you may have seen an article by sci-fi writer Ted Chiang in the New Yorker arguing that LLMs can’t be creative because they don’t make intentional choices about their outputs.

To quickly sum up the paper: their argument is they played around with LLMs, and it seems they can generate creative prompts. Their spicy claim is sometimes even more creative prompts than humans. They are bullish on the promise of LLMs assisting research at the very beginning, the most human part of the work.

The first question is: do we buy it? Kate, maybe start with you. Do you buy this claim?

Kate Soule: So, the paper makes interesting assessments of how a model can support research tasks. But what struck me most is they had to search through 4,000-plus examples to get 200 unique research topics. While the models were good at creating—they focused on novelty, new ideas human subjects in the study hadn’t thought of—is it that models are more creative, or are we just brute-forcing, automating a search through thousands of scenarios until we get a unique number? You can roll dice until a unique number comes up.

That’s where there’s room for debate on what it means to be creative or novel. Did the humans get a fair shake?

Tim Hwang: Marina, as a researcher, do you think this is a tool you might use, or is it mostly game-playing? If I’m uncharitable, is it just a magic eight-ball? If it generates something unique and great, awesome, but is there anything uniquely creative about it?

Marina Danilevsky: I don’t want to say “a broken clock is correct twice a day,” but if you do something enough, you’ll stumble across something new. Does that mean something is more creative? Especially with the New Yorker article you brought up, where creativity is a choice—are models creative, or are they a tool for brute-force searching through a larger number of randomly generated ideas? To me, it’s the second one, which is valuable.

It’s valuable because it can take humans a lot of time, and we come in with biases. If something is brought up, you can immediately say, “That makes no sense,” or “I didn’t think about it that way.” That’s valuable, but there’s no judgment from these models.

I know this has been used in other science fields like medicine or chemical compounds, where there are thousands of possibilities, and humans don’t have time. They’re used as a filter, a brute-force method to get down to what’s possible, and then human intuition—based on 20 years of experience—says, “This is an idea worth pursuing.” Can I tell you why? Not to the extent I can prompt an LLM, but it’s a sum of my experiences. That’s great and useful, but creativity means intent, and there is no intent.

Tim Hwang: Yeah, that’s right. So you kind of buy the paper’s utility but question calling it “creativity.” The problem is giving it a word with baggage, like the great creative artist or scientist. The paper itself is not creative; applying machine learning to help sift through stuff and say what’s not garbage isn’t new. They’re applying it to a specific use case, which is great and useful, but that part isn’t new.

Marina Danilevsky: Yeah, asking machine learning to help you go through a bunch of stuff and say what’s definitely not garbage, or “here are other things I’ve tried”—that’s not new. They’re applying it to an extremely specific use case; that’s great, but that part to me is not new, while being useful.

Tim Hwang: Maya, maybe I’ll turn to you. This conversation makes me think of debates about interpretability 10 years ago. One line was, “You don’t want to use systems if you don’t understand how they work.” Another group of ML chauvinists said, “If the model always succeeds, why care how it’s done?” There’s a similar bias with AI and creativity: we don’t just want the right answer; we want it in the right way. In the creative context, we want it to be more than a random number generator.

Do you fall on one side? If these tools help get more creative results, are you happy? Or should part of our research agenda be to get systems to be Capital-C creative? What does that mean? It’s a big question.

Maya Murad: I don’t have a blanket answer; it’s context-dependent. In the arts world, source attribution is really important. A lot of artwork is done in the style of an existing artist; credit should be given. We want to assign this characteristic to AI because we have social values regarding IP and giving credit. We want to align AI systems with how our world functions and how we give credit where it’s due.

So I think it depends on the cost of not having that explainability—and not just explainability, but other values that matter to us, ensuring the technology aligns with our societies, not us adapting around it.

Tim Hwang: Yeah, that makes a lot of sense. It’ll be an interesting struggle. Where this goes is: “Model, you came up with an intuitive, counterintuitive, creative result. Can you explain why or how you reached it?” That kind of interpretability starts to look like chain-of-thought.

Maya Murad: I think we’re giving too much credit to these systems. The paper about generating novel ideas—first, what do you mean by novelty? Is it just something net new, and is that useful? How the system works is just the statistical probability of the next word. When we come up with novel ideas, there’s meaning, value, and intent behind it. I’m trying to push forward an idea I care about. It’s tough to equate the two and put them on the same standing. One can be a tool for the other to give you an idea you didn’t think about, but I don’t think one replaces the other.

Tim Hwang: Yeah, for sure. This plays into the “stochastic parrots” debate. Some say they’re just that; others say they’re more. But Maya, you’re outlining a third path: yeah, they’re stochastic parrots, but that’s really powerful. Let’s not downplay that; the stochastic parrot is incredibly useful in certain domains. We shouldn’t sell that short.

Maya Murad: Yeah, absolutely. I think this whole industry of generative AI was unlocked by how good these stochastic parrots were. I read that initial paper, but what took us all back was that this was going to have really useful applications if implemented right. But it doesn’t mean it can inherently assign reason and intent. I don’t think we’re in a world where we’re there.

Tim Hwang: Well, Marina, I’ll let you have the last word. In four or five years, are you going to have an LLM co-author on a paper, or is that a total pipe dream?

Marina Danilevsky: No, and I don’t think that’s the right aim. I don’t think I’ll have that co-author. But it could be that we’re asked, when publishing now, to say if we’ve used AI in our work. An LLM could help sift through related work, manage your bibliography, figure out similar and different things—sure. But to go with what Maya and Kate said about intent: that’s not what the technology can do. Intent comes from beings that are alive, that can give it. That’s why you can’t have actual AI art. Art moves us because of the intent behind the person who made it. AI art now—you can feel the care and intent behind the people who made the LLM. Think of the effort we pour into making something useful. The intent is not in the technology itself; it’s in the people trying to create something intended to be used, helpful, efficient, and effective. That’s where the intent is, not the tech itself.

Tim Hwang: That’s great. I’m applauding. Maya, Kate, Marina, in a nightmare landscape of jargon and hype, this panel is a light in the darkness. I appreciate you all taking the time this morning to stop by Mixture of Experts. Hopefully, we’ll have you on again in the future.

And for all you listeners out there, if you enjoyed what you heard, you can get us on Apple Podcasts, Spotify, and podcast platforms everywhere. We will see you next week on Mixture of Experts.