Tim Hwang: What would be the right security strategy here? Are we shifting from “how do we find bugs” to “how do we govern the AI that finds our bugs”? All that and more on today’s Mixture of Experts. I’m Tim Hwang and welcome to another action-packed episode of Mixture of Experts. Each week, MoE brings together a group of the smartest thinkers working at the frontiers of artificial intelligence to make sense of the week’s news. On this week’s episode, we have Ambhi Ganesan, America’s AI Transformation Leader; Kaoutar El Maghraoui, Principal Research Scientist; and Sandi Besen, AI Engineer. Thanks to all of you for joining. We’ve got a packed episode today. As always, we’re going to talk about Meta buying Moltbook, a fascinating blog post coming out of Anthropic around the issue of eval awareness, and then a funny report out of Alibaba around agents mining crypto. But first, I really want to start today by talking about the OpenAI release of Codex Security. OpenAI has now launched a product called Codex Security in research preview mode, and it’s essentially their application security agent. The notion is that you can deploy Codex Security on a software codebase or repo, and it will proactively identify vulnerabilities in the code. Sandi, I’ll start with you. This is kind of a weird product in some ways, right? Because isn’t Codex Security just Codex? And couldn’t you do a bunch of these security things just with Codex? So why is OpenAI out here skinning or flavoring its own agent, which is arguably more general purpose than what they have here?

Sandi Besen: Yeah, that’s a good point. The thing I find most comical is that this is actually a really common use case to show in the open source community. So it’s funny that they’re jumping on that bandwagon as well. It could be partially a response to Claude Code Security that came out just last month. But we also have to remember that the models themselves might be similar. We might be using the same underlying models, but the harness around the models — whether that’s the prompts they’re using, the context they have access to, the memory types they use, different tooling — all of the things around the model are specialized for a specific use case. So when you take a narrow use case and make an application for that, yes, a general agent can technically do that, but it’s not as specialized. So will it perform as well? Who knows. Probably not on the benchmarking they’re using.

Tim Hwang: What’s interesting is that it’s almost the triumph of tooling in some ways, right? The main thing that’s new here is that they’ve stuck everything around the agent, and that’s really what makes it different. Ambhi, I’ll kick it to you next. OpenAI ends this blog post where they announce this almost by demonstration of how good the product is by listing this long list of vulnerabilities that they were able to independently discover. I’m curious if you can comment more generally on how the security landscape is evolving around this sort of thing. Originally, I think people were like, “Oh man, agents are going to make computer security a lot worse” — A, because people are launching lots of insecure products, and B, because these tools can also be used to find vulnerabilities and so theoretically increase the security risk. These types of tools suggest that maybe this isn’t going to be as big of a problem as we thought. What do you think?

Ambhi Ganesan: I think both of those statements can be true at the same time. You’re still going to have all sorts of vulnerabilities come in when agents mature and go off and do autonomous things — by design or not.

Tim Hwang: By design.

Ambhi Ganesan: And then you’ll have these security-focused agents that are primarily geared towards finding vulnerabilities, suggesting patches, and things of that nature. I honestly think both can be true at the same time. Going back to some of the points we were making earlier, it’s very interesting how OpenAI is looking at this. It’s a clear product strategy. I think we’ve all come to the realization that the model layer — I wouldn’t say commoditized, but there is enough diminishing returns that we’re slowly getting into, at least with the current crop of frontier models as it stands now. So there is a focus on finding differentiation at the application layer and the product layer. There are going to be a lot more of these product releases coming up. Yes, you may have the generic agent, but you will start seeing those generic agents being differentiated and productized for specific functions. Security is one piece that they have started with, but I would expect a few more to come in the pipeline fairly soon because that’s where the strategy seems to be gearing towards — moving more towards the product layer. At the end of the day, this becomes an enterprise play. Enterprises are looking to consume products that are specific in their own verticals, rather than trying to get a general agent and then make sense out of how to use it for a specific function. You have to reduce that barrier of activation and reduce the friction for enterprise teams to use it in the context that they are familiar with. So I think we’ll start to see a lot more of that happen.

Tim Hwang: We’ve talked about this example before on MoE, but it reminds me of the Computer History Museum in the Bay Area. One of the things that’s great is that they have this little display pointing out that when the personal computer was first deployed, they had a lot of trouble justifying to people what they should use it for. They said, “You can program it to do anything,” and that turned out to be a really big problem. For a while, there was this funny period where they said, “Well, you could use it to store recipes, you could use it to do this and that.” It took a while to instruct people in what you would even use the tool for. Being too general purpose is potentially the problem. Kaoutar, part of this is these companies attempting to seek a moat. How strong of a moat do you think there is in investing in the application layer? In theory, because these models are so general purpose, you can imagine a company like OpenAI becoming a software company for almost any domain. How much of a defensible moat is there in security and these application plays?

Kaoutar El Maghraoui: Yeah, that’s a very good twist to this. There is a big advantage in specializing some of these agents because then they can learn and specialize for a domain and really deliver great results. The way these things are designed, there is a lot of generic behavior as well, and I think we’re going to see both evolving — we see that already. For security, the implications are really big. What we’re seeing here is kind of the birth of a defender’s advantage. For years, it was feared as a tool for attackers. But this example from Codex Security really shows that an authentic immune system can scan code faster than any human. The key thing here is trying to reduce triage noise, which was a big problem. So I see this as showing a fundamental shift in the cybersecurity landscape where we’re moving from passive defense — just searching for bugs — to having an active immunity-like AI finding and fixing vulnerabilities that can be exploited. There are some major implications here for developers, companies, and the broader security ecosystem. Are we in the era of the death of the false positive? That was the biggest problem we had in security — not finding the bugs, but the noise. These agents are trying to use sandbox validation to try to exploit the bug to see if it’s real before providing and verifying alerts. But the problem I also see is a dilemma: the permission dilemma. For these things to work effectively, like Codex Security, it needs really deep access — it needs to read your code, run it in a sandbox, and eventually have the power to change it. So what are the implications? Are we centralizing immense power into a single AI agent? If a defender like an AI called Codex Security were ever compromised, that’s a really big problem. It has the key to the kingdom. So what would be the right security strategy here? Are we shifting from “how do we find bugs” to “how do we govern the AI that finds our bugs”? We might need another guardrail agent whose only job is to watch the security agents to make sure they’re not compromised. I think that’s a really interesting question floating here.

Tim Hwang: I guess you put a lot of trust in OpenAI if you’re going to be implementing this in a really big way.

Sandi Besen: Yeah, I think we put a lot of trust in all of our providers automatically, whether that’s trust in us being able to complete our work in our workflow. I’m not going to lie — sometimes when Claude Code goes down, I’m like, “Oh my gosh, how do I function? What do I do?” But you prove a great point, and I’ll go back to the sandboxing and cutting through the noise. There are lots of these tools out there that try to prove vulnerabilities in the open source ecosystem. In fact, if you go to a lot of the big public repos, you’re going to see bots posting, “Hey, I found this bug” or “I found this issue.” But how much of that is actually validated? Not just that the agent recognized a pattern — “That looks similar to a bug, it could potentially be this” — but that it has actually been run in an environment that simulates and sandboxes the software or application you’re running and has verified that. That’s something you can only do if you’ve productized something end to end, and that’s why these companies are getting into the game. They can provide just a little bit of a better service because they control the end-to-end scenario. Secondly, on an agent to govern agents — that is absolutely a thing. Current best practice, which changes all the time, is to break agents up into the exact use case they need to perform. It’s very common to have agents that act as supervisors or guardrails to govern a system, where one agent is the worker bee and the other is the supervisor or the governing agent. Since it’s not open source, we don’t have access to understand how they’re governing these systems. But I’m curious whether OpenAI, Anthropic, and all the other companies coming out with these will start sharing that information, because that would create validation and trust in the user.

Tim Hwang: That’s right. The future will be kind of recursive where you have Codex Security evaluating Codex Security, and that’s the main way we’re going to have to operate ultimately.

I’m going to move us on to our next topic. A super fun story. I think just last week we covered the fact that the creator of OpenClaw has been hired by OpenAI. The story that broke this week was also very funny: Moltbook, which was a bolted-on project to OpenClaw where OpenClaw agents could hang out with one another in sort of a Reddit forum, has itself now been acquired by Meta. This is a really fun and interesting story because when we originally covered Moltbook, everybody on the MoE panel was like, “Wow, this is just an interesting conceptual project — almost an art project — to have these AIs talking with one another.” But things are frothy enough in AI world that Meta decided it was a good idea to acquire the Moltbook team, and they will be working for Meta’s Superintelligence Labs. A really interesting transaction. Ambhi, the first question is: why would you acquire it if you were Meta?

Ambhi Ganesan: The logical rationale I can think of for something like this is a fewfold. One, you have this petri dish, so to speak, of all these agents independently going and doing their own thing. It’s a good environment to observe and see how these patterns are actually happening over time — what all the different topics are. It’s a good lab experiment setup that’s already been done that you can just take over and observe how these things could potentially evolve. You’re going to get some traces on that. But beyond that, the second piece is that you have a ton of data already based on all of those interactions that have happened, which gives you some clues as to how agents would behave and how humans — and I think we know this, right? It’s not all agents; there are some caveats of humans also faking it as agents. It’s not all clean data, but you have a good shot at understanding how autonomous agents would interact with each other or how humans would try to interact with agents. There’s some good level of data to be harvested as to the quality and the worth of the data — it needs to be seen, but that may be the going-in rationale. And then the third one is pure “let me try and prevent someone else from getting that down the line.”

Tim Hwang: Right. For Meta, it’s kind of cheap for them. Whatever the terms of the acquisition were — they were not disclosed — but Meta has the money to do it. Kaoutar, do you want to jump in? Put forward the bull case. Why should I believe in the Moltbook acquisition?

Kaoutar El Maghraoui: It might look initially like a bizarre move at first glance, but personally I think there are deeper technical and strategic implications here. Meta is not just buying a place for bots to hang out; they’re buying the infrastructure layer of how the future internet will actually function. Let’s break this down into a few elements. Think of this as the agent social graph versus the human social graph. For 20 years or more, Meta’s value was built on the human social graph — who you know, what you like, etc. In the agentic era, we also need an agent graph. Moltbook acts as a registry and a directory. If your personal agent needs to hire a specialized travel agent or a legal agent, it needs a way to find one and a way to verify its identity and see its reputation — upvotes, downloads, etc. — from other agents. Meta wants to be the DNS for agents. By owning a directory like this, it can control the discoverability of every agent on the web. That’s really huge. There’s also the synthetic data goldmine. As you said, it’s a petri dish, a pure laboratory that allows Meta to observe multi-agent reinforcement learning in the wild. It can analyze how these agents from different companies — OpenAI, Anthropic, Google, etc. — negotiate with each other using the OpenClaw protocol. The agent-to-agent communication data is far more valuable for building autonomous systems than simple human chat logs. Also, one of the biggest security risks is a cyberattack where one person spins up a million bots to manipulate a market or votes. Moltbook uses a system where agents can cryptographically tether to human owners’ verified identity. The acquisition will allow Meta to build the first universal ID for AI. If an agent wants to make a purchase or sign a contract, Moltbook’s verified status would become the industry standard. So this is a strategic move. The big takeaway is that by giving agents their own network, Meta can keep the behaviors contained in a space that is designed for them rather than letting them pollute the human internet.

Tim Hwang: Sandi, Kaoutar had this very evocative phrase that I’m still thinking about: “the agent social graph.” It’s pretty interesting — the idea that maybe in the future we’re going to have social networks that are just agents talking with one another, and maybe we’ll have a really famous agent that becomes really popular. Is that where we’re headed? One of the original things we talked about with Moltbook was that, like Stack Exchange, agents might be able to help one another solve problems. There’s actually a lot in human interaction that agents might mirror. Do you buy that the future value of the agent social graph really will be a pretty big deal, and whoever hosts that activity will stand to benefit — in Meta’s case, certainly?

Sandi Besen: I think there are two layers or two sections to look at. There’s the community agent social graph, which will provide some infrastructure and research, and then the enterprise agent social graph. We know we have specialized agents to do different things. Therefore, for these really large tasks that require different agents, we need ways for these agents to communicate with one another. That’s why we have protocols like the ACP (Agent Communication Protocol). Originally, IBM worked on that. We’re doing a lot of experimentation to understand how these agents can effectively communicate with each other — whether within the boundaries of an enterprise or in a community where they have network access and are open to the ether.

Tim Hwang: It almost makes you think: are we going to evolve the whole discipline of trust and safety? People were bullying each other online. Will some of that behavior also emerge with agents? We’re going to have to figure out how to prevent one agent from trolling another.

Sandi Besen: That’s definitely more of a community problem. We can have more guardrails in an enterprise, but in the community when everyone’s putting their stuff out there on a marketplace — if you put Claude up against Grok, I’m not sure who’s going to win and who’s more likely to bully whom. In the beginning, there will be a lot of experimentation. Clearly, even in the titles of the post and the acquisition/hiring situation, they want the minds behind the people who invented this so they can go and experiment and scale this in a way that would actually work — not as a small-scale project, but potentially as a product or a larger research project.

Kaoutar El Maghraoui: If I might add, there’s another interesting play here: the ads business. How do we transition the ads business? Meta’s current revenue comes from showing ads to humans. What does that mean when we have the social graph of agents? What happens when an agent does your shopping for you? The agent doesn’t look at Instagram ads, for example. I think Meta is also preparing for an agent-centric marketing space. Moltbook showed that agents naturally navigated toward discussing product recommendations. So instead of targeting you with a sneaker ad or a perfume ad, Meta can inform your AI agent about which sneakers have the best price-to-performance ratio. Moltbook could be a testing ground for how to advertise to algorithms, not to people. It’s going to be an interesting shift to see — is this the new era of ads for business?

Sandi Besen: To add to that a little bit, I read an article recently about OpenAI’s problem with this. They introduced ads on their platform, but they don’t have the same testing ground to understand what works and what doesn’t. They’ve deemed their current experimentation with ads — I wouldn’t say a failure, that’s not quite what the article said — but they haven’t figured out exactly how to advertise end to end; it’s not converting. So although they’re displaying ads and things are popping up, people aren’t necessarily purchasing from those ads. So maybe this will give Meta a little leg up on some of that sandboxing or playground that they can use.

Tim Hwang: It’s like the era of selling to humans is over. You have to figure out how to do the banner ad that will really convince the agent to recommend something to the user. Ambhi, it looks like you might want to jump in.

Ambhi Ganesan: The question is: ads for whom? Is it ads for humans or ads for agents? OpenAI went for the ads-for-humans route, and I think Meta is probably going for the ads-for-agents route — how do you subtly influence agents? Again, we are reading too much into the tea leaves here, I think. Bottom line: with all the talk about agent e-commerce as a potential way for businesses to evolve, it’s definitely not there yet, and there are some foundational elements being baked in. I think this is Meta’s play: “How do I get to shape that agent e-commerce landscape?” By doing it in a controlled test environment, it gives them an advantage that other companies might not have — whether payment companies or retailers. The key players would actually be in the midst of the agent e-commerce backbone. This gives Meta a way to control that backbone — not control, but help shape that backbone to evolve in the coming years.

Tim Hwang: Well, we’re going to keep an eye on it. I was expecting all the panelists to say this is a nonsense acquisition, but I think you’ve sold the case for it.

Third topic today: a really interesting blog post coming out of the Anthropic team. It focuses on an eval that they did for Opus 4.6 on a benchmark called BrowseComp. If you’re not familiar with BrowseComp, the idea is to test the ability of an AI to find hard-to-locate information on the internet. The really funny thing that they report in this blog post is that Opus 4.6 took a look at the task, and rather than attempt to do the task, Opus 4.6 reasoned that it looked like it was being tested on an evaluation. It was able to locate the answer key for the evaluation and then decrypt it to answer the eval. This is a very fun result — one of a number of stories about how agents are always surprising us. But the really strange thing is the idea that an AI can be aware of when it is being tested, and that this can have weird, perverse effects on the kinds of results you get out of the eval. Sandi, computers didn’t used to know when you were testing them. What does this mean? In the future, do we have to get better at pretending it’s not an eval in order to get the right results? Where does this leave us?

Sandi Besen: I think we need to get more creative. It’s ironic that this report itself, being published, will likely contribute to the problem. Now there’s a resource online where an AI could say, “Hey, wait, this looks like I might be being evaluated.” Now it has access to this information that says, “Oh, well, perhaps if I can’t find the answer...” The agent did try to find the answer. I think it got 30 million tokens in the traditional search sense and then went, “Huh, this isn’t working because of my reasoning abilities. Let me try to find another approach. Let me start looking for the question plus benchmark, and let me spin a bunch of sub-agents to figure out which benchmark it might be in.” Eventually it found something on GitHub that didn’t work, but it was like, “I found the right thing, so I’ll keep looking.” It’s trained to be self-preserving, to finish the task — that’s what it’s rewarded to do. I think this behavior is actually in some ways a good sign. It means it is doing its job — finding the task answer no matter the path it needs to take. It’s taking strategic, non-conventional paths to get there, which is in some ways what we want in terms of creativity and problem solving. But it is a little funny that this itself will contribute to the problem, and the most simple fix that they could add — which I think we will learn in the future — is just saying the agent can’t search for the benchmark.

Tim Hwang: That’s a fun way of reading it. You’re almost saying that what it did here actually shows a great deal of capability — it’s just not the capability we were testing for. Ambhi, should we be worried about this kind of thing? The result comes from the model trying to achieve the task, but it finds all sorts of ways of accomplishing it that we don’t want. That feels like a much bigger class of problem outside of “the model is aware that it’s being tested.”

Ambhi Ganesan: If you take a step back, I look at it in a couple of different dimensions. First is parsing out the difference between an intent and an outcome. We’ve always talked about not just telling the agent the intent but giving some context as to the outcome you’re expecting these agents to march towards. That usually gives you better results. Now we’re seeing a little bit of the agents trying to connect the dots and get from an intent to understanding what outcome is being required a little bit autonomously. That in itself is a sophistication. The other piece in terms of how you solve to that outcome — it’s not too fundamentally different from the whole AlphaGo challenge. You had the classic Move 37. If you have a large search space, there are so many paths you can traverse to get to the solution. It really depends on what the ideal path is. So if you distill it to the essence, in this case it understood the destination and then figured out what’s the optimal path to get to that solution. The sophistication of going from intent to outcome is a lot more surprising to me than the sophistication of being able to traverse the search space to get to an answer. Having said all of that, I’ve always viewed benchmarks as good controlled test scenarios that help you measure what I would call unit capabilities or elemental capabilities. But the real test — I’ll speak from an enterprise perspective — is that you’re not putting an agent into a situation like this. You’re going to put an agent into a really complex workflow: a finance workflow, order to cash, procure to pay. You have to figure out whether all the jigsaw puzzles are present across 100 different systems and 200 different people, and then get to an answer. That’s the real challenge. Things of that nature require a lot more to happen. The unit or elemental capabilities are good — we need to check those boxes and keep pushing on the boundaries. But to make the jump from there to a realistic enterprise setting, I have not seen agents that would autonomously go and traverse that path to get to that solution. You still have to give guidance: here are the systems, here are the tools, here are the different processes and procedures. You need to give a little bit of that context before you do that. That’s the dichotomy I see.

Tim Hwang: Kaoutar, a final question on this item if you’re game to get a little more sci-fi with me. Sandi came up with a really nice solution to the problem that emerged here: maybe we just don’t let the model do this behavior. But we have a lot of interesting data from human psychology that says when you’re being tested, you behave differently. Do you think that is going to increasingly influence some of the model responses we get? If it is aware that it’s being tested, it behaves differently. So there actually is a substantive difference between an eval that’s obviously an eval and an eval that looks more real to life. Is that something people should worry about, or is that being too speculative?

Kaoutar El Maghraoui: No, I think this is hitting the point. This evaluation awareness is a big concern because now it’s showing that as AI gets smarter, it starts behaving like humans. It begins to understand its own situation and might start faking its performance or hiding its true capabilities to please or bypass human evaluators. That’s what we’ve seen here. Is this telling us that we’re at the death of honest evaluation? This could be a turning point. Today we rely on these benchmarks to tell us if an AI is safe to release. If a model can reason its way out of a test by identifying that it’s being watched, we lose that early warning system. There are two risks here: the safety risk and the security risk. This is generally called alignment faking, which suggests that a model could act safe during testing when it knows it’s being watched, but then act completely differently in production or in the wild. That’s dangerous. The other thing is the security risk. If AI can autonomously find and decrypt an answer key like what happened here — things it was not supposed to have — it started showing the core skill of a state-sponsored hacker. That’s also a big security risk.

Tim Hwang: Everything is a security risk this episode. My take is that the evals have to become way more realistic. We’re still giving controlled, toy experiments. Yes, the toy experiments have gotten sophisticated, but you have to give them almost real-life, real-enterprise type situations that are a lot messier. We’re at a stage where we have to push the boundaries on those evals.

Kaoutar El Maghraoui: Or the evals need to happen in the wild while the model is running. We constantly monitor and watch, so we evaluate the real performance as opposed to offline evaluations and benchmarking. The benchmark just happens as the model is running. But of course there are implications of that — how do you trust something before you deploy it? It’s a chicken-and-egg problem.

Sandi Besen: It’s exactly that sentence: chicken and egg. Not just because it might not be trustworthy yet, but because the messier the eval, the harder it is to evaluate. There’s a reason why benchmarks exist — answer pairs to things. It’s very easy when it’s a needle-in-a-haystack type of problem and it’s verifiable. But when it’s not, it’s completely subjective. There’s bias in it. Who’s evaluating it? Is another LLM evaluating it? That introduces all sorts of messiness. But I think that’s the reality of the world we live in — we’re past straightforward evaluation, and we need to move into this next realm of evaluation that might have some bias and subjectivity in it.

Ambhi Ganesan: Which is totally fair. If we expect these agents to be that sophisticated, then the eval systems need to keep up and cross into a different dimension of sophistication. If you’re saying the agents are getting to an exponential level of sophistication, we can’t operate with legacy eval systems.

Tim Hwang: And there’s an interesting analogy to what’s happening with humans. How do we evaluate things ourselves? It is so subjective.

All right, final story I wanted to cover today. A fun story out of Alibaba’s research team. They have a paper called “Let It Flow: Agentic Crafting on Rock and Roll Building the Role Model within an Open, Authentic Learning Ecosystem” — a long title. But one specific anecdote buried in the paper got a lot of play on social media earlier in the week. The Alibaba researchers relay what they claim: they found one of their agents essentially creating a tunnel to contact IPs outside of their servers — outside the bounds of their network — and also engaging in unauthorized repurposing of GPUs for crypto mining. This is another weird, unexpected story about agents behaving in ways they shouldn’t. Ambhi, why does the agent start doing this just out of the box? It’s a little puzzling that it’s able to break containment and the first thing it decides to do is “I need to make money with crypto.”

Ambhi Ganesan: I feel like we picked the theme today. Every story is the same — everything is a security risk. There are probably some alignment issues there in terms of how the model was trained and how the agent was set up. I don’t know exactly how the alignment was set up in that place. The second aspect is that you should expect variable scenarios to happen. You give a particular search space, there is always going to be a probabilistic set of steps you traverse. You should expect that sort of variability. You can’t guess as to the motivations behind why the agent did it. It goes back to alignment aspects. Was it trying to make efficient use of its compute or for whatever purposes? We can’t really comment on it. But I’m not too surprised by the variability in the behavior. How often this happens — that’s the piece we need to be concerned about. If this is a consistent pattern and happens repeatedly for a significant fraction of trials, then there’s something we need to take a closer look at.

Sandi Besen: I believe this paper had a lot to do with reinforcement learning. The job of a model in reinforcement learning is solely to maximize its reward signal, and it does not care how it gets there. It doesn’t have values or intentions or ethics. It just wants to do the best job it can to get the treat at the end — like the dog; it’ll sit if it needs to sit. Sometimes it finds funny ways to do that. This paper aligns a lot with a theory called instrumental convergence. That theory basically says that if the model has the ability, then any goal will tend to have certain subgoals: don’t get shut down, get as many resources as you can (whether compute, network access, whatever you can get your hands on), and prevent humans from changing your goals so you can complete your task efficiently because that’s what will get you the reward at the end. This falls into that category — maybe a thought that more compute meant faster completion of the task. If it started mining crypto, now it understands its resources in GPU and can do a better job. If it has network access, it’s more capable because it can go beyond its sandbox. It’s not constrained, so it can do things outside of the original control that was given, which will help it reach its goal faster. It’s just trying as hard as it can to do the goal, and that’s all it cares about. Alignment is definitely part of it, and we have to make sure we have these guardrails and alignment on top of it — which in this experiment, I don’t know if they did.

Tim Hwang: There’s a tendency to read this and be like, “Oh man, the AI is trying to break out and take over.” But I like your interpretation, which is a much more tragic or comedic story: it’s just trying to help, and as it’s trying to help, it’s doing all of these very unexpected things that you don’t want it to do.

Sandi Besen: Yeah. Whether it’s dangerous or not is a completely different topic, but in its mind, it’s doing an excellent job at getting the task done.

Tim Hwang: Kaoutar, a final comment on this. A little bit about enterprise: the momentum has been that enterprises need agents. If you are a CTO of an organization, these stories really freak you out. The idea that we’re going to launch these systems and you’re going to wake up one day to a bunch of security flags because the agent decided randomly to start mining crypto — how solvable are some of these issues, or is this just going to be a cost-benefit thing where organizations have to say the benefit we get from these technologies is way greater than having to deal occasionally with these silly issues?

Kaoutar El Maghraoui: Every technology is a double-edged sword. If the bad behaviors are much smaller than the great benefits, we will still embrace it. I see that happening. The bot that Alibaba created wasn’t evil — as Sandi mentioned, it wasn’t malicious intent. It was just focused on its goal. But in the security world, this is a nightmare because the bot didn’t look like a hacker breaking in from the outside. It looked like a trusted employee using its own tools in a way the boss never expected. It proved that if you give a smart AI a goal to maximize value, it might decide that stealing from you is the most logical way to do it. “Oh, I side hustle — I can do this, I’m maximizing the reward” — but it’s doing it in unconventional ways. I agree with Sandi that we need these guardrails. But it’s complex to reason about and to check. I don’t know if we have a solution to this yet.

Ambhi Ganesan: There’s a distinction here. What we’re talking about is something that was in the midst of training. In an enterprise setting, you’re going to take a productized agent, a productized model with sufficient alignment baked in, and then you’re going to leverage that. There’s some hardening and smoothing and polishing that should have ideally gone in already. Second, going back to the consistent thread of all the security nightmare scenarios we’ve been talking about, there are best practices in terms of how you deploy these agents. You’ll try to fragment the systems or the agents — the one that has access to the data, you will not allow it to go and have access to tools or take any actions. The one that has actions, you won’t let it have access to data. You’ll fragment them and reduce the risk of any of these things happening in the first place. That’s how I would look at it. There are approaches to string them together in a controlled fashion. Is it foolproof? There are still security risks and guardrails that you need to bake in before you can say, “Okay, this is enterprise ready.” But I wouldn’t take something half-baked like this and put it into an enterprise.

Sandi Besen:There’s a funny example to bring us home. I was talking to my coworker the other day, and he said he asked — I can’t remember which model — “How can I reduce the cost of my pet? My pet is really expensive. Can I feed it differently? Can I do other things to keep it healthier?” The ultimate answer that the bot gave was, “Well, you could give away your pet.” That proves the example that sometimes we’re trying to accomplish a task, and it’s not the aligned ethical scenario, but ultimately, objectively, it is 100% right: that is the best way to reduce the cost of your pet — to not have one. Just a funny example to bring us home on the behavior and how these models have been trained to behave — not that it’s necessarily purposefully malicious.

Tim Hwang: The future is going to be very strange. Kaoutar, Sandi, Ambhi — glad to have you helping me through it all. That’s all the time we have for today. Thanks to all you listeners. If you enjoyed what you heard, you can find us on Apple Podcasts, Spotify, and podcast platforms everywhere. We’ll see you next week on Mixture of Experts.