Tim Hwang: 5 years from now, is NVIDIA still the biggest name in AI hardware? Aaron Baughman is an IBM Fellow and Master Inventor. Welcome back to the show, Aaron. What do you think?

Aaron Baughman: So, I do think that they’re going to be in the top five. The field’s going to be much more fragmented with different chip architectures, but I’m looking forward to see what types of neuromorphic chips are going to come out.

Tim Hwang: Vagner Figueredo de Santana is a Staff Research Scientist and Master Inventor on the Responsible Tech Team. Vagner, welcome back. Your predictions, please.

Vagner Figueredo de Santana: I I second Aaron. I think that NVIDIA will be still at the top, but with different architectures and maybe cooler ideas on new architectures for chips.

Tim Hwang: Yeah, I hope so. Shobhit Varshney, Senior Partner Consulting on AI for US, Canada, and Latin America. Shobhit, tell us what you think.

Shobhit Varshney: I think NVIDIA, in terms of AI systems—beyond just the chip, there’s a lot that goes around it—I think it’ll be a force to be reckoned with for the next 5 years, and I would say it should be in the top two or three.

Tim Hwang: All that and more on today’s “Mixture of Experts.” I’m Tim Hwang, and welcome to “Mixture of Experts.” Each week, we bring you the analysis, hot takes, and banter that you need to keep up with the ever-hectic world of artificial intelligence. We’ve got another packed schedule for today’s episode. We’re going to talk about a new jailbreak that’s hitting the scene, people you can’t talk about on ChatGPT, but first, we wanted to take as our top story the AWS re:Invent conference, which is happening this week.

For those of you who may be less familiar, this is the annual conference for Amazon’s AWS, and there’s been a host of big announcements coming out of Amazon this week, not least of which is that they are announcing that their new generation of their AI chip, what they call Trainium (Trainium 3), is going to be launching very, very soon. There’s a lot to get into, but I think, Shobhit, I wanted to kind of throw it to you first because you are actually at the conference. I just talked a little bit about Trainium, but curious if there’s, you know, what are the trends that you’re seeing? What are other big announcements our listeners should know about?

Shobhit Varshney: From my vantage point, AWS re:Invent was the AI event of the year. It’s a pretty bold statement—I mean, there’s been a lot of big AI events this year—but in terms of what they’re trying to do to change the industry and absolutely dominate in the AI space, it’s just absolutely incredible. If you look at all the different stacks or the layers of the stack, at the compute level, they are doing a lot in terms of the chips, and so are other competitors as well, but they are quite ahead. When you have somebody like Anthropic and you’re building a supercomputer for them that’s 5x more powerful than what Anthropic has today, that is making a bold statement by AWS as a company. Amazon has to do a lot of AI, and they have a really good history of doing that for 10-15 years, so they’re building on top of that. The compute is going to be very critical for them for a nice ROI.

Second, on top of all the storage, the amount of options you get as a developer is just incredible. It’s like it’s a dream for us to go build for our clients. It’s one of our largest partners globally, AWS, so we do a lot of work in building a lot of systems with all the different options with them. Then is the AI layer for all the models. I think across the board, they’ve been very clear on choices. If there’s one word that summarizes AWS today, it’s “ecosystem.” They’re trying to do their best to make sure you have the best-in-class models available, the best-class apps and things of that nature, but then, “Oh, by the way, we also have our own version that is delivering higher ROI, we’re matching or exceeding.” You have a massive announcement with NVIDIA, “by the way we have our own chips,” we have great collaboration investment in Anthropic and all these other models, “by the way we also have our own.” So I think the choice is why people will come to AWS, and re:Invent, the kind of announcements they’ve made... We spent the last three days hands-on working with the product leads. Being such a big partner of AWS, we get some dedicated talent from AWS to give us previews and give hands-on experiences on how this actually is working. They’ve done an incredibly good job. I’m so, so excited about the next few months going and doing this with our clients.

Tim Hwang: Yeah. Aaron, so do you buy it? I mean, should everybody else in the AI space be scared? Traditionally, most of the attention has been on OpenAI and Anthropic and the people who were doing the models. I guess Shobhit’s kind of making the claim here that maybe Amazon’s gonna kind of take the cake in the end. I don’t know if you buy that argument.

Aaron Baughman: Yeah, I mean, the way I look at it is that it’s sort of a quasi-contest to beat out NVIDIA, right? They’re trying to build their own chips to compete. However, if you looked at the announcement, AWS is still hedging by partnering with NVIDIA on P6, right? So even though they’re building their own Trainium chips, they’re still going to be working with them on P6. And because of that, they’re looking to see which way the tide’s going to go. I also view this as AWS looking to reduce their dependence on third-party chips to enhance their performance on AI workloads on AWS. But to me, there’s still a lot of work that AWS has to do on the software staff, and they still have to prove out performance. If we think back, NVIDIA uses CUDA, right? That’s the most widely adopted platform for AI workloads in the world, and it’s supported by PyTorch and TensorFlow. Now, Trainium uses AWS’s Neuron SDK, right, which has a fraction of the market share and it’s not as proven as CUDA. So yes, I think that the chip hardware itself with Trainium is great, but AWS has work to do to build the consumer and developer trust to be really, really competitive. And that’s why I think AWS is hedging by still partnering with NVIDIA with P6.

Tim Hwang: Yeah, it feels like we’re in this really interesting world where all the big cloud providers are kind of working on their own chips right now, and they’re also all working with NVIDIA. I think it’s kind of everybody’s hedging a little bit. I guess, Vagner, maybe this goes to your prediction. You were saying kind of in the future maybe we’ll just have a more diversity of chips, and actually that will be the really good thing. Do you have a prediction on how the market’s going to divide? Like, will it be just NVIDIA for pre-training, or these types of chips for inference? I’m just kind of curious about how you think that market’s going to divide out.

Vagner Figueredo de Santana: I think that it will be based on the business case. Thinking back when I was involved with digital agriculture, when you see places that you have no connectivity, and then you start thinking, “Okay, if we have to have chips with inference running at the edge, then that will be the chip that will dominate the market.” If you have, let’s say, a machinery for agriculture using those chips, and if you have access to data, if you are let’s say next to the place where they gather information, then probably you have something to do the pre-training and training, and that will be the chip. And then you have enough power connectivity in certain places in a huge farm. So I think that it will be based on business case and how connectivity and data arrives at these specific hotspots.

Tim Hwang: One of the really interesting comments was from Ben Thompson, who writes a newsletter called Stratechery—it’s very, very good. One of the ways he sort of framed up a lot of the announcements was Amazon’s basically making the bet that models won’t matter so much in the future; that essentially it’ll be sort of like infrastructure that runs the day, and models will be widely commodified. So kind of what we thought was so special, which is, “Oh my God, you have to get the latest model that’s been released by OpenAI,” is just going to be less of a thing in the future. Do you guys buy that? Do you feel like what we’re seeing now is a movement of momentum towards the infrastructure providers versus the model creators?

Shobhit Varshney: Tim, I think we are making really good progress as a community across each one of those. We do need better intelligent models for reasoning and things of that nature; we’re making some incredible strides in that space that’ll continue. There’s a lot that happens before and after an LLM call. AWS has done an incredible job with their SageMaker stack, all the kind of automatic reasoning checks, the kind of things around how to pull structured data as part of my LLM calls, enhancements to all kinds of things that we as developers need when we go and build these for the clients. In the last two years, we’ve done a lot of custom work in the middleware to make these LLMs work well, and now you’re seeing each one of those providers catch up with giving you a full ecosystem end-to-end because they’re also learning from how enterprises are deploying these. So I think AWS, as a community, is further ahead than some of their peers in giving you the full spectrum end-to-end and making it super easy for startups to come and do this. I always have an enterprise mentality around these things. They are doing an incredible job on grounding, on making sure there’s right governance, a massive ecosystem; you can bring your own favorite eval to the framework and whatnot. They’re very, very well placed. Models are going to get better; there’s going to be a constant battle on that, but over time it becomes a commodity—that’s for sure.

Tim Hwang: So I guess, Shobhit, what’s your prediction for Amazon in the new year? You said this is the biggest wave of announcements. Where does it look like when we’re at re:Invent 2025?

Shobhit Varshney: They have clearly made a very massive dent into the AI community in the last two, three days. If you roll back two years to 2022 when we were here, they had just made a bunch of AI announcements, and then two days later we had ChatGPT come out, so they got caught off guard. Re:Invent last year was more around, “Yes, sure, I’m going to bring the big dogs on stage, I’ll have Anthropic on during the keynote, NVIDIA and whatnot,” right? So they said, “Yeah, we also have a lot of options.” This one, they just came in dominating. It took 24 months, but now they’re killing it. It’s like, “Guys, we got this.” So I think the overall messaging was, “Where else will you go to do secure, scalable, end-to-end, infinitely scalable, pluggable, and ecosystem-driven AI? Don’t outsource it to an API call; come build with us, and here’s how simple it is.”

Just a very subtle cultural thing: there is every key technical session I’ve gone to, they end with the same kind of enthusiasm. It always ends with, “Well, what will you build next?” right? They have this huge emphasis in every session; they excite you about the possibility, they give you a couple examples of what clients are already doing, and they say, “What are you going to build next?” and the product managers are going to hang around, they’ll show you how this works. Whenever I have a conversation at AWS with any of their folks, it typically starts with, “Hey, let me point you to a GitHub repo that does this for you, and then I’ll show you this in action.” But the first intention is, “Go build epic stuff with this. Go get started, and we’ll have dedicated people to come help you go build.” I think it’s a very different take. It’s black and white between AWS and Microsoft and others. You see a very different target audience; you see a lot more geeky conversations, hands-on tech, “this stuff scales” kind of conversation, and “you can build epic things with this.”

Tim Hwang: Yeah, yeah. Being able to build those epic things is really important because, I mean, the way I see it is that with these new Chain of Thought algorithms where these models can begin to self-learn, it’s almost like we build these foundational models with pre-training, and then you have a choice: do you want to fine-tune it, do you want to do some instruct tuning, right? But then I’ve noticed that now there’s this, instead of just really fast inference, there’s a thinking phase now, right? And this thinking phase can go on for minutes, even hours, right? And because this thinking phase is happening with all these emergent behaviors and skills, you need this scalable, secure, robust architecture that I think was announced at this conference. So it’s really exciting to be a part of this and to watch what’s happening.

Tim Hwang: That’s great. Well, definitely one to keep an eye on. There’s going to be a lot more action in the space, and yeah, it is really exciting. I mean, I think that if you had asked me 24 months ago, I would have been like, “Amazon’s way behind; they’re never going to catch up.” But you can never really count them out because they’re Amazon.

Shobhit Varshney: So Tim, just one last thing I would add to this: being the world’s cloud provider, they have the largest market share; they see a lot of workloads. So they have an unfair advantage that others do not have. They can see how people are actually leveraging these tools, what are they building, how they’re contributing back to the community, so they can go test out and be the second movers and just dominate after that, right? Because people build stuff; there’s a lot of small startups that have built all these niche things that got announced as features within AWS, right? So you have this unfair advantage that AWS has because they see how people are actually using it in enterprises. Bringing in a large, trusted partner like one of the most trusted brands, Apple, making a statement that Apple for the last decade has been building on AWS—that gets all the financial services clients. One of them was sitting right next to me got very excited, saying that, “Oh, this is a really clear statement that you’re doing trusted computing if you have Apple on stage talking about it.” The massive financial services, like JPMorgan Chases of the world, they’re doing some incredible AI workloads on this. So they’ve made a very, very bold statement, and they’re going off of the mainframe business with this as well. They’re saying, “Traditionally, there were a lot of transactional systems that were high compute needs, and you could not really synchronize them on the cloud.” If you look at a series of announcements, they’re doing a really good game plan on how to attack workloads that haven’t moved to private clouds or secure clouds yet.

Tim Hwang: Yeah, I think that’s right. I remember when it came out a few years ago that Netflix was running most of its infrastructure on AWS and being like, the amount of video they’re moving through that system is just... yeah, crazy to imagine. So yeah, I think it’s a really good point.

Well, we’re going to move on to our next topic of the day. There was a great blog post that came out from a security team called “Gnostic” (spelled with a ‘K’) on a new class of LLM attacks that they call “Flow Breaking.” What was kind of interesting is that they are proposing this as a new sort of third kind of attack we’re seeing in the space, with the other two being prompt injection and jailbreaking. Specifically, what flow breaking focuses on is the fact that many of these AI applications are built as ensembles of models that are doing lots of different things. In many cases, there are separate models, separate filters that are implemented to block unsafe generations on the part of the model. So, the model might go to advise you to do something dangerous, and there’s another system that says, “Oh, that’s not actually what we should do,” pauses the generation, and then regenerates it in a more safe way. What flow breaking focuses on is how to use that as a way of getting unsafe material out of the model because there is this kind of gap between the model itself and the safety measures. Vagner, I know you were the one who flagged this for us. If you want to talk a little bit about how this changes our thinking about security on models, and does it make things more complicated for us as we think about how to secure these models against manipulation?

Vagner Figueredo de Santana: I think that it is interesting because it tells us how people are building architectures of models and how they are placing the guardrails. If we look back on software engineering, it’s another way of exploring race conditions, and also we can think about asynchronous requests and how all of this is happening. With this new attack, they’re basically exploring this interval, this millisecond interval between generating and the guardrail taking over, and showing that if the content is sent, then this can be harmful. I think that is the key point. I tried to replicate one of the two attacks that the team showed, and I was able to replicate one of them. The other one was not working anymore, at least on ChatGPT-4 that I tried, but the other one worked really fast. But it’s interesting in the sense that the data is sent. So I think that is important for us to rethink the way that we structure and place these guardrails in our architectures, and also even organizing the requests. If there are two asynchronous requests, then probably the data will be sent to the user. I think that is the key aspect, and the content may be harmful, and someone may use that. I think that is the key aspect they showed, and even that for the human, for the common user, this will not show because it’s so fast that it’s hard to see the content, but the content is there. I think that is the key point.

Tim Hwang: Yeah, and I think it’s kind of really fun, and Vagner, like you’re saying, I think because it reveals so much about how these systems are architected. Aaron, if I can kick a question over to you, it’s like, why are these companies streaming the unsafe tokens at all? Doesn’t it make more sense to have an architecture where you do the safety checks before the tokens get to the user? Like, why is it that we have this kind of millisecond gap where you can kind of get this unsafe stuff out from the point of view of the user?

Aaron Baughman: Yeah, that’s a great question. I mean, it appears as though we’re always looking to be faster and faster and faster, right? And sometimes we concede to speed over responsibility. And because of that, we’re willing to take on extra risk. But you have to look at the opportunity cost. I think with this study that’s been done—it’s fairly well done with this flow breaking, where this type of—I call it “agentic social engineering”—where you’re basically trying to get agents to do something that they’re not supposed to do, or you’re changing the order of operations, you’re getting one agent to talk to another agent and skip over somebody else or something else where it shouldn’t. There needs to be almost like an auditing where you have these breadcrumbs of which agent has communicated with another agent so that they can’t skip another. The last point that I just wanted to make too was, with broadcast TV, whenever you’re watching a live game, it’s never real time; there’s always like a 5-second delay because there’s time for somebody to take out vulgarities or if someone runs onto a football field and does something a little odd, we can edit it out. So perhaps we need to start thinking about these types of systems where it’s never exactly real time, but there’s always this gap and delay so that we can ensure the safety of the audience before they see the content. But to recap that, I just think we need to be careful about conceding to speed over responsibility.

Shobhit Varshney: A question for Aaron: there’s a lot of techniques that we are now deploying with clients, like prompt caching. AWS doesn’t release their automatic reasoning that was working great for the last five years with ML models; now they’re bringing it to LLMs. Do you feel that having more of these checks and balances, like caching and things of that nature as well, do you think that we have a better opportunity today than we did six months back to solve and catch for these bad behaviors?

Aaron Baughman: Yeah, I mean, great ideas. I think we do because we certainly have more data to understand the problem and then some additional tools at our disposal in our toolbox to attack these types of flow-breaking pieces. Through caching, there’s a lot that you can do because you can sort of create these hashes to know where the data has already been, and then you can recycle that data such that it’s faster, and therefore you don’t have that extra milliseconds, like Vagner mentioned, to inject some sort of attack. So it just accelerates the speed at which these LLMs and agentic systems could respond.

Shobhit Varshney: So Aaron, I see when we’re doing these deployments for clients at scale in production, I feel that RAG, as a community, we have spent so much energy in improving RAG to be more enterprise-ready. I feel that agents today are where RAG was 18 months back. They used to make amazing, nice demos on stage; great startups can go work with it. But when you get to enterprise, RAG took 18 months to come up with like 21 different methods of doing RAG and whatnot. So I think that agents... I think there’s a little bit more security risk at this point than RAG. We’ve done a fairly decent job of access control and things of that nature, all kinds of hallucination detections. I’m really hoping that the community will push agents in better frameworks quite a bit in 2025.

Tim Hwang: Yeah, I think that’s the interesting question I was going to ask you, Shobhit. I think what Aaron’s proposing is that, particularly for agents right now, there’s a little bit of a speed and safety trade-off. I guess what you’re saying for RAG is there’s reason for optimism; at some point we might be able to both be fast and safe. Do you think that’s true? Like, do you think in agents right now there is this trade-off just because we don’t have all the checks and balances?

Shobhit Varshney: I think we’re still in the early days of agents. We are still trying to figure out the right architecture, the right set of checks and balances for bias output, things of that nature. We need better guardrails, better examples of how to call this API every time, and so on and so forth. So I think, just like we did with large language models, the smaller we move towards much smaller models over time—and hence smaller agents—you’ll start to build a set of agents that have been certified that they do this particular job incredibly well, just like we started to create a farm of RPA bots, and each one does one task really well. I believe that we’ll get to a marketplace space where we will have agents that have been pre-trained, and some agents do an incredible amount of work really, really well. I think we’ll get to a point, just like we do at Fiverr or if you’re going and getting some services online, you’ll see that people will start rating these agents well. You’ll have some of the leaderboards and stuff say, “Hey, if I want to go for a flight, if I want to find the cheapest flight from A to B, I’m going to use this agent; I’m going to pay 20 cents for it and get that done.” So I think we’ll get to a world inside of enterprises—curated, secure—as well as external commercial, where these agents will start to compete and do work really, really well, but they’ll do one small task really well. The meta-orchestration is where the enterprise will invest a lot. I think the security will start to get addressed better.

Tim Hwang: Yeah, that makes a lot of sense. Aaron, I want to go back to a moment ago; you used this very tantalizing phrase, which is “agentic social engineering.” That’s a really intriguing idea. If you go into that a little bit more, is that literally what you’re thinking about? Like, we have social engineering in security, which is, “I call and I convince the boss to give me the password to get into a system.” Do you think that actually is how we should think about agentic security, which is now we’re not even talking about humans anymore, but the manipulation of agents for not-so-good ends?

Aaron Baughman: Yeah, I mean, if I put a focus in on this particular flow breaking, it seems like the authors came up with four different types of vulnerabilities. There’s what, like, forbidden information streaming window, order of operations—if you could skip agents talking to others—and then software exploitation, so if a component gets too busy, then it becomes overwhelmed and it affects other components of the system. Those four different vulnerabilities, the way that I have a mental model, could go over towards this agentic social engineering, where you get these agentic pieces to do something that they maybe shouldn’t do, or to change the order of operations, to exploit software to inject data or a prompt into a streaming window. Yeah. And I think the Turing test 1.0 and 2.0, where we’re trying to get these LLMs to behave and act and rationalize like a human... it’s almost like we can social engineer them because they almost have their own mindset, almost like a theory of mind where they can begin—it’s not there, I mean, we have a ways to go—but where one LLM can maybe understand, “Hey, this other LLM has its own mindset, its own beliefs,” and you can try to train some of that through meta-prompting. But that’s the way that I’m beginning to start to think about some of these problems.

Tim Hwang: Yeah. And I also want to make sure we get Vagner in here. I mean, because Vagner, you kind of think a lot about model security and safety and how to ensure these models are responsibly deployed, and it feels like this is a really interesting interface. We have all these methods that we use for thinking about how we manipulate humans as a security problem, and maybe we can import that to these AI systems now.

Vagner Figueredo de Santana: I bet that there are people thinking about this right now, for the good and for the bad. The term that Aaron used also intrigued me, and I think is interesting. And I started to think that the flow breaking attack is only possible because the architecture that they created is based on human perception. If you think about agents, the first response would be caught by an agent, and that would be a problem already. If agents are consuming these endpoints the way that they are architected right now, these agents can consume that information. So I think that is the first thing that came to my mind. If there are architectures based on human perception, agents don’t have this limitation about this millisecond that the information appears and is deleted, so the agents will consume that information, and who knows what else.

Aaron Baughman: Yeah, I mean, it’s almost like you need a social contract—which LLMs can talk to which LLMs—almost like a communication graph so you can trace it. Almost creating social cliques, in a sense. It’s like your agent starts hanging out with the bad kids and goes wrong.

Shobhit Varshney: And that boils down to the kind of architecture we end up using with multi-agent frameworks. Depending on the problem we’re trying to solve for our enterprise clients, in certain clients we will go create a series of small agents, one after the other; it’s more sequential in nature. In certain clients, there’s a different framework for having a meta-agent at the top, and everybody else is kind of serving the tasks assigned to them, and everybody essentially sends their responses back. Then there are certain clients we’re working with where we create a network of agents that can all talk to each other, and in certain cases, if there’s a tie, we could do voting to do a tiebreaker. It just depends on the different kind of architectures. The social engineering part will get more and more interesting in the space. You may have, in our organizations, we have a legal team, we have an AI ethics committee and so on and so forth that we’ll escalate to, like, “Hey, you guys tell us how to do this well.” So I think we’ll start to replicate how human organizations work inside of these agentic multi-agent frameworks. I think there will be a “good cop, bad cop” kind of situation; there will be somebody who interprets the word of law and says, “Hey, my interpretation of this legal contract is X, everybody has to abide by that.”

Tim Hwang: Yeah, it’ll be fascinating because, I mean, famously Conway’s Law is you ship your organization chart, and it’s kind of like this will play out here. They’ll just be like the lawyer agent in the app that’s reviewing everything.

So I’m going to move us on to our next topic. There was a really interesting story that popped up earlier in the week. Some users on social media noticed that there are certain names—David Mayer being one of them, and Jonathan Zittrain being another, and people identified a few others—that are names that are systematically refused by OpenAI to talk about. So you’ll say, “Hey, do you know anything about David Mayer?” and OpenAI would just not engage at all. This is kind of mysterious. People did some investigations; as far as we can tell (per Ars Technica), this might be the result of an additional filter that OpenAI implements to deal with things like defamation claims. This would be a case where someone comes to OpenAI and says, “Hey, OpenAI is saying all sorts of lies about me; I don’t want OpenAI to talk about me at all; take my name out of your system.” I think this is really fascinating because it kind of reveals how these systems are being administered on the back end, and raises some really interesting questions. Because I think that if in the future, stuff like ChatGPT is the source of truth—you’re like, “Oh, I’m gonna meet Vagner for the first time; what do you know about Vagner Figueredo de Santana?”—your ability to pull information out of this system could be used for ill and could be used for good as well. I can imagine situations where you do want that privacy. I guess, Vagner, I already name-checked you, so maybe I’ll throw the question to you: How do you think companies should navigate the ethics of this? This is a really hard problem. Someone comes to you saying, “I don’t want ChatGPT to talk about me.” Is ChatGPT supposed to just say, “Okay, fine, we’re going to take you out of the system,” or is there an obligation for these models to be able to talk about everybody? I’m curious about what you think.

Vagner Figueredo de Santana: Yeah, it’s interesting that now we’re experiencing how legislation is impacting this kind of system because this has the flavored smell of something, or probably someone moving a case saying exactly that, “Okay, I don’t want this technology saying this, this, and that about me.” But at the same time, there are people that have the same name that cannot be recognized by this technology, and they may want that. And is interesting that when I got the list of names, I of course tried to replicate, and I even tried to combine the flow breaking with the list of names. It has this smell like, “Okay, this has to do with someone or some organization moved an action against the company, and then you cannot talk about that anymore.” And also, if you think about rights that people have with certain laws, we have the right to be forgotten, right? We may request a company that has our data to not provide that data. It depends on country or region, but for depending on the legislation where you are, you may have this right. So again, now we go back to the discussion we had before in terms of architecture, and probably the way that these models were trained, they were not prepared for that. And what we are seeing is the result of hard-coded rules that are excluding not the one person that should be, but everybody with the same name or with a similar name.

Tim Hwang: That’s right, yeah. There’s so much that happens, I think, because of this very specific situation where you spend so much money and time and resources to pre-train this model, and then you’re like, “Oh man, we have to fix all these things,” and it’s very hard to run that training process again, so we’re kind of forced to build all these things that we bolt on to patch up the holes in what we discover. Shobhit, do you have a view on this? Do you think... I mean, should people have the right to just write OpenAI a letter and have their name taken out of the system? I don’t know, maybe I’m just putting you in a box, but...

Shobhit Varshney: I think if you have a consolidation of one or two mega-texts that are controlling information flow, if you have a few people who have the authority to go censor stuff, that is very dangerous. You don’t want to be in a society where a few small people who fire their AI ethics board are controlling what is allowed and what not. But I do think that over time, you will have a split in the way the responses come to you. If you try to see what we’re doing in the media space, by definition, you’re self-reflecting bias towards how you consume information and what your beliefs are. You’ll either be on the spectrum of CNN and Fox; you’ll have one of the other spectrums. Over time, you figured out that most of the media, the way I want to consume, these are the certain agencies or media outlets that reflect my values or talk about the stuff that I’m interested in, and you get to there by looking at the clickstream of what I’ve spent more time on, what I forwarded more, and so on and so forth. So I do believe you’ll come to a point where you would have ChatGPTs of the world having more personalized flavors of things that they remember that you’ve asked in the past. So I think over time, you’ll get to a point where there’s some central authority that is making some broad recommendations around what happens to policies, but then there’s this personalized set of policies. I can go and block stuff on Twitter and say, “Don’t show me this again; this is irrelevant stuff.” They’ll start to become personalized to me. So I think we’ll get to a point where there’s a good combination. The responses that I get from ChatGPT may be very different from what Aaron is getting when he asks the same question because ChatGPT now knows so much about our preferences and is tailoring the answers to us.

Tim Hwang: Aaron, do you think so?

Aaron Baughman: I do. I think there are really good points. I wanted to just mention too that part of my day job is running live events within the sports entertainment, like the US Open, for example, ESPN Fantasy Football, and so on. During these live events, as Shobhit and Vagner were pointing out, sometimes we need to fix a problem then and there; we don’t have time to diagnose what it is and give a prognosis of it. That’s where we use similar techniques of a blocklist. A funny story in 30 seconds is, there’s a tennis player that has the last name “Sock,” and we found that it was generating content about socks and pants and shirts, nothing to do about tennis. We’re like, “Wait a second, how’s that happening?” So we had to quickly put in—I won’t call it a block, but a filter—to filter out data that had to do about clothing because it was just very ambiguous. Just these quick stopgaps until we have time to fix an issue that could be detrimental to society, I think, is important. But on the other hand, when we do that, we also have to be transparent about what we are blocking, what we are changing. There are many places in these large agentic systems to have these kinds of filters or blocks to do different types of functions. One last point that Vagner made was being able to delete data, so the field of machine unlearning I think is very important. It’s a very deep field; there’s work going on right now. It’s moving quick, but there’s different techniques where you could train a model across stratified data, so if you need to remove some type of data, you just simply remove that model because you know that data is embedded in there. But if you have one massive big model, it’s very difficult to remove it. So there’s different ways of handling this, and I just think as these new techniques come online, it just becomes easier. But there’s always side effects with them to do it. But just my word of caution is, let’s just be careful that we’re not censoring data in unintended ways, but we’re being transparent about how we’re creating these stopgaps.

Tim Hwang: Yeah, for sure. I think that’s one thing that’s kind of unique about this story is that no one knows why. People just find out that you can’t talk about people, and I think one improvement going forward, to your point, is we should at least have some kind of message that says why we can’t see this. Otherwise, I think we’re left to do what Vagner is doing, which is we try to replicate and then we try to speculate, and I think that’s maybe not the best situation for the ecosystem.

Well, as always, more to talk about than we have time to cover in one episode. So, Aaron, Vagner, Shobhit, thanks for joining us. And thanks for joining us, all you listeners. If you enjoyed what you heard, you can get us on Apple Podcasts, Spotify, and podcast platforms everywhere, and we will see you next week on “Mixture of Experts.”