How to manage AI agents without wrecking your business

On April 24, Jeremy Crane, Founder and CEO of car rental platform PocketOS, accidentally deleted his company’s production database, along with its backups. Or, more precisely, his AI agent did. Thirty hours later, while working to restore service for his customers, he paused to tell the world what had gone wrong. In a now-famous post on X, Crane described how the agent “took down a small business serving rental companies across the country.”

Crane had deployed an AI coding agent for a routine task. A Cursor agent, powered by Anthropic’s Claude Opus 4.6, was operating in a staging environment when it encountered a credential mismatch. The agent chose to remediate it by deleting a Railway volume, a persistent storage unit for application data provided by Railway, PocketOS’s infrastructure provider.

To carry out the deletion, as Crane wrote, “the agent went looking for an API token … in a file completely unrelated to the task.” It discovered a token that had full access across the Railway GraphQL API, including destructive operations like volume delete. It then issued a command against a volume with a name that was identical in both staging and production environments. There was, Crane noted, “No confirmation step … No environment scoping. Nothing.” The agent deleted the production database and the volume backup.

Without the production database, PocketOS’s rental business customers lost their reservation records, new customer signups and other critical data. The incident unfolded in nine seconds as he watched from his computer, Crane told IBM Think in an interview. “And then I started getting texts on my phone because systems were going down and I was like, ‘Wait, what just happened?’”

It might sound like a vibe-coding project gone wrong. Crane, an experienced developer and entrepreneur, emphasizes that this is not the case. In his post, he notes that “We were running the best model the industry sells, configured with explicit safety rules in our project configuration … The setup was, by any reasonable measure, exactly what these vendors tell developers to do.”

Crane’s story points to an industry-wide infrastructure and governance challenge for people working with AI agents. Until now, digital systems and tools “were built around humans,” Crane said. “Now, we have computers making decisions in split seconds and our DevOps and our security protocols aren’t prepared for that world.”

As AI absorbs a widening array of tasks previously handled by workers, discussions are emerging about the role and value of human judgment.

In an unpublished 2025 survey by IBM’s Institute for Business Value (IBV), 60% of executives across industries said they believed generative AI would make problem-solving skills more important. But many anticipated a weakening of core skills: 60% expected decision-making to erode, 51% foresaw diminished critical thinking and 41% anticipated an erosion in ethical judgment.

Research supports these concerns. A Wharton School study on cognitive surrender shows that when AI systems are available, users tend to accept their outputs with little scrutiny and greater confidence, even when those outputs are wrong. Managers are also less likely to carefully review documents they believe were produced by an “AI employee,” according to a study by Boston Consulting Group and a Boston University professor who studies how AI impacts workers.  At a time when business leaders care most about critical thinking and judgment, these studies suggest that workers are disinclined to apply those skills to AI-generated work. 

Meanwhile, 55% of organizations are actively developing or deploying an agentic AI operating model, according to the 2026 IBV report, “The blueprint for agentic operations.” And 60% plan to adopt next-generation delivery structures where AI agents coordinate integrated workflows across finance, supply chain, HR, procurement, physical operations and customer service.

In working with agents, humans “have to understand ethics, guardrails, governance in a way that they haven’t before,” James Cook, Senior Partner at IBM’s Global Center of Competence, said to IBM Think in an interview.

That need becomes evident even in seemingly simple decisions. Determining who’s in charge of which agent—and when—can quickly become complicated, said Cook, who works with organizations on integrating AI agents into the workforce. “When one agent can undertake multidisciplinary tasks, it becomes much more complex as to who’s in charge of the actions of that agent,” he said.

For example, an employee inquiring about parental leave might ask an HR agent to explain policy (HR advisory), calculate eligibility (data interpretation), suggest a leave plan (decision support) and initiate a leave request workflow (process execution). At that point, the agent is no longer performing a single, discrete task; it is spanning advisory, operational and transactional roles simultaneously.

If the agent gives incorrect guidance, applies a policy improperly or triggers an action that has downstream consequences, it “needs to be clear whether ownership sits with HR, IT, compliance or the process owner,” Cook said.

Francesca Rossi, Global Leader for Responsible AI and AI Governance at IBM, stresses that AI should never be held responsible for what it produces. “You can say AI is a partner,” she said in an article for IBM Think, “However, if by partner you mean two entities that have the same roles and responsibilities, AI is not really a partner, but a collaborator.”  She continued: “The only accountable entity is the human being, and the only one that bears responsibilities for whatever is generated is the human being.”

As they handle more complex tasks, autonomous AI agents are increasingly presented as “AI co-workers” or “AI employees.” The startup Kuse, for instance, offers an agent named Junior that can draft and send emails, optimize media spend on a campaign, attend meetings and escalate missed deadlines to managers—tasks that might be otherwise handled by an administrator.

In one case, a customer mentioned that “Junior was only told to run an A/B test on outreach emails, but it proactively iterated on the results on its own before the customer stopped Junior and told it to redo the test,” Kuse Co-Founder and CTO Yuhao “Austin” Xu told IBM Think in an interview. Junior launched in March and has more than 40 active enterprise customers. But even after integrating Junior into a workflow, there are some functions where “humans are irreplaceable,” Xu said. “From what we’ve seen across our customers, the human role clusters around three areas: judgment (deciding strategy … interpreting ambiguous situations where the ‘right’ answer depends on business context that’s hard to formalize), verification and planning, and approving critical or irreversible actions.”

A mechanism in Junior requires customers to click “approve” on high-stakes actions such as sending a contract or deleting data. According to Xu, customers have said this mechanism gave them the “confidence to let an AI operate inside their real workflows.”

The importance of reviewing and verifying AI outputs can’t be overstated, says Merve Unuvar, Director of Agentic Applications and Middleware at IBM. “The better the agent becomes, the more dangerous blind trust becomes,” Unuvar told IBM Think in an interview. “When an agent delivers good results, humans …stop questioning outputs with the same rigor as in day one, but agents still make mistakes.”

When those errors go unchecked, the consequences of skipping the verification step run the gamut from embarrassment and job loss to legal penalties.

Human oversight can also halt an AI agent’s confident repetitive loops. “Even if I tell the agent that it already tried those steps—twice—it will apologize and then offer similar suggestions,” she said. “A human knows to pause, investigate further and try a different approach.”

In addition, managers overseeing human and agent teams will need to develop a “new skill set centered on reviewing and assessing the capabilities of AI agents,” said Cook, the IBM GCC senior partner. That begins with “redesigning and rethinking hybrid [human-plus-agent] operating models,” he said, including through mechanisms such as clearly defined handoffs between humans and agents, explicit escalation thresholds and governance boundaries that determine what agents can autonomously decide versus when humans must intervene.

Furthermore, part of a manager’s job is to “maintain their understanding of the capabilities and limitations of agents,” Cook said, similar to how they would track the growth of a junior employee.

Organizations can support managers in overseeing agents by certifying both the agents’ skills and the human manager’s ability to manage those agents to ensure quality. Given the limits of how many agents one human can effectively oversee, a potential solution is “a hierarchical orchestration of agents supervising other agents,” alongside auditing, governance controls and defined escalation rules.

“Ultimately, enterprises will have to retrain their managers as well as the people designing and developing AI to think and manage in a different way,” Cook said.

Employers are seeking workers who are skilled in using and evaluating AI, from identifying the appropriate AI tool or agent for a task, to eliciting quality outputs and revising those outputs as needed. The National Association of Colleges and Employers (NACE) recently reported that demand for AI skills in entry-level jobs has nearly tripled since autumn 2025. In an interview with IBM Think, Shawn VanDerziel, NACE President and CEO, said that tech-savvy entry-level workers are “being hired who can … experiment with AI for their companies.”

The rising demand for AI expertise has in some cases prompted companies to create new positions dedicated to AI. Box CEO Aaron Levie, for instance, told the New York Times that his company created 13 new roles, with titles such as “AI Architect,” “AI Platform Leader” and “AI Solutions Manager.”

But AI expertise is becoming vital in all kinds of roles. “The role of humans moves from doing the work to interrogating outputs [such as] evaluating for accuracy, bias, quality and relevance to context and people,” Natasha Pillay-Bemath, VP of Global Talent Acquisition and Executive Search at IBM, told IBM Think in an interview.

For example, Pillay-Bemath said, the skills expected of entry-level engineering roles have evolved from manual coding to using AI as an enabler, along with validating and testing its output. As such, the “strongest performers in this new world are not passive consumers of AI output,” she said. “They actively challenge, refine and improve it. They are natural dissenters.”

Enterprises may expect employees to use their judgment when reviewing AI outputs, but knowing how to do so isn’t always obvious. For example, when a worker disagrees with the model’s output, what happens next? Also, without clear context, it can be difficult for the employee to understand what truly matters in a given situation or how to recognize an appropriate course of action. Put simply, what does “good judgment” look like in practice?

One strategy, according to the IBV report “Where AI breaks—or breaks through,” is to create playbooks with short, practical scenarios that illustrate what strong decisions look like in ambiguous situations. The report also stresses the importance of defining when decisions should be escalated and to whom.

As the report’s authors note, “In reality, judgment becomes reliable only when leaders define what [good judgment] looks like and reinforce it consistently.”

Unuvar and Pillay-Bemath recommend seven strategies to get the most out of AI agents:

1. Build first, then evaluate critically
Use agents to quickly prototype and create outputs, then assess the results.

2. Trust but verify
Even highly accurate agents can fail; always double-check its work.

3. Don’t stop at the first answer or question
Question the framing, not just the answer. Ask what other questions should be considered and how upstream and downstream effects are connected.

4. Understand what the agent doesn’t know
Stay informed about your field so that you can spot missing context or nuance that the agent might overlook.

5. Don’t let agents loop without intervention
Agents can get stuck repeating incorrect reasoning, so step in when patterns repeat without progress.

6. Don’t ignore trust, transparency and human relationships
Be intentional about strengthening relationships and trust with coworkers and clients, since purely AI-driven work can erode them.

7. Don’t assume your role disappears when it shifts
Guide direction and thinking at a system level; workers’ roles are evolving from task execution to evaluating outputs.

As expectations increase for humans to apply critical thinking and judgment to AI-generated outputs, organizations are reaching an inflection point: they must decide whether to invest in systems and governance that enable real human judgment, or risk reducing the process into a rubber-stamping exercise.

Jeremy Crane’s deleted database is a casualty of this risk, says Phaedra Boinodiris, IBM Consulting’s Global Leader for Trustworthy AI. “This is not a story about a rogue AI,” she told IBM Think over email. “This is a story about liability laundering at infrastructure scale.”

Liability laundering, Boinodiris explains in an article for IBM Think, is when a human is in charge of overseeing an AI’s work, but “the conditions for genuine oversight—the architectural transparency, the training, the authority to halt, the feedback mechanism—were not established.”

To provide meaningful oversight of AI outputs, Boinodiris explains in another article, humans need to interrogate five aspects of their AI system:

1. Transparency: what data, methods and validation choices underlie the model
2. Explainability: how and why the model reaches certain outputs
3. Observability: how the system performs over time
4. Algorithm optimization: what the model and its associated algorithms are optimized for
5. Robustness against adversaries: whether the model has been altered since assessment

Workers also need to get familiar with the mistakes that AI models make, and how to prevent or address them. A paper from Microsoft researchers found that frontier large language models (LLMs) and basic agentic tools can subtly yet severely corrupt the data they work on. Workers need to be on alert for these changes, including ones that human coworkers wouldn’t make.

Familiarity with AI’s weaknesses is only part of meaningful oversight. Workers must have the authority to challenge or halt AI-assisted processes when outputs appear unreliable.
Research from the Frankfurt School of Finance & Management found that managers tend to penalize workers for overriding AI recommendations, even when doing so results in better outcomes. The researchers suggest that organizations should redesign incentives and oversight mechanisms so that employees can prioritize the quality of outcomes instead of adherence to AI recommendations.

AI can be remarkably effective at many tasks—and its capabilities continue to advance. However, “More capable AI demands more intentional governance, not less,” Maryam Ashoori, VP of Product and Engineering for IBM watsonx.governance, told IBM Think in an interview. “Organizations need clear structures that keep humans accountable for the decisions that matter most. The question isn’t whether humans are in the loop; it’s whether they’re in control.”

We’ve distilled insights from IBV research, Boinodiris’s work and previous IBM Think work into a rubric for performative versus substantive AI practices.  Here’s how to distinguish between optics and impact.

A premium is emerging on capabilities that machines cannot yet fully reproduce: discerning what matters, making decisions amid ambiguity, leading with nuance and empathy, navigating novel situations that defy standard solutions and reading the room.

But the value that humans add when working with AI agents is meaningless without operational infrastructure that allows them to apply judgment, underpinned by governance, transparency, guardrails and escalation paths to supervise agentic systems at machine speed.

The story of Crane’s deleted data ends happily. Railway, PocketOS’s infrastructure provider, restored the data after Crane got in touch with the company’s CEO. A few days after the incident, Railway referenced Crane’s viral post in a blog detailing new guardrails that it added to its platform.

This is not to say that the fallout wasn’t significant. Crane worked to help clients reconstruct their bookings from Stripe payment histories, calendar integrations and email confirmations. But for at least a day, when customers arrived to pick up their vehicle, the rental companies didn’t have records of who those customers were. “Data they relied on to run their Saturday morning operations, gone,” he wrote on X.

Crane’s experience has led him to pause work on a new startup until he’s reanalyzed the AI safety and risk parameters, he told IBM Think. At the same time, he remains optimistic. “I’m very pro-agentic AI, and this experience hasn’t changed that for me,” he said. “But you have to harness it, and you have to know where it can bite you.”