The value of experimentation: A guide to experimenting with new AI technologies

According to a recent BCG survey of over 1,000 C-suite executives, “the vast majority ranked AI and gen AI among their 3 top technology priorities for 2024, but 66% expressed ambivalence or dissatisfaction with their progress,” highlighting a gap between intention and action. Technological innovation moves quickly, making agility more critical than ever.

Experimentation can empower employees to innovate rapidly. Not only does it drive innovation, but it also enables companies to make evidence-based decisions that drive value. For example, experiments have helped eBay uncover millions in wasted annual advertising spend, empowered H&R Block to build trust with customers, and enabled Booking.com to scale from a start-up to one of the largest accommodation platforms. Experimentation fuels leading tech companies such as Netflix, Microsoft, Meta and many others.

Despite the compelling evidence for experimentation, determining where to begin can be challenging. With the ever-growing landscape of artificial intelligence (AI) technologies, how do you choose the right one? Should you focus on enterprise-wide solutions or specialized tools? Should you use a vendor or create your own solution? What use cases should you focus on? We share guidance to help you overcome stagnation and become a force for innovation.

When considering AI solutions, you can think about them across a spectrum:

• Enterprise-wide solutions: These address problems relevant to all employees, such as enterprise search and insight generation. They are typically prebuilt and require minimal development power.
• Specialized tools for power users: These solutions require advanced parsing and insight generation capabilities. They might need legal support and development power to create customized solutions for high-impact use cases.
• Custom solutions: These are specialized tools created to address specific use cases. They require development power, legal support and a deep understanding of your industry and its use cases.

Here are four tactical tips for experimenting with new AI technologies:
 

1. Choose pilot users: Select a group of engaged, problem-solving individuals, such as members of your innovation or technology teams.
2. Define relevant use cases: Gather insights from pilot users to identify the most valuable use cases. Explore resources such as this IBM AI Academy course to learn how to select the right use cases for your business.
3. Provide training and support: Offer user guides, tips, prompt libraries and clear explanations of what the tool can and cannot do. This helps users quickly get up to speed and maximize the tool’s potential.
4. Track impact relentlessly: Use surveys and track usage metrics—such as daily or weekly usage, search satisfaction and click rates—to track the tool’s effectiveness. Incorporate pilot feedback into your product roadmap and vision.

When deciding between AI tools or identifying use cases for your business, consider these key evaluation criteria:

1. The AI tool’s capabilities: Evaluate the tool’s performance, accuracy and reliability for your use case. Test it with sample datasets or use a proof-of-concept (PoC) to demonstrate feasibility.
2. Compatibility with existing systems: Ensure that the tool integrates seamlessly with your current technology stack, including data sources, application programming interfaces and other software components.
3. Alignment with business objectives: Determine whether the AI tool supports your business goals and objectives, such as increasing productivity, efficiency or revenue. Consider how it addresses specific challenges within your organization.
4. Cost and licensing: Evaluate the tool’s pricing structure, including subscription fees, setup costs and ongoing maintenance expenses. Consider whether the cost aligns with your budget and offers good value for its features.
   
   

By carefully considering these evaluation criteria, you can make informed decisions about which AI tool best suits your business needs and identify use cases that maximize their potential impact.

Establish a governance board that includes professionals from across your organization, prioritizing members from the legal team and, if available, an innovation team. This helps ensure compliance and provides a unified perspective on all AI initiatives.

IBM’s work with PepsiCo showcases the importance of standard operating procedures for implementing AI. PepsiCo’s governance board helps streamline AI ideas and PoCs from global teams by assessing, validating and approving use cases against their responsible AI principles. The board also shared best practices and accelerators while helping mitigate risks.

This centralized approach enabled PepsiCo to identify gaps in existing capabilities and objectively evaluate AI vendors. By mapping common AI use cases to standard architectural patterns, they developed reusable AI services for all teams, standardized deployment and expedited time-to-value.

With the right approach, experimenting with new AI technologies can lead to significant benefits for your organization. With this framework, you can successfully navigate the AI landscape and unlock the full potential of these innovative technologies.