Most everyone has heard of large language models, or LLMs, since generative AI has entered our daily lexicon through its amazing text and image generating capabilities, and its promise as a revolution in how enterprises handle core business functions. Now, more than ever, the thought of talking to AI through a chat interface or have it perform specific tasks for you, is a tangible reality. Enormous strides are taking place to adopt this technology to positively impact daily experiences as individuals and consumers.

But what about in the world of voice? So much attention has been given to LLMs as a catalyst for enhanced generative AI chat capabilities that not many are talking about how it can be applied to voice-based conversational experiences. The modern contact center is currently dominated by rigid conversational experiences (yes, Interactive Voice Response or IVR is still the norm). Enter the world of Large Speech Models, or LSMs. Yes, LLMs have a more vocal cousin with benefits and possibilities you can expect from generative AI, but this time customers can interact with the assistant over the phone. 

Over the past few months, IBM watsonx development teams and IBM Research have been hard at work developing a new, state-of-the-art Large Speech Model (LSM). Based on transformer technology, LSMs take vast amounts of training data and model parameters to deliver accuracy in speech recognition. Purpose-built for customer care use cases like self-service phone assistants and real-time call transcription, our LSM delivers highly advanced transcriptions out-of-the-box to create a seamless customer experience.

We are very excited to announce the deployment of new LSMs in English and Japanese, now available exclusively in closed beta to Watson Speech to Text and watsonx Assistant phone customers.

We can go on and on about how great these models are, but what it really comes down to is performance. Based on internal benchmarking, the new LSM is our most accurate speech model yet, outperforming OpenAI’s Whisper model on short-form English use cases. We compared the out-of-the-box performance of our English LSM with OpenAI’s Whisper model across five real customer use cases on the phone, and found the Word Error Rate (WER) of the IBM LSM to be 42% lower than that of the Whisper model (see footnote (1) for evaluation methodology).

IBM’s LSM is also 5x smaller than the Whisper model (5x fewer parameters), meaning it processes audio 10x faster when run on the same hardware. With streaming, the LSM will finish processing when the audio finishes; Whisper, on the other hand, processes audio in block mode (for example, 30-second intervals). Let’s look at an example — when processing an audio file that is shorter than 30 seconds, say 12 seconds, Whisper pads with silence but still takes the full 30 seconds to process; the IBM LSM will process after the 12 seconds of audio is complete.

These tests indicate that our LSM is highly accurate in the short-form. But there’s more. The LSM also showed comparable performance to Whisper´s accuracy on long-form use cases (like call analytics and call summarization) as shown in the chart below.

How can you get started with these models?

Apply for our closed beta user program and our Product Management team will reach out to you to schedule a call.As the IBM LSM is in closed beta, some features and functionalities are still in development2.

Sign up today to explore LSMs

1 Methodology for benchmarking:

  • Whisper model for comparison: medium.en
  • Language assessed: US-English
  • Metric used for comparison: Word Error Rate, commonly known as WER, is defined as the number of edit errors (substitutions, deletions, and insertions) divided by the number of words in the reference/human transcript.
  • Prior to scoring, all machine transcripts were normalized using the whisper-normalizer to eliminate any formatting differences that might cause WER discrepancies.

2 IBM’s statements regarding its plans, direction, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.  The information mentioned regarding potential future product is not a commitment, promise, or legal obligation to deliver any material, code or functionality. The development, release, and timing of any future features or functionality remains at IBM’s sole discretion.

Was this article helpful?

More from Artificial intelligence

Self-service that delights customers: How the IBM Partner Ecosystem is harnessing generative AI assistants in the banking and financial sectors

4 min read - Thanks to the transformative benefits promised by generative artificial intelligence (AI), the banking and financial sectors are at a turning point. From redefining a bank’s competitive edge in customer relationships to streamlining core banking operations and strengthening cyber-resiliency, AI technologies can unlock numerous new capabilities. Institutions are already seizing the opportunity. The 2024 Global Outlook for Banking and Financial Markets from the IBM Institute for Business Value (IBM IBV) revealed that 78% of the 600 executives surveyed tactically deploy generative…

Meta releases new Llama 3.1 models, including highly anticipated 405B parameter variant

7 min read - On Tuesday, July 23, Meta announced the launch of the Llama 3.1 collection of multilingual large language models (LLMs). Llama 3.1 comprises both pretrained and instruction-tuned text in/text out open source generative AI models in sizes of 8B, 70B and—for the first time—405B parameters. The instruction-tuned Llama 3.1-405B, which figures to be the largest and most powerful open source language model available today and competitive with the best proprietary models on the market, will be available on IBM®™ today where…

IBM watsonx Challenge empowers partners to solve real-world problems with AI

2 min read - In June, IBM invited ecosystem partners in Europe, the Middle East and Africa to participate in an IBM watsonx™ Challenge, a hands-on experience designed to bring the watsonx platform capabilities to some of the most important members of the IBM ecosystem. These ecosystem partners, who sell, build or service IBM technologies, enthusiastically embraced the challenge. Participants formed teams and focused on quickly crafting a solution to one of three selected challenges.   The challenges included using prompt engineering to analyze…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters