AI understands you? Yeah, right

Scotland is known for its majestic landscapes, kilt-wearing bagpipers and literary icons. But it also has an abundance of something that’s not quite a staple of glossy tourism brochures: sarcasm. So much so that when one Scottish company explored implementing a new agentic AI customer support system, the planners worried how the phone-based system would respond to the droll delivery common among their countrymen.

“They said, ‘We are a very, very sarcastic people. Is it going to pick up on it? Is it going to sound stupid if it doesn’t?’” recalled Tom Hebner, a Vice President at NeuraFlash, the AI consulting firm designing the system, in an interview with IBM Think.

But the Scots don’t have a monopoly on sarcasm. When a person in any country uses sarcasm while engaging with an interface powered by AI and natural language processing (NLP), there’s a high probability that they will be misunderstood. A deadpan “I love being kept on hold” or “Thanks for your lack of help” may not register as intended.

“In NLP, one of the key hard things is understanding sarcasm and emotion,” Anupama Ray, a Senior Research Scientist at IBM Research in India, told IBM Think. Ray pointed to research on machine learning models and sarcasm. “For detecting sarcasm, they’re often at 60 to 70%.”

Researchers use what are known as F1 scores to evaluate and compare model performance. Scores in the 75 to 80% range are considered strong, but even popular large language models (LLMs) aren’t making the grade. A 2025 study out of the Quantum Intelligence and Language Understanding Lab at China’s Tianjin University evaluated sarcasm detection in more than a dozen leading LLMs. The models’ F1 scores rarely exceeded 65% (or 0.65), averaged across different datasets. 

Among LLMs, “[i]n simplified or constrained expressions, sarcasm detection already shows practical potential,” Yazhou Zhang, an Associate Professor at Tianjin who led the study, said in an email to IBM Think. “However, in open-world dialogue, it remains unreliable.”

These underwhelming performances come despite decades of research on computational systems and sarcasm detection. In fact, every few years, a headline claims that someone has cracked the code, but in reality, progress has been largely incremental—for good reason. Sarcasm, it turns out, lives right in the weak spot of a typical AI model. 

“A lot of AI really depends on the most likely output,” Aditya Joshi, a Senior Lecturer at the School of Computer Science and Engineering at the University of New South Wales in Sydney, Australia, told IBM Think. “Sarcasm spins this on the head because it is actually a rare phenomenon. The most likely output based on the words that you read would actually be incorrect. And I think this is a violation of a basic maxim of AI.” 

Despite the challenge, researchers including Ray, Joshi and Zhang are working to build market-ready sarcasm detection. Their work points to the complex behaviors and rituals around sarcasm that help humans convey and detect it. But to impart that context to AI models, researchers face that classic nettlesome challenge to machine learning: they need training data.

Training AI models takes lots of data, and training for sarcasm detection is no exception. A lack of large, high-quality datasets has proven a major hurdle in advancing sarcasm detection training, IBM’s Anupama Ray said. That shortage is especially acute for multimodal datasets that include some combination of text, audio and visual dimensions.

Early sarcasm detection projects largely focused on text datasets. Research suggests that’s why those efforts fell short: for a 2025 survey paper, researchers Xiyuan Gao, Shekhar Nayak and Matt Coler from the University of Groningen in the Netherlands scoured AI sarcasm detection literature dating back to 2006 and concluded that “no single modality or feature set is sufficient.”

The University of Groningen team, Ray and other researchers today want to train models to identify prosodic cues (such as tone, vocal pitch or rate of speech) and facial expressions (like pursed lips or raised eyebrows) to recognize sarcasm. This multimodal sarcasm detection, they believe, could help virtual assistants and assistive care bots engage more effectively with users.

“Datasets need to be multimodal, as audio and video cues are critical for sarcasm detection,” Ray said. “Also, such training sets should have the context of the conversation, as often the context determines whether the sentence in question is sarcastic or not, even when spoken in a flat tone or with a straight face.”

But designing multimodal datasets for sarcasm detection is complicated. The Multimodal Sarcasm Detection Dataset (MUStARD), perhaps the most famous multimodal dataset for sarcasm, consists of video clips from popular television sitcoms like The Big Bang Theory, Friends and The Golden Girls. For example, it includes this scene from the Friends episode “The One with Ross’s Tan”:

Chandler: Hold on, there’s something different.
Ross: I went to that tanning place your wife suggested.
Chandler: Was that place the sun?

MUStARD, created in 2019 by researchers based in Michigan and Singapore, is a rich, annotated dataset, but has a severe limitation. The researchers started with more than 6,000 samples, but most didn’t contain sarcasm. To curate a collection with a balanced number of sarcastic and non-sarcastic clips, they winnowed the training set to fewer than 700 video clips—a pittance by AI training standards.

Fortunately, other researchers have built upon the MUStARD team’s work. In 2022, Anupama Ray and a team of researchers at the Indian Institute of Technology Bombay doubled the size of the dataset and enriched it with additional annotations, including labels indicating the intensity and valence (positive or negative) of emotions associated with the sarcasm in the samples. The team called the new dataset MUStARD++ and emphasized its relevance to AI-powered conversational systems. “Detecting emotions and sarcasm is crucial for all services involving human interactions,” they wrote.

Meanwhile, Xiyuan Gao of the University of Groningen piloted another approach to augmenting MUStARD: she and fellow researchers used a deep-learning based translation service to translate MUStARD’s English-language clips into other languages, then translate them back into English. This “back translation” process resulted in paraphrased versions of the original samples: for instance, in one case, the word “and” replaced “plus,” while “rather” replaced “fairly.” These re-translations were different enough from their sources to constitute unique datapoints. The team then turned these paraphrased samples into audio recordings through sarcasm-oriented speech synthesis, using a fine-tuned FastSpeech 2 model designed to better preserve sarcastic intonation.

Through this process, the team got 20 times the text data. “Then we used speech synthesis to synthesize the sarcastic voices,” Gao explained in an interview with IBM Think. “The key point is to keep everything aligned. We don’t want the text with one meaning, and the [intonation] of the audio saying something else.” It was an innovative process that produced enough samples to support “scientifically meaningful” work, Matt Coler, Gao’s colleague, said. “This, for us, is the first step.”

The next step involved moving beyond highly stylized fictional sarcasm to something more authentic: podcasts. Last year, Coler and Gao took part in research—led by their university colleague Zhu Li—to create a sarcasm detection dataset based on episodes of a literature- and mythology-focused podcast fittingly called The Overly Sarcastic Podcast. Part of what makes the new dataset, dubbed PodSarc, unique is that it’s based on real people talking to each other, rather than scripted interactions between actors.

“In the open domain, there are hardly any datasets which are close to real sarcasm in day-to-day life,” said Shekhar Nayak, who also worked on the PodSarc research. PodSarc, he told IBM Think, is “more realistic compared to sitcom data, and it does not have artifacts like a laugh track.”

What’s most notable about the PodSarc dataset, however, is not necessarily its source material. The 2025 paper introducing PodSarc largely focuses on the researchers’ annotation strategy.

Annotation for multimodal sarcasm datasets is typically an arduous process, with annotators reviewing hundreds or thousands of audio or video clips and then labeling them as sarcastic or non-sarcastic. Studies like Ray’s, which explore the emotions associated with sarcasm, require even more labeling. But with PodSarc, the researchers assigned the first round of labeling to the LLMs GPT-4o and LLaMA 3, before tasking further rounds to a human annotator. “Initial annotations by LLMs help save time and cost,” Nayak said.

While LLMs have become a common data labeler, deploying them here may seem counter-intuitive: after all, isn’t the whole point of this research teaching models to detect sarcasm better? Deploying them for sarcasm dataset annotation suggests they’re already good enough. But the research, like Zhang’s study, says otherwise.

Untangling these contradictions means discerning between experimental and real-world environments. In an academic setting, with the right human intervention, present-day LLMs can still be put to work, with fruitful results. In the PodSarc research, Nayak said, GPT-4o tended toward false negatives, while LLaMA 3 toward false positives, with the result that they disagreed with each other more than a quarter of the time. That’s when the human annotator stepped in. For now, this human –and AI tag-teaming seems to give the best results, Nayak said—at least “until LLMs evolve.”

Some researchers are going beyond the traditional text, audio and visual modalities to incorporate more novel features into training datasets. A year after her work on sarcasm and emotion recognition, Anupama Ray joined other researchers in publishing a paper on how gaze behavior—that is, how people’s eyes move while they read text—can indicate sarcasm in the text.

Ray explained that in attempting to understand written sarcasm, readers often diverge from linear reading patterns to re-examine certain words or phrases. “Sometimes we go back to things, and sometimes we skip forward to things,” she told IBM Think. “The number of times you go back for a sarcastic sentence is more than for a non-sarcastic sentence.”

But recording and studying gaze patterns is a complicated process. Diptesh Kanojia, who worked with Ray on the 2023 study, recalls spending hours with participants in Mumbai as they stared into screens while sensors tracked the most miniscule of eye movements.

“To be able to measure the eye movement on screen [with the precision of] one by 2000th of a second or one by 1500th of a second requires such gigantic setups and data to be collected from multiple people,” said Kanojia, currently a Senior Lecturer at the Surrey Institute for People-Centered AI in the UK. “We would have to provide them breaks every 15 or 20 minutes, and drinks and snacks.”

Fortunately, the work got results. Kanojia, Ray and fellow researchers developed a framework incorporating their gaze behavior data into sarcasm detection architecture. They included an attention mechanism for fusing gaze features with other features (text, audio and video) and a neural network that used those fused features to discern sarcasm. In their 2023 paper, the team reported a substantial improvement in sarcasm detection compared to detection without gaze data.

While these studies dramatically improved the data available for sarcasm detection, they all dealt only with English data. Gao and Zhang have separately contributed to Chinese sarcasm datasets, and there are also Spanish and Hindi datasets with labeled sarcasm samples. Overall, however, sarcasm data still suffers from a lack of linguistic diversity.

“The lack of language diversity restricts the scope of multilingual sarcasm research, potentially overlooking the cultural nuances inherent in sarcastic communication,” Gao, Nayak and Coler concluded in their 2025 survey paper. The team called for the development of culturally diverse datasets for future global research efforts.

Back in Australia, Aditya Joshi, at the University of New South Wales, is doing his part to improve cultural diversity in sarcasm research datasets. His language of choice? English. No, really. Joshi and other researchers have found that there are diversity gaps even within English-focused sarcasm and sentiment research, partly because forms of English vary from country to country. In a study published in 2025, Joshi found that leading LLM models performed considerably worse on Indian English than on British or Australian English. “If you are a company dealing with a multicultural user base and you’re trying deploy your models in a new country, your models may not perform as well as you’re claiming,” even within English-speaking countries, he explained in an interview with IBM Think.

With these concerns in mind, Joshi and his co-authors developed Benchmark for Sentiment and Sarcasm Classification for Varieties of English (BESSTIE). The text-based dataset features sarcastic and non-sarcastic language from Google reviews and Reddit posts in British, Australian and Indian English. The goal, Joshi’s team says, is to help organizations recognize and address bias in AI sarcasm and sentiment detection.

“It makes sociological sense from a linguistic equity perspective,” Joshi said. “It is also important for us to understand these language varieties and the way our language models perform on them as a stress test—for the technology of the future.”

As researchers develop better datasets and release them publicly, will major AI companies take advantage of them to improve their own models’ sarcasm detection? If they are, they’re keeping those plans under wraps. Researchers told IBM Think they hadn’t heard directly from AI giants on the subject, and companies such as OpenAI, Anthropic and Meta didn’t respond to emails from IBM Think.

Meanwhile, at least one conversational AI company is aiming to solve the problem without any use of sarcasm detection datasets. Frank Reply, a startup based in Frankfurt Germany, is instead focused on employing what the company calls “metacognitive prompting” to improve interactions with digital assistants, as Frank Reply Senior Manager Daniel Fitzpatrick and UX Lead Maria Müller explained in an interview with IBM Think. Their method entails giving an LLM detailed instructions “to teach it how to think,” Fitzpatrick said. “We’ve been highly successful in using linguistic principles to do so.”

To teach models to detect sarcasm, Fitzpatrick and Müller explained, Frank Reply relies on a set of linguistic principles identified by the 20thcentury British philosopher and linguist Paul Grice: that for participants in conversations to achieve mutual understanding, communication needs to be informative, truthful, relevant and clear. When a speaker flouts these rules, the listener might infer an unstated meaning—a phenomenon known as Gricean implicature. Sarcasm is a type of Gricean implicature: when someone steps in, drenched from a rainstorm, and says “Beautiful day out!” they’re not trying to trick anyone. They assume the audience will infer their real meaning from the circumstances.

Frank Reply’s approach is to remind the model to check for those circumstances.

“We tell the speech interface, if the context of the scenario would indicate that implicature or sarcasm would play a role, use these instructions to detect that sarcasm, and we specifically tell [the model] to look for Gricean implicature,” Müller said. Large language models are as versed in Gricean implicature as in any other well-documented concept.

Since this method is so new, Frank Reply doesn’t yet have statistics on the performance of their methodology, but they’re forging ahead. “For us,” Fitzpatrick said, “it’s one necessary step in the long march towards making completely human-like voice interactions.”

In the academic world, researchers are leveraging their new datasets to develop their own models and frameworks for AI sarcasm detection, such as Anupama Ray’s framework incorporating gaze behavior and an attention mechanism into detection architecture.

The University of Groningen’s researchers also devised a system that relied on an attention mechanism for fusing features (text and audio) from their augmented dataset. The resulting model, when trained on the team’s largest augmented dataset, achieved an 81% F1 score.

The University of Groningen’s Xiyuan Gao, however, cautioned that the impressive score wasn’t necessarily a reliable proxy for real-world performance. “I’d say there’s no single F1 score showing that a sarcasm detector is ready,” she wrote in an email to IBM Think. “For evaluating readiness, I think a system-level criteria is more suitable …[It’s] about if the system is robust across different speaking styles, and how it handles uncertainty.” Accordingly, in their 2026 paper, Gao and her colleagues positioned their research as “pav[ing] the way” for future research.

Notably, neither Gao’s nor Ray’s team centered LLMs in their sarcasm detection frameworks. Multimodal LLMs fine-tuned with new datasets “might be the next step to advance sarcasm detection and understanding,” Ray said. “However, current methods using multimodality and context are still better than single LLMs looking only at the textual inputs.”

According to Nayak, applications currently powered by LLMs, such as voice assistants, could potentially incorporate elements of the sarcasm researchers’ work. Based on a specific task, it could even be possible for applications to switch between LLMs and sarcasm detection frameworks. “There is no ‘one perfect solution’ to every intended application,” he said.

Back at Neuraflash, Hebner’s own search for a sarcasm solution—at least the kind his Scottish clients asked about—came up empty. For now, his client will have to make do without, though Hebner hopes that that won’t be the case forever.

Building sarcasm detection into AI-powered conversational systems, Hebner said, is a step toward improving the customer experience. When customers feel understood, they’re more likely to keep using the system rather than growing frustrated and demanding to speak to a human representative.

“If they say something sarcastic and [the system] doesn’t pick it up … then they’ll break out and they won’t use the system,” he said. “We can make it so that interacting with companies is easy and not something you dread … We can actually do something good.”