Home Topics What is Natural Language Processing? What is NLP (natural language processing)? 
Learn about IBM's NLP solutions Subscribe to AI topic updates
Robot face processing human talk and learning from it

Updated: 6 June 2024
Contributor: Jim Holdsworth

What is NLP?

Natural language processing (NLP) is a subfield of computer science and artificial intelligence (AI) that uses machine learning to enable computers to understand and communicate with human language. 

NLP enables computers and digital devices to recognize, understand and generate text and speech by combining computational linguistics—the rule-based modeling of human language—together with statistical modeling, machine learning (ML) and deep learning. 

NLP research has enabled the era of generative AI, from the communication skills of large language models (LLMs) to the ability of image generation models to understand requests. NLP is already part of everyday life for many, powering search engines, prompting chatbots for customer service with spoken commands, voice-operated GPS systems and digital assistants on smartphones.

NLP also plays a growing role in enterprise solutions that help streamline and automate business operations, increase employee productivity and simplify mission-critical business processes.

How to choose the right AI foundation model

Use this model selection framework to choose the most appropriate model while balancing your performance requirements with cost, risks and deployment needs.

Related content

Register for the white paper on AI governance

Benefits of NLP

A natural language processing system can work rapidly and efficiently: after NLP models are properly trained, it can take on administrative tasks, freeing staff for more productive work. Benefits can include:

Faster insight discovery: Organizations can find hidden patterns, trends and relationships between different pieces of content. Text data retrieval supports deeper insights and analysis, enabling better-informed decision-making and surfacing new business ideas.

Greater budget savings: With the massive volume of unstructured text data available, NLP can be used to automate the gathering, processing and organization of information with less manual effort.

Quick access to corporate data: An enterprise can build a knowledge base of organizational information to be efficiently accessed with AI search. For sales representatives, NLP can help quickly return relevant information, to improve customer service and help close sales.

Challenges of NLP

NLP models are not perfect and probably never will be, just as human speech is prone to error. Risks might include:

Biased training:  As with any AI function, biased data used in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias.

Misinterpretation: As in programming, there is a risk of garbage in, garbage out (GIGO). NLP solutions might become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise.

New vocabulary: New words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure—and either way, this creates a complication.

Tone of voice: When people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable.

Human language is filled with many ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn—and many never stop learning. But then programmers must teach natural language-driven applications to recognize and understand irregularities so their applications can be accurate and useful.

How NLP works

NLP combines the power of computational linguistics together with machine learning algorithms and deep learning. Computational linguistics is a discipline of linguistics that uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure. 

The parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output—either text or speech—understandable to both NLP models and people.

Self-supervised learning (SSL) in particular is useful for supporting NLP because NLP requires large amounts of labeled data to train state-of-the-art artificial intelligence (AI) models. Because these labeled datasets require time-consuming annotation—a process involving manual labeling by humans—gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data.

Three different approaches to NLP include:

Rules-based NLP: The earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable.

Statistical NLP: Developed later, statistical NLP automatically extracts, classifies and labels elements of text and voice data, and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging.

Statistical NLP introduced the essential technique of mapping language elements—such as words and grammatical rules—to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones).

Deep learning NLP: Recently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, unstructured data—both text and voice—to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses neural network models. There are several subcategories of models:

  • Sequence-to-Sequence (seq2seq) models: Based on recurrent neural networks (RNN), they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English).

  • Transformer models: They use tokenization of language (the position of each token—words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. Transformer models can be efficiently trained by using self-supervised learning on massive text databases. A landmark in transformer models was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works.

  • Autoregressive models: This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, Llama, Claude and the open-source Mistral.

  • Foundation models: Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the IBM Granite™ foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text.

For a deeper dive into the nuances between multiple technologies and their learning approaches, see “AI versus. machine learning versus deep learning versus neural networks: What’s the difference?

NLP tasks

Several NLP tasks typically help process human text and voice data in ways that help the computer make sense of what it’s ingesting. Some of these tasks include:

Linguistic tasks

  • Coreference resolution is the task of identifying if and when two words refer to the same entity. The most common example is determining the person or object to which a certain pronoun refers (such as, “she” = “Mary”). But it can also identify a metaphor or an idiom in the text (such as an instance in which “bear” isn’t an animal, but a large and hairy person).

  • Named entity recognition (NER) identifies words or phrases as useful entities. NER identifies “London” as a location or “Maria” as a person's name.

  • Part-of-speech tagging, also called grammatical tagging, is the process of determining which part of speech a word or piece of text is, based on its use and context. For example, part-of-speech identifies “make” as a verb in “I can make a paper plane,” and as a noun in “What make of car do you own?”

  • Word sense disambiguation is the selection of a word meaning for a word with multiple possible meanings. This uses a process of semantic analysis to examine the word in context. For example, word sense disambiguation helps distinguish the meaning of the verb “make” in “make the grade” (to achieve) versus “make a bet” (to place). Sorting out “I will be merry when I marry Mary” requires a sophisticated NLP system.


User-supporting tasks

  • Speech recognition, also known as speech-to-text, is the task of reliably converting voice data into text data. Speech recognition is part of any application that follows voice commands or answers spoken questions. What makes speech recognition especially challenging is the way people speak—quickly, running words together, with varying emphasis and intonation.

  • Natural language generation (NLG) might be described as the opposite of speech recognition or speech-to-text: NLG is the task of putting structured information into conversational human language. Without NLG, computers would have little chance of passing the Turing test, where a computer tries to mimic a human conversation. Conversational agents such as Amazon’s Alexa and Apple’s Siri are already doing this well and assisting customers in real time.

  • Natural language understanding (NLU) is a subset of NLP that focuses on analyzing the meaning behind sentences. NLU enables software to find similar meanings in different sentences or to process words that have different meanings.

  • Sentiment analysis attempts to extract subjective qualities—attitudes, emotions, sarcasm, confusion or suspicion—from text. This is often used for routing communications to the system or the person most likely to make the next response.

See the blog post “NLP vs. NLU vs. NLG: the differences between three natural language processing concepts” for a deeper look into how these concepts relate.

Now available: IBM® watsonx.ai™

The all-new enterprise studio that brings together traditional machine learning along with new generative AI capabilities powered by foundation models.

NLP use cases across businesses

Organizations can use NLP to process communications that include email, SMS, audio, video, newsfeeds and social media. NLP is the driving force behind AI in many modern real-world applications. Here are a few examples:

  • Customer assistance: Enterprises can deploy chatbots or virtual assistants to quickly respond to custom questions and requests. When questions become too difficult for the chatbot or virtual assistant, the NLP system moves the customer over to a human customer service agent.

    Virtual agents such as IBM watsonx™ Assistant, Apple’s Siri and Amazon’s Alexa use speech recognition to recognize patterns in voice commands and natural language generation to respond with appropriate actions or helpful comments. Chatbots respond to typed text entries. The best chatbots also learn to recognize contextual clues about human requests and use them to provide even better responses or options over time.

    The next enhancement for these applications is question answering, the ability to respond to questions—anticipated or not—with relevant and helpful answers in their own words. These automations help reduce costs, save agents from spending time on redundant queries and improve customer satisfaction.

    Not all chatbots are powered by AI, but state-of-the-art chatbots increasingly use conversational AI techniques, including NLP, to understand user questions and automate responses to them.

  • FAQ: Not everyone wants to read to discover an answer. Fortunately, NLP can enhance FAQs: When the user asks a question, the NLP function looks for the best match among the available answers and brings that to the user’s screen. Many customer questions are of the who/what/when/where variety, so this function can save staff from having to repeatedly answer the same routine questions.

  • Grammar correction: The rules of grammar can be applied within word processing or other programs, where the NLP function is trained to spot incorrect grammar and suggest corrected wordings.

  • Machine translation: Google Translate is an example of widely available NLP technology at work. Truly useful machine translation involves more than replacing words from one language with words of another. Effective translation accurately captures the meaning and tone of the input language and translates it to text with the same meaning and desired impact in the output language.

    Machine translation tools are becoming more accurate. One way to test a machine translation tool is to translate text from one language and then back to the original. An oft-cited, classic example: Translating “The spirit is willing, but the flesh is weak” from English to Russian and back again once yielded, “The vodka is good, but the meat is rotten.” Recently, a closer result was “The spirit desires, but the flesh is weak.” Google translate can now take English to Russian to English and return the original, “The spirit is willing, but the flesh is weak."       
  • Redaction of personally identifiable information (PII): NLP models can be trained to quickly locate personal information in documents that might identify individuals. Industries that handle large volumes of sensitive information—financial, healthcare, insurance and legal firms—can quickly create versions with the PII removed.

  • Sentiment analysis: After being trained on industry-specific or business-specific language, an NLP model can quickly scan incoming text for keywords and phrases to gauge a customer’s mood in real-time as positive, neutral or negative. The mood of the incoming communication can help determine how it will be handled. And the incoming communication doesn’t have to be live: NLP can also be used to analyze customer feedback or call center recordings.

    Another option is an NLP API that can enable after-the-fact text analytics. NLP can uncover actionable data insights from social media posts, responses or reviews to extract attitudes and emotions in response to products, promotions and events. Information companies can use sentiment analysis in product designs, advertising campaigns and more.

  • Spam detection: Many people might not think of spam detection as an NLP solution, but the best spam detection technologies use NLP’s text classification capabilities to scan emails for language indicating spam or phishing. These indicators can include overuse of financial terms, characteristic bad grammar, threatening language, inappropriate urgency or misspelled company names.

  • Text generation: NLP helps put the “generative” into generative AI. NLP enables computers to generate text or speech that is natural-sounding and realistic enough to be mistaken for human communication. The generated language might be used to create initial drafts of blogs, computer code, letters, memos or tweets. With an enterprise-grade system, the quality of generated language might be sufficient to be used in real time for autocomplete functions, chatbots or virtual assistants.

    Advancements in NLP are powering the reasoning engine behind generative AI systems, driving further opportunities. Microsoft® Copilot is an AI assistant designed to boost employee productivity and creativity across day-to-day tasks and is already at work in tools used every day.  

  • Text summarization: Text summarization uses NLP techniques to digest huge volumes of digital text and create summaries and synopses for indexes, research databases, for busy readers who don't have time to read the full text. The best text summarization applications use semantic reasoning and natural language generation (NLG) to add useful context and conclusions to summaries.

NLP use cases by industry
  • Finance: In financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media.
  • Healthcare: New medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions.
  • Insurance: NLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing—leading to greater optimization of processing and employee efforts.

  • Legal: Almost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and helping ensure that all relevant details are captured for consideration.
Get started with NLP

Python and the Natural Language Toolkit (NLTK)

The Python programing language provides a wide range of tools and libraries for performing specific NLP tasks. Many of these NLP tools are in the Natural Language Toolkit, or NLTK, an open-source collection of libraries, programs and education resources for building NLP programs.

The NLTK includes libraries for many NLP tasks and subtasks, such as sentence parsing, word segmentation, stemming and lemmatization (methods of trimming words down to their roots), and tokenization (for breaking phrases, sentences, paragraphs and passages into tokens that help the computer better understand the text). It also includes libraries for implementing capabilities such as semantic reasoning: the ability to reach logical conclusions based on facts extracted from text.

Using NLTK, organizations can see the product of part-of-speech tagging. Tagging words might not seem to be complicated, but since words can have different meanings depending on where they are used, the process is complicated.

Generative AI platforms

Organizations can infuse the power of NLP into their digital solutions by leveraging user-friendly generative AI platforms such as IBM Watson NLP Library for Embed, a containerized library designed to empower IBM partners with greater AI capabilities. Developers can access and integrate it into their apps in their environment of their choice to create enterprise-ready solutions with robust AI models, extensive language coverage and scalable container orchestration.

More options include IBM® watsonx.ai™ AI studio, which enables multiple options to craft model configurations that support a range of NLP tasks including question answering, content generation and summarization, text classification and extraction. Integrations can also enable more NLP capabilities. For example, with watsonx and Hugging Face AI builders can use pretrained models to support a range of NLP tasks.

Related solutions
Watson natural language processing solutions

Accelerate the business value of artificial intelligence with a powerful and flexible portfolio of libraries, services and applications.

Explore natural language processing
Watson Natural Language Processing Library for Embed 

Infuse powerful natural language AI into commercial applications with a containerized library designed to empower IBM partners with greater flexibility.

Explore Watson Natural Language Processing Library for Embed
Resources Discover IBM's Granite LLM

Granite is IBM's flagship series of LLM foundation models based on decoder-only transformer architecture. Granite language models are trained on trusted enterprise data spanning internet, academic, code, legal and finance.

Free, hands-on learning for generative AI technologies

Learn the fundamental concepts for AI and generative AI, including prompt engineering, large language models and the best open source projects.

Natural Language Processing with Watson®

Learn about different NLP use cases in this NLP explainer.

Enhance your applications with IBM embeddable AI

Visit the IBM Developer's website to access blogs, articles, newsletters and more. Become an IBM partner and infuse IBM Watson embeddable AI in your commercial solutions today. Use IBM Watson NLP Library for Embed in your solutions.

Watson® understands the language of your business

Watch IBM Data and AI GM, Rob Thomas as he hosts NLP experts and clients, showcasing how NLP technologies are optimizing businesses across industries.

Getting started with Natural Language Understanding

Learn about the Natural Language Understanding API with example requests and links to additional resources.

Take the next step

Train, validate, tune and deploy generative AI, foundation models and machine learning capabilities with IBM watsonx.ai, a next-generation enterprise studio for AI builders. Build AI applications in a fraction of the time with a fraction of the data.

Explore watsonx.ai Book a live demo