A natural language processing system can work rapidly and efficiently: after NLP models are properly trained, it can take on administrative tasks, freeing staff for more productive work. Benefits can include:

Faster insight discovery: Organizations can find hidden patterns, trends and relationships between different pieces of content. Text data retrieval supports deeper insights and analysis, enabling better-informed decision-making and surfacing new business ideas.

Greater budget savings: With the massive volume of unstructured text data available, NLP can be used to automate the gathering, processing and organization of information with less manual effort.

Quick access to corporate data: An enterprise can build a knowledge base of organizational information to be efficiently accessed with AI search. For sales representatives, NLP can help quickly return relevant information, to improve customer service and help close sales.

NLP models are not perfect and probably never will be, just as human speech is prone to error. Risks might include:

Biased training:  As with any AI function, biased data used in training will skew the answers. The more diverse the users of an NLP function, the more significant this risk becomes, such as in government services, healthcare and HR interactions. Training datasets scraped from the web, for example, are prone to bias.

Misinterpretation: As in programming, there is a risk of garbage in, garbage out (GIGO). NLP solutions might become confused if spoken input is in an obscure dialect, mumbled, too full of slang, homonyms, incorrect grammar, idioms, fragments, mispronunciations, contractions or recorded with too much background noise.

New vocabulary: New words are continually being invented or imported. The conventions of grammar can evolve or be intentionally broken. In these cases, NLP can either make a best guess or admit it’s unsure—and either way, this creates a complication.

Tone of voice: When people speak, their verbal delivery or even body language can give an entirely different meaning than the words alone. Exaggeration for effect, stressing words for importance or sarcasm can be confused by NLP, making the semantic analysis more difficult and less reliable.

Human language is filled with many ambiguities that make it difficult for programmers to write software that accurately determines the intended meaning of text or voice data. Human language might take years for humans to learn—and many never stop learning. But then programmers must teach natural language-driven applications to recognize and understand irregularities so their applications can be accurate and useful.

NLP combines the power of computational linguistics together with machine learning algorithms and deep learning. Computational linguistics is a discipline of linguistics that uses data science to analyze language and speech. It includes two main types of analysis: syntactical analysis and semantical analysis. Syntactical analysis determines the meaning of a word, phrase or sentence by parsing the syntax of the words and applying preprogrammed rules of grammar. Semantical analysis uses the syntactic output to draw meaning from the words and interpret their meaning within the sentence structure. 

The parsing of words can take one of two forms. Dependency parsing looks at the relationships between words, such as identifying nouns and verbs, while constituency parsing then builds a parse tree (or syntax tree): a rooted and ordered representation of the syntactic structure of the sentence or string of words. The resulting parse trees underly the functions of language translators and speech recognition. Ideally, this analysis makes the output—either text or speech—understandable to both NLP models and people.

Self-supervised learning (SSL) in particular is useful for supporting NLP because NLP requires large amounts of labeled data to train state-of-the-art artificial intelligence (AI) models. Because these labeled datasets require time-consuming annotation—a process involving manual labeling by humans—gathering sufficient data can be prohibitively difficult. Self-supervised approaches can be more time-effective and cost-effective, as they replace some or all manually labeled training data.

Three different approaches to NLP include:

Rules-based NLP: The earliest NLP applications were simple if-then decision trees, requiring preprogrammed rules. They are only able to provide answers in response to specific prompts, such as the original version of Moviefone. Because there is no machine learning or AI capability in rules-based NLP, this function is highly limited and not scalable.

Statistical NLP: Developed later, statistical NLP automatically extracts, classifies and labels elements of text and voice data, and then assigns a statistical likelihood to each possible meaning of those elements. This relies on machine learning, enabling a sophisticated breakdown of linguistics such as part-of-speech tagging.

Statistical NLP introduced the essential technique of mapping language elements—such as words and grammatical rules—to a vector representation so that language can be modeled by using mathematical (statistical) methods, including regression or Markov models. This informed early NLP developments such as spellcheckers and T9 texting (Text on 9 keys, to be used on Touch-Tone telephones).

Deep learning NLP: Recently, deep learning models have become the dominant mode of NLP, by using huge volumes of raw, unstructured data—both text and voice—to become ever more accurate. Deep learning can be viewed as a further evolution of statistical NLP, with the difference that it uses neural network models. There are several subcategories of models:

• Sequence-to-Sequence (seq2seq) models: Based on recurrent neural networks (RNN), they have mostly been used for machine translation by converting a phrase from one domain (such as the German language) into the phrase of another domain (such as English).
  
  
• Transformer models: They use tokenization of language (the position of each token—words or subwords) and self-attention (capturing dependencies and relationships) to calculate the relation of different language parts to one another. Transformer models can be efficiently trained by using self-supervised learning on massive text databases. A landmark in transformer models was Google’s bidirectional encoder representations from transformers (BERT), which became and remains the basis of how Google’s search engine works.
  
  
• Autoregressive models: This type of transformer model is trained specifically to predict the next word in a sequence, which represents a huge leap forward in the ability to generate text. Examples of autoregressive LLMs include GPT, Llama, Claude and the open-source Mistral.
  
  
• Foundation models: Prebuilt and curated foundation models can speed the launching of an NLP effort and boost trust in its operation. For example, the IBM Granite™ foundation models are widely applicable across industries. They support NLP tasks including content generation and insight extraction. Additionally, they facilitate retrieval-augmented generation, a framework for improving the quality of response by linking the model to external sources of knowledge. The models also perform named entity recognition which involves identifying and extracting key information in a text.
  

For a deeper dive into the nuances between multiple technologies and their learning approaches, see “AI versus. machine learning versus deep learning versus neural networks: What’s the difference?”

Organizations can use NLP to process communications that include email, SMS, audio, video, newsfeeds and social media. NLP is the driving force behind AI in many modern real-world applications. Here are a few examples:

• Customer assistance: Enterprises can deploy chatbots or virtual assistants to quickly respond to custom questions and requests. When questions become too difficult for the chatbot or virtual assistant, the NLP system moves the customer over to a human customer service agent.
  
  Virtual agents such as IBM watsonx™ Assistant, Apple’s Siri and Amazon’s Alexa use speech recognition to recognize patterns in voice commands and natural language generation to respond with appropriate actions or helpful comments. Chatbots respond to typed text entries. The best chatbots also learn to recognize contextual clues about human requests and use them to provide even better responses or options over time.
  
  The next enhancement for these applications is question answering, the ability to respond to questions—anticipated or not—with relevant and helpful answers in their own words. These automations help reduce costs, save agents from spending time on redundant queries and improve customer satisfaction.
  
  Not all chatbots are powered by AI, but state-of-the-art chatbots increasingly use conversational AI techniques, including NLP, to understand user questions and automate responses to them.
  
  
• FAQ: Not everyone wants to read to discover an answer. Fortunately, NLP can enhance FAQs: When the user asks a question, the NLP function looks for the best match among the available answers and brings that to the user’s screen. Many customer questions are of the who/what/when/where variety, so this function can save staff from having to repeatedly answer the same routine questions.
  
  
• Grammar correction: The rules of grammar can be applied within word processing or other programs, where the NLP function is trained to spot incorrect grammar and suggest corrected wordings.
  
  
• Machine translation: Google Translate is an example of widely available NLP technology at work. Truly useful machine translation involves more than replacing words from one language with words of another. Effective translation accurately captures the meaning and tone of the input language and translates it to text with the same meaning and desired impact in the output language.
  
  Machine translation tools are becoming more accurate. One way to test a machine translation tool is to translate text from one language and then back to the original. An oft-cited, classic example: Translating “The spirit is willing, but the flesh is weak” from English to Russian and back again once yielded, “The vodka is good, but the meat is rotten.” Recently, a closer result was “The spirit desires, but the flesh is weak.” Google translate can now take English to Russian to English and return the original, “The spirit is willing, but the flesh is weak."       

• Redaction of personally identifiable information (PII): NLP models can be trained to quickly locate personal information in documents that might identify individuals. Industries that handle large volumes of sensitive information—financial, healthcare, insurance and legal firms—can quickly create versions with the PII removed.
  
  
• Sentiment analysis: After being trained on industry-specific or business-specific language, an NLP model can quickly scan incoming text for keywords and phrases to gauge a customer’s mood in real-time as positive, neutral or negative. The mood of the incoming communication can help determine how it will be handled. And the incoming communication doesn’t have to be live: NLP can also be used to analyze customer feedback or call center recordings.
  
  Another option is an NLP API that can enable after-the-fact text analytics. NLP can uncover actionable data insights from social media posts, responses or reviews to extract attitudes and emotions in response to products, promotions and events. Information companies can use sentiment analysis in product designs, advertising campaigns and more.
  
  
• Spam detection: Many people might not think of spam detection as an NLP solution, but the best spam detection technologies use NLP’s text classification capabilities to scan emails for language indicating spam or phishing. These indicators can include overuse of financial terms, characteristic bad grammar, threatening language, inappropriate urgency or misspelled company names.
  
  
• Text generation: NLP helps put the “generative” into generative AI. NLP enables computers to generate text or speech that is natural-sounding and realistic enough to be mistaken for human communication. The generated language might be used to create initial drafts of blogs, computer code, letters, memos or tweets. With an enterprise-grade system, the quality of generated language might be sufficient to be used in real time for autocomplete functions, chatbots or virtual assistants.
  
  Advancements in NLP are powering the reasoning engine behind generative AI systems, driving further opportunities. Microsoft® Copilot is an AI assistant designed to boost employee productivity and creativity across day-to-day tasks and is already at work in tools used every day.  
  
  
• Text summarization: Text summarization uses NLP techniques to digest huge volumes of digital text and create summaries and synopses for indexes, research databases, for busy readers who don't have time to read the full text. The best text summarization applications use semantic reasoning and natural language generation (NLG) to add useful context and conclusions to summaries.
  

• Finance: In financial dealings, nanoseconds might make the difference between success and failure when accessing data, or making trades or deals. NLP can speed the mining of information from financial statements, annual and regulatory reports, news releases or even social media.
  

• Healthcare: New medical insights and breakthroughs can arrive faster than many healthcare professionals can keep up. NLP and AI-based tools can help speed the analysis of health records and medical research papers, making better-informed medical decisions possible, or assisting in the detection or even prevention of medical conditions.
  

• Insurance: NLP can analyze claims to look for patterns that can identify areas of concern and find inefficiencies in claims processing—leading to greater optimization of processing and employee efforts.
  
  
• Legal: Almost any legal case might require reviewing mounds of paperwork, background information and legal precedent. NLP can help automate legal discovery, assisting in the organization of information, speeding review and helping ensure that all relevant details are captured for consideration.

Python and the Natural Language Toolkit (NLTK)

The Python programing language provides a wide range of tools and libraries for performing specific NLP tasks. Many of these NLP tools are in the Natural Language Toolkit, or NLTK, an open-source collection of libraries, programs and education resources for building NLP programs.

The NLTK includes libraries for many NLP tasks and subtasks, such as sentence parsing, word segmentation, stemming and lemmatization (methods of trimming words down to their roots), and tokenization (for breaking phrases, sentences, paragraphs and passages into tokens that help the computer better understand the text). It also includes libraries for implementing capabilities such as semantic reasoning: the ability to reach logical conclusions based on facts extracted from text.

Using NLTK, organizations can see the product of part-of-speech tagging. Tagging words might not seem to be complicated, but since words can have different meanings depending on where they are used, the process is complicated.

Generative AI platforms

Organizations can infuse the power of NLP into their digital solutions by leveraging user-friendly generative AI platforms such as IBM Watson NLP Library for Embed, a containerized library designed to empower IBM partners with greater AI capabilities. Developers can access and integrate it into their apps in their environment of their choice to create enterprise-ready solutions with robust AI models, extensive language coverage and scalable container orchestration.

More options include IBM^® watsonx.ai™ AI studio, which enables multiple options to craft model configurations that support a range of NLP tasks including question answering, content generation and summarization, text classification and extraction. Integrations can also enable more NLP capabilities. For example, with watsonx and Hugging Face AI builders can use pretrained models to support a range of NLP tasks.