How to summarize text with Python NLP and extractive text summarization

Modern transformer model architectures based on neural networks dominate many NLP tasks. This tutorial focuses on classical approaches that remain valuable in data science workflows where interpretability, limited dependencies and predictable summary length matter. These methods are often used to automate the generation of concise summaries from a large corpus without requiring a labeled dataset.

By the end of this tutorial you’ll understand:

• How frequency-based, graph-based and semantic summarization algorithms work
• How the strengths and limitations of each approach manifest
• How to implement these algorithms in Python with the Sumy library
• When to choose extractive versus abstractive text summarization for your projects

Text summarization can be broadly categorized into two approaches:

1. Extractive summarization selects and combines existing sentences directly from the source text to create a summary. Think of it like highlighting the most important sentences in a document. This tutorial focuses on extractive methods, which dominated the field for decades and remain valuable for their interpretability and reliability.
2. Abstractive summarization generates new sentences to convey the original meaning, similar to how we might paraphrase or rewrite key points. Modern large language models (LLMs) like generative pre-trained transformer (GPT), Granite and Claude excel at this approach, although extractive methods still offer advantages in transparency and computational efficiency. Abstractive systems typically rely on transformers pre-trained on large-scale data, sometimes fine-tuned for specific summarization tasks or machine translation objectives.

Automatic text summarization began in 1958 with Hans Peter Luhn, an IBM researcher who published “The automatic creation of literature and abstracts”. Luhn’s algorithm was groundbreaking in its simplicity: determine sentence importance by counting the frequency of meaningful words. Though basic by today’s standards, this frequency-based approach established the foundation for subsequent work in the field.

Luhn’s statistical method had clear limitations—it couldn’t capture semantic relationships, context or nuance in language. Over the following decades, researchers expanded on his work by incorporating:

• Graph-based methods like LexRank, which identify important sentences by analyzing similarity patterns across the entire document.
• Semantic approaches like LSA, which uncover hidden thematic structures with linear algebra to understand meaning beyond surface-level word matching.

Understanding these algorithms illuminates fundamental concepts in information retrieval (IR) and natural language processing (NLP), while showing the field’s evolution from rule-based systems to sophisticated deep-learning models used today. Today, these models are commonly accessed through platforms like Hugging Face, exposed through an API and powered by frameworks such as PyTorch.

The following section provides a step-by-step walkthrough for implementing classic extractive text summarization algorithms in Python.

To run this project, clone the GitHub repository by using GitHub as the HTTPS URL. For detailed steps on how to clone a repository, refer to the GitHub documentation. You can find this specific tutorial inside the ibmdotcom-tutorials repo under the generative AI directory.

This tutorial uses a Jupyter Notebook to demonstrate text summarization with Python through the Sumy, a lightweight Python library rather than a large-scale artificial intelligence system. Jupyter Notebooks are versatile tools that allow you to combine code, text and visualization in a single environment. You can run this notebook in your local IDE or explore cloud-based options like watsonx.ai Runtime, which provides a managed environment for running Jupyter Notebooks.

Whether you choose to run the notebook locally or in the cloud, the steps and code remain the same. Simply ensure that the required Python libraries are installed in your environment.

The following Python code installs the required packages and prepares the environment for running extractive summarization techniques.

As previously mentioned, the Luhn algorithm is a statistical, frequency-based approach to extractive summarization. Luhn’s algorithm works on the premise that the most important sentences in a document are those sentences that contain the most significant words. This approach makes Luhn effective for quickly extracting salient sentences without semantic modeling. The algorithm determines significant words by how frequently (but not too frequently) they occur.

Luhn algorithm workflow

1. Preprocessing: Stop words are common words that appear frequently in language (like “a”, “this”, “is”, “the”) but carry little meaningful information. These words are filtered out of the text data. A technique called stemming is applied to reduce words to their root forms (“running”, “runs”, “ran” -> “run”).
2. Word scoring: The frequency of each word is calculated. Words that appear with moderate to high frequency are considered significant, while common words (including stop words) and rare words are given less weight.
3. Sentence scoring: Each sentence is scored based on clusters of significant words. A sentence’s score is based on the density of significant words within these clusters, calculated as the ratio of significant words to total words in the cluster.
4. Summary generation: The top-scoring sentences are selected and presented in their original order to create a summary.

Try it out yourself by running the following codeblock:

Here is an example of the expected output (you can get different summarization results depending on factors like library versions, input formatting and tokenization):

LexRank is an extractive summarization algorithm that applies the concept of graph-based ranking to text summarization techniques focused on sentence centrality. It ranks sentences based on their similarity to other sentences.

LexRank algorithm workflow

1. Generate a similarity graph: Each sentence is represented as a node in a graph. The similarity is calculated between every pair of sentences (typically through cosine similarity on TF-IDF vectors). Sentences are connected with edges weighted by similarity scores.
2. Compute sentence centrality: Importance scores are calculated for each sentence. The process uses iterative voting inspired by Google’s PageRank algorithm. Each sentence begins with an equal score. In each iteration, a sentence’s score is updated based on the scores of the sentences it is connected to. Sentences that are similar to many highly scored sentences will themselves receive higher scores. This process repeats until the score stabilizes, with a dampening factor ensuring convergence. The result is a reinforcement effect where sentences discussing central themes naturally accumulate higher scores.
3. Select the top sentences: The highest-scoring sentences are extracted to form a summary, presented in their original document order.

Try the LexRank summarization as follows:

Here is the example summarization result with LexRank:

You might notice that the URL summary is not as complete as the previous algorithm. Sometimes, LexRank produces short or unusual summaries when summarizing content from a URL. This issue happens because LexRank relies on comparing sentence similarity and webpages often contain short headings or fragmented text that don’t provide enough context for the algorithm to rank sentences meaningfully.

In contrast, Luhn looks at word frequency, so it can still pick out the most important sentences even in sparse or messy text. This example illustrates that while LexRank is powerful for well-structured documents, it’s not always the best choice for web scraping or heading-heavy content.

Latent semantic analysis (LSA) is a technique that extracts hidden conceptual meaning from a text. LSA identifies the core concepts in a document and selects sentences that best represent those concepts.

LSA algorithm workflow

1. Create a term-sentence matrix: The algorithm begins by building a matrix where rows represent words, columns represent sentences, and each cell contains a word’s frequency (or TF-IDF weight) in that sentence. Before sentence construction, documents undergo truncation to control size and reduce noise.
2.  Apply singular value decomposition (SVD): Decompose this matrix in three matrices to capture the underlying semantic structure: U (words to topics), Σ (relative topic strength), Vᵀ (topics to sentences). SVD identifies the most important “topics” or “concepts” in the document by finding patterns in how words co-occur across sentences.
3. Score sentences: A calculation is made that represents the most important concepts identified by SVD for each sentence. Sentences with strong representation across top concepts receive higher scores.
4. Generate summary: Top-scoring sentences are selected and presented in their original order.

Try out LSA yourself here:

Here are example summarization results use LSA:

LSA differs from Luhn and LexRank because it focuses on the underlying concepts or topics in a text rather than just word frequency or sentence similarity. Luhn is great when you want a broad summary based on important keywords, and LexRank works well for well-structured text where sentence relationships matter.

However, LSA is ideal when you want a coherent, concept-focused summary, especially for longer documents with multiple paragraphs. It can highlight the main ideas without getting distracted by repeated keywords or short headings. In short, choose LSA when understanding the key themes is more important than capturing every high-frequency term.

In this tutorial, you explored three classic extractive summarization algorithms—Luhn, LexRank and LSA—and learned how they approach the task in different ways. Luhn focuses on word frequency, LexRank uses sentence similarity and LSA identifies underlying concepts to select the most meaningful sentences.

Each method has its strengths: Luhn works well for general keyword-based summaries. LexRank is effective for structured text with clear sentence relationships. LSA shines when you want a coherent, concept-focused overview of longer documents.

Understanding these approaches gives you the foundation to choose the right extractive summarization technique for your projects and shows how the field has evolved from simple rule-based methods to sophisticated semantic analysis. These classic approaches also serve as strong baselines when evaluating modern abstractive systems or designing hybrid pipelines for real-world use cases.