What is topic modeling?

Topic models specifically identify common keywords or phrases in a text dataset and group those words under a number of topics. They aim to uncover the latent topics or themes characterizing a set of documents. In this way, topic models are a machine learning-based form of text analysis used to thematically annotate large text corpora.²

Users can readily generate topic models using scikit-learn’s Natural Language Toolkit (NLTK) and gensim in Python.

As an unsupervised learning method, topic models do not require user-generated labels of training data, as in supervised text classification tasks. Rather, topic models generate, and by extension annotate, large collections of document with thematic information in the form of word groups known as topics.³ But how do topic models produce these groups of words?

Topic modeling essentially treats each individual document in a collection of texts as a bag of words model. This means that the topic modeling algorithm ignores word order and context, simply focusing on how often words occur, and how often they co-occur, within each individual document.⁴

Most topic modeling approaches begin by generating a document-term matrix. This matrix models the text dataset with documents as rows and individual words as columns, or vice-versa. Values in the matrix indicate the frequency with which a given word appears in each document. This matrix can then be used to generate a vector space, where n words equals n dimensions. A given row’s value indicates that document’s position in the vector space. Documents that use words in similar groups and with comparable frequency will thus reside closer together in vector space. From here, topic models treat proximity in vector space as documents sharing similar conceptual content or topics.⁵

Topic models are not synonymous with bag of words however. While the latter merely counts the presence of words within a collection of documents, topic models group commonly co-occurring words into sets of topics. Each topic is modeled as a probability distribution across a vocabulary of words. Each document in the collection is then represented in terms of those topics.⁶ In this way, topic models essentially attempt to reverse engineer the discourses (that is, topics) that produced the documents in question.⁷

Topic modeling algorithms are not so much alternative methods to one task as they are sequential developments meant to resolve issues initially found in bag of words models. Term frequency-inverse document frequency (TF-IDF) is a modification of bag of words intended to address the issues resulting from common yet semantically irrelevant words by accounting for each word’s prevalence throughout every document in a text set. Latent semantic analysis builds on TF-IDF with the principal intent of addressing polysemy and synonymy. This gave birth to probabilistic latent semantic analysis, from which grew latent Dirichlet allocation. This latter’s distinguishing characteristic is that all documents in a collection share the same set of topics, albeit in different proportions.⁸

Latent semantic analysis (LSA) (also called latent semantic indexing) deploys a technique known as singular value decomposition in order to reduce sparsity in the document-term matrix. This alleviates problems resulting from polysemy and synonymy—that is, single words with multiple meanings or multiple words with a single shared meaning.

Data sparsity essentially denotes when a majority of data values in a given dataset are null (that is, empty). This happens regularly when constructing document-term matrices, for which each individual word is a separate row and vector space dimension, as documents will regularly lack a majority of the words that may be more frequent in other documents. Of course, text data preprocessing techniques, such as stopword removal or stemming and lemmatization, can help reduce the size of the matrix. LSA offers a more targeted approach for reducing sparsity and dimensionality.

LSA begins with the document-term matrix, which displays the number of times each word appears in each document. From here, LSA produces a document-document matrix and term-term matrix. If the document-term matrix dimensions are defined as d documents times w words, then the document-document matrix is d times d and the term-term matrix w times w. Each value in the document-document matrix indicates the number of words each document has in common. Each value in the term-term matrix indicates the number of documents in which two term co-occur.⁹

Using these two additional matrices, the LSA algorithm conducts singular value decomposition on the initial document-term matrix, producing new special matrices of eigenvectors. These special matrices breakdown the original document-term relationships into linearly independent factors. Because many of these factors are near-zero, they are treated as zero and thrown out of the matrices. This reduces the model’s dimensions.¹⁰

Once model dimensions have been reduced through singular value decomposition, the LSA algorithm compares documents in the lower dimensional space using cosine similarity. Cosine similarity signifies the measurement of the angle between two vectors in vector space. It may be any value between -1 and 1. The higher the cosine score, the more alike two documents are considered. Cosine similarity is represented by this formula, where x and y signify two item-vectors in the vector space:¹¹

Latent Dirichlet allocation (LDA) — not to be confused with linear discriminant analysis — is a probabilistic topic modeling algorithm. This means it generates topics, classifying words and documents among these topics, according to probability distributions. Using the document-term matrix, the LDA algorithm generates topic distributions (that is, lists of keywords with respective probabilities) according to word frequency and co-occurrences. This assumption is that words that occur together are likely a part of similar topics. The algorithm assigns document-topic distributions based on the clusters of words that appear in the given document.¹²

For example, say we generate a LDA model for a collection of news articles that has the following partial output:

Here, we have two topics that may likely be described as immigration (Topic 1) and astronomy (Topic 2). The scores attached to each word are the probability of that keyword appearing in its given topic. The probabilities attached to each document are that document’s respective probabilities of belonging to a mixture of topics given the distribution and co-occurrence of words from each topic within that document. For example, the table’s first row lists border under Topic 1 with a 40% probability and space in Topic 2 with a 60% probability. These percentages indicate the probability of their respective terms occurring in that topic across the whole corpus. The first document row reads Document 1: Topic 1: .95, Topic 2: .05. This means that, based the occurrence of words in Document 1, the model projects Document 1 as being 95% derived from Topic 1 and 5% derived from Topic 2. In other words, our hypothetical LDA model assumes these are the topics and proportions of those topics used to generate the model.

Of course, polysemous words in particular create problems for such discrete categorizations — for example, alien may refer to a human immigrant or an extra-terrestrial creature. If our algorithm encounters alien in a document, how does it determine to which topic the word (and by extension, the document) belongs?

When assigning topics to words, the LDA algorithm uses what is known as Gibbs sampling. The Gibbs sampling formula is:

Understanding this equation’s exact operations and hyperparameters requires foundational knowledge in statistics and Markov Chain Monte Carlo techniques (the latter often employed in reinforcement learning). Nevertheless, we can summarize the equation’s principal components:

• The first ratio expresses the probability of topic t in document d. The algorithm calculates this probability according to the number of words in document d that belong to topic t. This essentially asks: how prevalent is topic t in document d?
• The second ratio expresses the probability of word w belonging to topic t. The algorithm calculates this probability by enumerating the occurrences of w in t over all word-tokens in t. This asks: with what frequency does word w appear in topic t throughout the rest of the corpus?

Note that Gibbs sampling is an iterative process. That is, a word is not sampled once, assigned a topic, and tossed aside. Rather, Gibbs sampling passes each word through multiple iterations, updating topic-word probabilities in light of one another.¹³

There are many use cases for topic models, from literary criticism¹⁴ to bioinformatics¹⁵ to hate speech detection in social media.¹⁶ As with many NLP tasks, a significant proportion of topic modeling research through the years concerns English and other Latin-script languages. More recently, however, research has explored topic modeling approaches for Arabic and other non-Latin languages.¹⁷

Ongoing research also addresses evaluation metrics for topic models. Indeed, there is no one metrics used to evaluate topic models. Past evaluation metrics have adopted qualitative and quantitative approaches. The former requires significant domain-specific knowledge to evaluate topic mode key terms for interpretability.¹⁸ Quantitative measures consist of log-likelihood and coherence scores, which aim to measure the likelihood and cohesion of topics within a model.¹⁹ A wide body of research argues, however, such quantitative metrics may be unreliable.²⁰

In attempt to resolve issues related to topic model evaluation, one study investigates artificial intelligence applications, notably large language models (LLMs), as a means of designing and evaluating LDA models for specific research objectives. LLMs, the study argues, can help resolve longstanding problems in topic modeling, namely, how to determine and evaluate the appropriate number of topics.²¹ Other studies also turn to LLM applications as a means to address the evaluation gap in topic modeling.²²