Clustering Unstructured Text with LLM Embeddings and HDBSCAN

A new article explains how to construct a text clustering pipeline by combining embeddings from large language models with HDBSCAN, a density-based clustering algorithm, to automatically discover topics in unlabeled text data.

The article emphasizes that while Generative AI often focuses on chat interfaces, large language models, or LLMs, have a broader range of applications. One of their most potent abilities is converting unstructured text into semantically rich mathematical representations known as embeddings. These representations can be utilized for various machine learning tasks, including clustering.

By integrating embeddings with advanced clustering techniques like HDBSCAN, users can uncover hidden topics, patterns, or categories within collections of text documents without prior labeling. The article provides a step-by-step guide to building a text-based clustering pipeline, starting with the installation of essential Python libraries.

It outlines the process of fetching a dataset of categorized news articles while intentionally omitting labels to focus on clustering based on similarity. The article also details how to obtain embeddings from the raw text using the all-MiniLM-L6-v2 model from Hugging Face’s sentence-transformers library and how to reduce the embedding dimensions using the UMAP algorithm.

Upon applying the HDBSCAN algorithm, the article notes that the results can vary based on hyperparameter settings and encourages experimentation with configurations. Initial findings indicate that HDBSCAN detected two clusters in the sample of 150 news articles, suggesting that the articles are easily separable by topic.

The article concludes by highlighting the benefits of combining LLM embeddings with HDBSCAN, such as capturing the semantic meaning of the original text and automatically determining the optimal number of clusters while identifying outliers that may distort statistical analysis.