Comparison of Text Representation Methods in Machine Learning

A detailed comparison of three text representation methods, Bag-of-Words, TF-IDF, and LLM-generated embeddings, was conducted to evaluate their effectiveness for classification and clustering in machine learning frameworks. The analysis utilizes the BBC news dataset, which consists of 2225 documents categorized into five types, to derive the text representations and build classifiers and clustering models.

The study focused on various machine learning classifiers trained on the different representations. The results indicated that TF-IDF combined with a support vector machine (SVM) classifier achieved the highest accuracy of 0.987. In contrast, LLM embeddings with SVM provided the quickest training time of 0.15 seconds. The best balance of performance and speed was found with logistic regression using TF-IDF, reaching an accuracy of 0.984 and a training time of 0.52 seconds.

The findings suggest that LLM embeddings, while advanced, may not always yield superior performance in scenarios with easily separable classes, such as the BBC news dataset. The simplicity and effectiveness of TF-IDF in capturing class patterns in this context were highlighted. Furthermore, in more complex datasets laden with noise or variability, LLM embeddings could potentially outperform other methods.

In the second comparison, k-means clustering was applied with k set to 5, revealing that LLM embeddings excelled, achieving an adjusted Rand index (ARI) score of 0.899. This result underscores the importance of semantic understanding in unsupervised learning tasks like clustering, where capturing patterns is crucial.

Overall, the analysis concludes with a recommendation for practitioners to consider starting with simpler methods like TF-IDF before exploring advanced techniques, especially when working with datasets that exhibit clear class boundaries and linear separability.