Decision Trees Applied to Text Classification

A tutorial on decision trees focused on text classification was published on August 20, 2025. The article explores how decision tree-based models can effectively handle unstructured data, including text and images, in addition to their usual tasks involving structured data.

The tutorial employs the UCI dataset for spam classification, which contains text-label pairs that classify emails as spam or ham. According to the article, there are 4,825 ham emails, representing 86%, and 747 spam emails, accounting for 14%. This illustrates a class-imbalanced dataset, which poses challenges for accurate evaluation metrics such as accuracy.

The methodology described involves splitting the dataset into training and testing subsets using stratified sampling to retain class proportions. The initial decision tree model is trained using TF-IDF vectorization, converting text data into structured numerical vectors. The results indicate that while the model performs reasonably well, it is somewhat influenced by the prevalence of the ham class. The spam class has a recall of 0.83, suggesting a need for cautious attention to avoid misclassifying important messages.

A second decision tree model uses embeddings for text representation. These embeddings are generated using pretrained models like GloVe, which map words to dense vectors, allowing the model to capture semantic meaning. However, this approach can lead to representation loss, which affects performance compared to the TF-IDF model.

Finally, the article compares the decision tree models with a Naive Bayes classifier. The Naive Bayes model achieves perfect precision of 1.00 for spam classification but has a recall of 0.70, indicating it misses a significant portion of actual spam emails. The tutorial concludes that for optimal performance, particularly in recall, further optimization of the decision tree model might be beneficial.