Seven Feature Engineering Tricks for Text Data

An increasing number of artificial intelligence and machine learning-based systems rely on text data, with language models being a prominent example. However, machines do not truly comprehend language; instead, they process numbers. Consequently, feature engineering steps are typically essential to convert raw text data into useful numeric data features that these systems can analyze and make inferences from.

This article presents seven straightforward tricks for feature engineering on text data. Depending on the specific model's complexity and requirements, a more or less ambitious set of these tricks may be necessary.

One effective technique is stopword removal, which helps reduce dimensionality, crucial for certain models that may experience the curse of dimensionality. Common words that primarily add noise, such as articles, prepositions, and auxiliary verbs, are eliminated, leaving only those that convey the majority of the semantics in the source text.

Another method involves reducing words to their root forms, which can unify variants, such as different verb tenses, into a single feature. In deep learning models based on text embeddings, this step is usually unnecessary. However, when data availability is extremely limited, it can still help ease sparsity and encourage the model to focus on core meanings rather than redundant representations.

The Bag of Words approach is one of the simplest methods to transform text into numerical features in classical machine learning. It encodes word frequency into vectors, resulting in a two-dimensional array of word counts that describe basic features. However, it is limited as it does not capture vital aspects of language understanding, such as word order and semantic relationships.

Term Frequency, Inverse Document Frequency (TF-IDF) enhances the Bag of Words approach by considering word frequency and overall relevance at both the document and dataset levels. Unique and important words receive a higher weight based on their frequency across texts.

Sentence-based n-grams capture interactions between words, allowing for the incorporation of both unigrams and bigrams. The basic tokenization approach involves removing punctuation and casing, producing a list of clean word tokens. Additionally, word embeddings serve as a powerful method to convert text into machine-readable information, effectively capturing semantics through numerically similar vectors.