Can LLM Embeddings Enhance Time Series Forecasting?

In a recent article, the effectiveness of incorporating large language model embeddings into time series forecasting has been examined. The article explores whether using LLM-generated embeddings as engineered features can lead to significant improvements in forecasting performance.

The discussion begins with the growing trend of utilizing large language models (LLMs) for various machine learning tasks, including predictive modeling. This raises the pivotal question of whether LLMs, when leveraged correctly, can enhance the performance of time series forecasting models.

To investigate this, the article provides a practical example involving the creation of two datasets for training forecasting models. One dataset includes only time series-related features, while the other integrates LLM-generated embeddings from a separate yet causally related dataset. The process includes importing necessary modules and executing feature engineering techniques to capture meaningful patterns in the data.

As part of the analysis, the article details the use of a pre-trained sentence transformer model to generate embeddings from combined news headlines. To mitigate overfitting and dimensionality issues, principal component analysis (PCA) is employed.

The results indicate that the accuracy of the model utilizing both time series features and embedding-based features is marginally higher than that of the baseline model, with full model accuracy reported at 0.5047619047619047. However, the differences in performance are deemed inconclusive.

The article concludes that while LLM embeddings may offer potential benefits in certain forecasting scenarios, they are not universally superior to traditional forecasting methods based solely on temporal features. Rigorous testing and validation are recommended to ascertain the true value of embedding-augmented forecasting models across various contexts.