Applying different deep learning architectures in time series data.
This project uses different deep learning architectures to forecast multiple time series datasets. In univariate time series datasets, general statistical forecasting shows better performance, but in multivariate time series datasets, different deep learning architectures can outperform statistical learning models like the ARIMA model. In today's world, there are many multivariate time series being generated continuously, such as health data in Fitbit or Apple Watch and electrical consumption under a powerhouse. Therefore, the model should be independent of seasonality, trend, and cyclability so that it can learn these patterns directly from the data. This independence increases the scalability of the model for multiple time series datasets. The motivation behind this project is to apply different probabilistic deep learning models to time series datasets. Time series datasets are suitable for statistical models like ARIMA and ETS, but their scalability is limited because seasonality and trend must be tracked manually. Besides, researchers are currently trying to integrate solutions to different problems using deep learning. Deep learning is easy to integrate with other AI models via plug-ins and pre-trained models. It can reveal hidden structure in the data. Because of open-source models, pre-trained models, and publicly available architectures from other researchers, it is easier to evaluate models and fine-tune them according to the specific problem.
Deep learning is used extensively in Natural Language Processing, image processing, and recommendation systems. It is not yet as popular in time series, but it has strong capabilities for capturing multiple streams of time series data, which is difficult for traditional algorithms like ARIMA or AR. In 2022, Makridakis et al. evaluated how well different forecasting methods work (Fig-1). They selected the two best statistical methods (ETS and ARIMA) and combined them into a method called Ensemble-S. They also selected the two best machine learning methods (MLP and BNN), added four deep learning methods (DeepAR, Feed-forward, Transformer, WaveNet), and combined them into Ensemble-DL. They evaluated these methods on a large benchmark, using two accuracy measures (sMAPE and MASE), as well as computational time (CT) and relative computational complexity (RCC) [1]. Their results showed that ETS and ARIMA were generally more accurate than the machine learning methods and most deep learning methods across all forecasting horizons. Both ensemble methods (Ensemble-S and Ensemble-DL) performed better than any individual method. Ensemble-DL was slightly better overall, but Ensemble-S was best in terms of MASE for short-term forecasting. Machine learning methods were also better than deep learning methods for near-term predictions. The researchers also found that models trained for one-step-ahead forecasting tend to be more accurate in the near future, whereas models trained for multi-step-ahead forecasting perform better for longer horizons. Statistical methods like ETS and ARIMA are usually configured for one-step-ahead forecasting, while machine learning methods like MLP and BNN are often configured for multi-step forecasting.
Traditional statistical algorithms like Exponential Smoothing and ARIMA have been staples in time series forecasting. These methods, while robust, are typically applied to individual series and require manual intervention to adjust for trends, seasonality, and autocorrelation. This one-size-fits-one approach struggles to scale to today’s complex, interconnected datasets—consider the vast array of product sales at Amazon or user engagement metrics at Instagram. Deep learning (DL), in contrast, is a contemporary approach that makes minimal assumptions and learns from intricate relationships within large, intertwined datasets. However, deep learning is not free of inductive bias; these biases are introduced by architectural choices. Such biases can be beneficial, as in Convolutional Neural Networks (CNNs), which excel in image recognition due to their spatial inductive bias. The selection of a deep learning model is thus a strategic decision that must align with the application domain. With ongoing advances, a wide range of DL models has emerged, each with unique inductive biases tailored to different forecasting challenges.
This type of model can integrate parallel time series streams, such as Apple Watch heart rate, daily activity, sleep patterns, blood oxygen levels, electrocardiogram, and mobility data. There are multiple streams of time series data. If these streams can be integrated into one model to perform forecasting, classification, and probabilistic prediction, it may help address critical health issues. Our dataset also has multiple streams—370 customer entities that consume energy differently while sharing the same 15‑minute time steps—so the same architecture can be applied across these series.
We have applied different models and compared them with our baseline ARIMA model (lower is better on the chosen metric):
- ARIMA: 1.22
- Bayesian LSTM: 0.82
- MQRNN: 0.55
- DeepTCN: 0.15
- N-BEATS: 0.06
From Table 1 and Figure 12, it can be seen that N-BEATS shows the best performance among all deep learning models, and all deep learning models outperform the statistical ARIMA model. These architectures can therefore be applied to a range of time series datasets. All models capture seasonality, trend, and cyclability automatically, which improves scalability. In different applications, these models can be used to forecast health outcomes and financial indicators.
For future work, the main focus should be on:
- Increasing the robustness of the model.
- Analyzing and reducing time complexity to decrease power and time consumption during training.
If these models are applied in financial forecasting, they must be robust to noise and outliers while still detecting important signals. For example, forecasting risk across an industry cluster to support risk management—such as predicting how the banking sector will react to changes in interest rates.
Some directions to improve robustness and efficiency include:
- Transformers are proven technology for long sequence-to-sequence problems, so they can be used to build more robust models.
- There may be thousands or millions of parallel data streams, many of which may not be directly related. To detect relationships, self-attention can be introduced, allowing the model to selectively focus on relevant series and time steps.
- Graph neural networks (GNNs) can be used where each client (or time series) is a node and edges capture relationships between clients. In a GNN framework, parallel time series streams can be modeled as a graph to better capture cross-series dependencies.
Deep learning models are often trained on millions of data points. In multiple time series settings, it is crucial to train models efficiently. Techniques such as FlashAttention, fusion kernels, and parallelization strategies can be applied to reduce training time and memory consumption.
These architectures are capable of forecasting multivariate time series data. They model dependencies among multiple predictors and can be tailored to specific requirements. This helps make forecasting more precise, accurate, and robust. In complex environments where forecasting is difficult, multivariate datasets can improve predictive performance by capturing the full system dynamics. In practical applications like finance, healthcare, or supply chain management, decisions are rarely based on a single variable, so multivariate time series models are especially valuable.
Recurrent Neural Networks (RNNs) suffer from gradient vanishing, and LSTMs can introduce bottlenecks during backpropagation. That is why more advanced architectures, such as convoluted autoregressive models, N-BEATS, and DeepTCN, can be particularly useful for forecasting. If the dataset contains millions of parallel series, N-BEATS can become computationally expensive, but integrating attention and appropriate parallelization across data streams can improve scalability. Graph Neural Networks may also help in this context by modeling relationships among series.
In conclusion, deep learning can be more efficient and scalable than traditional statistical learning for multivariate time series datasets.