4.1 Implementing SARIMAX

After identifying the ARIMA model parameters for our datasets, we move on to forecasting using SARIMAX. SARIMAX stands for Seasonal AutoRegressive Integrated Moving Average with eXogenous factors, enhancing ARIMA by including seasonal cycles and the influence of external variables.

Here’s an example of applying SARIMAX to the “Electric Production” dataset, keeping the latest three months’ data for validation:

import pandas as pd from statsmodels.tsa.statespace.sarimax import SARIMAX from statsmodels.tsa.seasonal import seasonal_decompose

data = pd.read_csv("Electric_Production.csv") monthly_data = data.IPG2211A2N.resample('M').mean().reset_index()

# Splitting the data into training and test sets train_data = monthly_data['IPG2211A2N'][:-3] test_data = monthly_data['IPG2211A2N'][-3:]

# Fit the ARIMA(1,1,1) model model = SARIMAX(train_data, order=(1, 1, 1)) model_fit = model.fit()

# Forecast the last three months forecast = model_fit.forecast(steps=3)

# Calculate the MAPE between actual and predicted values mape = mean_absolute_percentage_error(test_data, forecast) print(f"Forecast: {forecast}") print(f"Actual: {test_data}") print(f"MAPE: {mape}")

We utilize the Mean Absolute Percentage Error (MAPE) as a metric for assessing the forecast's accuracy. The same methodology can be applied across other datasets to ensure consistency in our forecasting strategy.

4.2 Time Series Forecasting with RNN

Recurrent Neural Networks (RNNs) excel in time series forecasting due to their ability to retain past information through hidden states. Unlike the linear modeling of SARIMAX, RNNs can model data nonlinearly, making them particularly adept at recognizing and predicting patterns over time.

Below is an example of using an RNN for forecasting the “Electric Production” dataset, focusing on the last three months for validation to evaluate our model’s predictive performance.

import torch import torch.nn as nn import numpy as np import pandas as pd from sklearn.preprocessing import MinMaxScaler from sklearn.metrics import mean_absolute_percentage_error from statsmodels.tsa.seasonal import seasonal_decompose from torch.utils.data import DataLoader, TensorDataset

# Assuming monthly_data is a DataFrame with your time series column 'IPG2211A2N' tmdata = monthly_data['IPG2211A2N'] data = tmdata.values.reshape(-1, 1)

# Decompose to remove the seasonal component result = seasonal_decompose(tmdata, model='additive', period=12) deseasonalized = tmdata - result.seasonal

# Normalize the data scaler = MinMaxScaler(feature_range=(-1, 1)) data_normalized = scaler.fit_transform(deseasonalized.values.reshape(-1, 1))

# Convert data into sequences def create_sequences(data, seq_length):

xs, ys = [], []

for i in range(len(data)-seq_length-1):

x = data[i:(i+seq_length)]

y = data[i+seq_length]

xs.append(x)

ys.append(y)

return np.array(xs), np.array(ys)

seq_length = 12 X, y = create_sequences(data_normalized, seq_length) X_train, X_test = X[:-3], X[-3-seq_length:-seq_length] y_train, y_test = y[:-3], y[-3:]

# Convert to PyTorch tensors X_train = torch.FloatTensor(X_train) y_train = torch.FloatTensor(y_train).view(-1) X_test = torch.FloatTensor(X_test) y_test = torch.FloatTensor(y_test).view(-1)

class SimpleRNN(nn.Module):

def __init__(self, input_size=1, hidden_layer_size=100, output_size=1):

super(SimpleRNN, self).__init__()

self.hidden_layer_size = hidden_layer_size

self.rnn = nn.RNN(input_size, hidden_layer_size)

self.linear = nn.Linear(hidden_layer_size, output_size)

def forward(self, input_seq):

rnn_out, _ = self.rnn(input_seq.view(len(input_seq), 1, -1))

predictions = self.linear(rnn_out.view(len(input_seq), -1))

return predictions[-1]

model = SimpleRNN() criterion = nn.MSELoss() optimizer = torch.optim.Adam(model.parameters(), lr=0.018) epochs = 220 for i in range(epochs):

for seq, labels in zip(X_train, y_train):

optimizer.zero_grad()

y_pred = model(seq)

single_loss = criterion(y_pred, labels.unsqueeze(-1))

single_loss.backward()

optimizer.step()

if i % 10 == 0:

print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

model.eval() preds_list = [] with torch.no_grad():

for i in range(len(X_test)):

seq = X_test[i].view(-1, 1, 1) # Reshape to (seq_len, batch_size=1, features=1)

pred = model(seq)

preds_list.append(pred.item())

# Convert predictions list to a numpy array for inverse scaling preds_array = np.array(preds_list).reshape(-1, 1) preds_inverse = scaler.inverse_transform(preds_array)

# Inverse transform the actual test labels y_test_inverse = scaler.inverse_transform(y_test.numpy().reshape(-1, 1))

# Calculate MAPE mape = np.mean(np.abs((y_test_inverse - preds_inverse) / y_test_inverse)) * 100 print(f'MAPE: {mape}%')

Key steps in the RNN time series forecasting process include:

• Preprocessing: Removes seasonality and normalizes the data for RNN readiness.
• Sequence Preparation: Transforms data into sequences for training, replicating temporal dependencies.
• RNN Architecture: Utilizes an RNN layer for temporal processing and a linear layer for predictions.
• Training: Iterates through epochs to minimize loss, updating the model through backpropagation.
• Forecasting: Predicts future values for the test set using learned patterns.
• Inverse Transformation: Converts forecasts back to their original scale for evaluation.
• Accuracy Evaluation: Uses MAPE to measure the model’s forecasting precision.

4.3 Time Series Forecasting with LSTM

Long Short-Term Memory (LSTM) networks are designed to enhance Recurrent Neural Networks (RNNs) by better managing long-term dependencies and outliers. However, the actual effectiveness of LSTMs can vary across datasets, highlighting the necessity for empirical testing. Future investigations will focus on data-driven evaluations of various algorithms, prioritizing real results over theoretical expectations. Below is an example of LSTM implementation:

class LSTMModel(nn.Module):

def __init__(self, input_size=1, hidden_layer_size=100, output_size=1):

super(LSTMModel, self).__init__()

self.lstm = nn.LSTM(input_size, hidden_layer_size)

self.linear = nn.Linear(hidden_layer_size, output_size)

def forward(self, input_seq):

lstm_out, _ = self.lstm(input_seq.view(len(input_seq), 1, -1))

predictions = self.linear(lstm_out.view(len(input_seq), -1))

return predictions[-1]

model = LSTMModel() optimizer = torch.optim.Adam(model.parameters(), lr=0.001) epochs = 180 # Training loop and prediction generation follow here.

The significant distinction in coding for LSTM, compared to RNN, lies in the model structure. LSTMs incorporate an nn.LSTM layer rather than nn.RNN, specifically designed to tackle long-term dependency challenges in time series data. This structural modification is crucial for utilizing LSTMs effectively in real-world forecasting tasks.

4.4 Time Series Forecasting with Facebook Prophet

Facebook Prophet is theoretically designed to enhance forecasting by addressing non-linear trends, seasonal variations, and holiday impacts. Its adaptability to various business forecasting needs is well-recognized, especially its capability to handle missing data and adjust to sudden trend shifts. Prophet is particularly beneficial for business contexts characterized by:

• Data that spans monthly to yearly intervals with notable seasonal patterns.
• Significant holidays with predictable occurrences.
• External factors that induce trend changes, such as product launches.
• Growth trends nearing saturation.

Here’s the Python code for applying Prophet to the “Electric Production” dataset:

start_date = '2020-01-01' dates = pd.date_range(start=start_date, periods=len(monthly_data['IPG2211A2N']), freq='M') df_prophet = pd.DataFrame(data={'ds': dates, 'y': monthly_data['IPG2211A2N'].values})

# Initialize the Prophet model with additional seasonality components model = Prophet(yearly_seasonality=True, seasonality_prior_scale=0.2)

# Adding monthly seasonality model.add_seasonality(name='monthly', period=30.5, fourier_order=8)

# Fit the model with your DataFrame model.fit(df_prophet[:-3]) # Exclude the last 3 months for validation

# Create a DataFrame for future predictions including the last 3 months future = model.make_future_dataframe(periods=3, freq='M')

# Use the model to make predictions forecast = model.predict(future)

# Focus on the last 3 months for validation forecast_last_3_months = forecast['yhat'][-3:].values

# Actual values for the last 3 months actual_last_3_months = df_prophet['y'][-3:].values

# Calculate the MAPE between actual and forecasted values mape = mean_absolute_percentage_error(actual_last_3_months, forecast_last_3_months)

print(f"Forecasted Values: {forecast_last_3_months}") print(f"Actual Values: {actual_last_3_months}") print(f"MAPE: {mape}")

Key parameters influencing forecasting include:

• seasonality_prior_scale (0.2): Adjusts the flexibility of seasonality. Lower values tighten seasonality, which is useful for consistent patterns while preventing overfitting.
• fourier_order (8): Determines the complexity of the seasonal model. Higher values capture detailed fluctuations but may lead to overfitting. Select based on your data’s seasonal variation.
• period (30.5) in model.add_seasonality: Defines the cycle length for added seasonality, approximating a month to align with the data’s seasonal frequency.

4.5 Time Series Forecasting with Attention Transformers

Initially developed for Natural Language Processing (NLP), Attention Transformers are now being explored for time series forecasting. Theoretically, their capacity to weigh the significance of different input data points enables a nuanced understanding of complex temporal relationships, moving beyond the sequential processing characteristic of RNNs.

The application of Transformers in time series is experimental, aiming to leverage their attention mechanisms for predicting trends and seasonal patterns across various datasets.

Here is the code for applying Transformers to the “Electric Production” dataset:

class PositionalEncoding(nn.Module):

def __init__(self, d_model, dropout=0.1, max_len=5000):

super(PositionalEncoding, self).__init__()

self.dropout = nn.Dropout(p=dropout)

position = torch.arange(0, max_len).unsqueeze(1)

div_term = torch.exp(torch.arange(0, d_model, 2) * -(math.log(10000.0) / d_model))

pe = torch.zeros(max_len, 1, d_model)

pe[:, 0, 0::2] = torch.sin(position * div_term)

pe[:, 0, 1::2] = torch.cos(position * div_term)

self.register_buffer('pe', pe)

def forward(self, x):

x = x + self.pe[:x.size(0)]

return self.dropout(x)

class TransformerModel(nn.Module):

def __init__(self, input_dim, d_model, nhead, num_layers, dim_feedforward, dropout=0.1):

super(TransformerModel, self).__init__()

self.model_type = 'Transformer'

self.pos_encoder = PositionalEncoding(d_model, dropout)

encoder_layers = nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout)

self.transformer_encoder = nn.TransformerEncoder(encoder_layers, num_layers)

self.encoder = nn.Linear(input_dim, d_model)

self.d_model = d_model

self.decoder = nn.Linear(d_model, 1)

def forward(self, src):

src = self.encoder(src) * math.sqrt(self.d_model)

src = self.pos_encoder(src)

output = self.transformer_encoder(src)

output = self.decoder(output)

return output

model = TransformerModel(input_dim=1, d_model=64, nhead=4, num_layers=4, dim_feedforward=256, dropout=0.2) train_data = TensorDataset(torch.FloatTensor(X_train), torch.FloatTensor(y_train)) train_loader = DataLoader(train_data, batch_size=16, shuffle=False)

optimizer = optim.Adam(model.parameters(), lr=0.001) criterion = nn.MSELoss()

# Training loop for epoch in range(120):

model.train()

total_loss = 0

for batch, (data, targets) in enumerate(train_loader):

optimizer.zero_grad()

data = data.permute(1, 0, 2) # Reshape for the transformer [seq_len, batch_size, features]

output = model(data)

loss = criterion(output.view(-1), targets)

loss.backward()

optimizer.step()

total_loss += loss.item()

if epoch % 10 == 0:

print(f'Epoch: {epoch}, Loss: {total_loss / len(train_loader)}')

# Forecasting and reseasonalizing predictions model.eval() preds = [] with torch.no_grad():

for seq in torch.FloatTensor(X_test):

seq = seq.unsqueeze(1) # Shape to [seq_len, batch_size=1, features]

pred = model(seq)

pred_last = pred[-1, :, :].squeeze().item()

preds.append(pred_last)

preds_inverse = scaler.inverse_transform(np.array(preds).reshape(-1, 1)) seasonal_component = result.seasonal[-len(preds):].values.reshape(-1, 1) final_predictions = preds_inverse + seasonal_component

y_test_actual = monthly_data['IPG2211A2N'][-len(preds):].values.reshape(-1, 1) mape = np.mean(np.abs((y_test_actual - final_predictions) / y_test_actual)) * 100 print(f'MAPE: {mape}%')

Key steps in time series forecasting with Transformers include:

• Positional Encoding: Introduces unique position information to data, assisting the model in understanding sequence order without step-by-step processing like RNNs.
• Transformer Model Setup: Contains layers for processing the data (encoders) and generating predictions (decoders), tailored for time series specifics.
• Model Training: Focuses on optimizing the model on training data to minimize prediction errors.
• Forecasting: Applies the trained model to predict future values while reshaping input data to meet transformer requirements.
• Re-adding Seasonality: Seasonal patterns removed earlier are reintegrated with predictions for accurate real-world relevance.
• Model Evaluation: Assesses performance using the Mean Absolute Percentage Error (MAPE).

Summary of Findings:

• RNN and Prophet: Both methods excel in terms of accuracy and consistency. RNNs are particularly effective with complex datasets, while Prophet is optimal for datasets exhibiting strong seasonality.
• Transformers: Despite their success in NLP, Transformers have shown limited effectiveness in time series forecasting, indicating a need for further refinement in this domain.
• SARIMAX and Prophet: These methods are preferable for datasets with a clear ARIMA structure or smaller sizes, as they may be less vulnerable to the overfitting risks associated with neural networks.
• Period/Cycle Handling: Accurately identifying and integrating period/cycle information is vital. Unlike RNN and LSTM, which require manual input, Prophet and SARIMAX are better equipped to automatically account for seasonal effects.
• Parameter Importance: In Prophet, parameters such as seasonality_prior_scale, period, and fourier_order significantly affect its ability to model seasonality. In RNN and LSTM, factors like learning rate and dropout=0.2 are crucial for effective learning and generalization.
• Speed of Computation: Prophet and SARIMAX offer faster computation times, providing a notable advantage over neural network-based methods like RNN and LSTM, which need more time to train on extensive datasets.

Additional Insights:

LSTM vs. RNN: We anticipated LSTMs would outperform RNNs, but our tests did not provide clear evidence of this. This raises questions about whether LSTMs are indeed better at managing long-term patterns and outliers. Even after adjusting settings, RNNs adapted well, particularly when we modified the learning rates, which had a negligible impact on LSTMs.

These unexpected results prompted me to create a specific dataset to test if LSTMs are superior to RNNs in scenarios where they are presumed to excel. This dataset features long-term repetitive patterns and sudden deviations (outliers), offering a rigorous examination of LSTM capabilities.

Data Generation Code:

# Generate Time Series: simulate a time series with 70 data points n_points = 70 t = np.arange(n_points) # Sine wave + linear trend + noise data = 2 * np.sin(t / 8) + 0.1 * t + np.random.normal(0, 0.5, n_points) data[-5:] += np.array([3, -1, 2, -1, 2]) # Introducing outliers

After executing both RNN and LSTM models on this generated dataset, I obtained the following results:

• LSTM MAPE: 23.32%
• RNN MAPE: 33.65%

These outcomes align with theoretical expectations that favor LSTMs for their superior long-term dependency management and outlier handling. This experiment underscores the significance of selecting the right model based on data characteristics. Future research will aim to validate these findings with real-world scenarios.