Why Implement a Baseline Model?

There are three primary reasons to include a baseline model in your project:

#### 1. Enhancing Data Comprehension

Creating a baseline model enhances our understanding of the data, particularly in relation to model formulation. During its development, several insights can be gathered, such as:

• Predictive Power of the Dataset: If the baseline model shows minimal predictive capability, it may suggest a weak signal or poor fit.
• Classifying Subsets of the Dataset: The baseline model can reveal which parts of the data are more challenging to categorize, guiding model selection.
• Target Value Classification: Observations alongside the baseline model may indicate which target values are difficult to identify.

#### 2. Accelerating Model Development

Having a baseline model simplifies the model development process and various other tasks. It serves as a reference point, and in some cases, the baseline model may sufficiently meet business needs.

#### 3. Establishing Performance Benchmarks

We utilize the baseline model to gauge performance metrics while developing the actual model. It enables us to evaluate whether we require a more complex model or if the simpler one suffices for the business context. Additionally, the baseline model can act as a benchmark for business KPIs.

Classifying Baseline Models

We can categorize baseline models into two types:

1. Simple Baseline Models
2. Machine Learning Baseline Models

Simple Baseline Models

A simple baseline model employs straightforward logic for its creation. It may involve random predictions or specific rule-based approaches. The goal is to construct a model that can serve as a reference point against the actual model.

Simple baseline models can be derived from basic statistics, business logic, or various stochastic models. The selection often depends on factors such as:

• Structured vs. unstructured data
• Supervised vs. unsupervised problems
• Classification vs. regression

For demonstration, let’s focus on a classification problem using the Breast Cancer dataset.

import pandas as pd from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.dummy import DummyClassifier from sklearn.metrics import accuracy_score, f1_score, recall_score, precision_score

df = load_breast_cancer() X, y = pd.DataFrame(df['data'], columns=df['feature_names']), pd.Series(df['target'])

Our dataset consists of 30 numerical features. Next, we can analyze the target distribution.

y.value_counts()

The distribution appears slightly imbalanced with the target value of 1. We aim to build a classification model to predict breast cancer patients and thus require a baseline model for comparison. We will create multiple simple baseline models utilizing the DummyClassifier from Scikit-Learn.

What is the DummyClassifier? It is a classifier that disregards input features, establishing a simple baseline for comparison with more advanced classifiers, employing various strategies.

First, let's define a function to generate the baseline model.

# Function for evaluation metrics def print_binary_evaluation(X_train, X_test, y_train, y_true, strategy):

dummy_clf = DummyClassifier(strategy=strategy)

dummy_clf.fit(X_train, y_train)

y_pred = dummy_clf.predict(X_test)

results_dict = {

'accuracy': accuracy_score(y_true, y_pred),

'recall': recall_score(y_true, y_pred),

'precision': precision_score(y_true, y_pred),

'f1_score': f1_score(y_true, y_pred)

}

return results_dict

Next, we will split the dataset.

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

Now, let’s evaluate each baseline model strategy using the aforementioned function.

Most Frequent (Mode)

This basic strategy predicts the most common label in the dataset. Since target 1 is the most frequent, the classifier will always predict 1.

print_binary_evaluation(X_train, X_test, y_train, y_test, 'most_frequent')

By consistently predicting 1, we achieve an accuracy of 0.62 and an F1 score of 0.76. While these figures may seem reasonable in a machine learning context, suggesting that all patients have breast cancer is not beneficial for business.

At the very least, we now possess a baseline model for comparison with our complex model. Let's explore another strategy.

Uniform

The uniform strategy develops a baseline model that predicts with a uniform distribution, implying that all targets share an equal probability of being predicted.

print_binary_evaluation(X_train, X_test, y_train, y_test, 'uniform')

The resulting metrics are closer to 50%, reflecting the uniform prediction distribution.

Stratified

A stratified strategy generates a baseline model that aligns with the target distribution, making it suitable for imbalanced datasets by accurately reflecting the actual distribution.

print_binary_evaluation(X_train, X_test, y_train, y_test, 'stratified')

The metrics are similar to those of the uniform strategy due to the comparable distribution.

In summary, the simple baseline model is a prediction model that disregards input data, focusing solely on the outcome. If we utilize the simple baseline model as a benchmark, we should aim to outperform the model that predicts all patients have breast cancer.

An alternative method for developing a simple baseline model is based on business logic, which requires domain knowledge and an understanding of the business context. I have previously written on this topic in another article.

Machine Learning Baseline Models

Baseline models are not limited to simple models; you can also employ machine learning models. The key distinction is that the ML baseline model is utilized solely for comparison against a more complex model.

For instance, if you intend to use the XGBoost model for predictions, your baseline model might be a decision tree. If you are researching a new model, you would want a state-of-the-art model as your baseline.

Regardless, a machine learning model can serve as your baseline model, but it should always be used to benchmark your primary model.