Prerequisites

To effectively follow this tutorial, you should have:

• A basic understanding of Python.
• Familiarity with NLP concepts.
• Knowledge of deep learning models, particularly neural networks and their architectures.
• Basic experience with the PyTorch library.

Setting Up the Environment

For this tutorial, we will be utilizing the transformers, torch, and pandas libraries to implement the BERT model. You can install these libraries using pip:

pip install transformers torch pandas

Dataset

We will work with the IMDb Movie Reviews dataset, which comprises 50,000 reviews from IMDb users. This dataset is split into training and testing subsets, each containing 25,000 reviews labeled as either positive or negative.

To download the dataset, we will use the urllib module and extract it using the tarfile module. Below is the code that downloads and extracts the dataset:

import urllib.request

import tarfile

import os

# Download and extract the dataset

filename = "aclImdb_v1.tar.gz"

if not os.path.isfile(filename):

urllib.request.urlretrieve(url, filename)

tar = tarfile.open(filename, "r:gz")

tar.extractall()

tar.close()

# Read the dataset into a pandas dataframe

train_texts = []

train_labels = []

test_texts = []

test_labels = []

# Read train dataset

for foldername in ['train/pos', 'train/neg']:

folderpath = 'aclImdb/' + foldername

for filename in os.listdir(folderpath):

with open(os.path.join(folderpath, filename), 'r') as file:

text = file.read()

label = 1 if foldername.split('/')[1] == 'pos' else 0

train_texts.append(text)

train_labels.append(label)

# Read test dataset

for foldername in ['test/pos', 'test/neg']:

folderpath = 'aclImdb/' + foldername

for filename in os.listdir(folderpath):

with open(os.path.join(folderpath, filename), 'r') as file:

text = file.read()

label = 1 if foldername.split('/')[1] == 'pos' else 0

test_texts.append(text)

test_labels.append(label)

# Convert to dataframe

train_df = pd.DataFrame({'text': train_texts, 'label': train_labels})

test_df = pd.DataFrame({'text': test_texts, 'label': test_labels})

This snippet downloads the aclImdb_v1.tar.gz file, extracts its contents, and reads the training and testing datasets by iterating through the respective directories. It stores the text and labels in separate lists before converting them to dataframes.

Preprocessing the Data

Before training the BERT model, we must preprocess the data, converting the text into numerical tokens suitable for model input. We will use the BertTokenizer class from the transformers library for tokenization. The following code tokenizes the text and pads the sequences to a fixed length:

from transformers import BertTokenizer

# Load the BERT tokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

# Tokenize the text

train_encodings = tokenizer(list(train_df['text']), truncation=True, padding=True)

test_encodings = tokenizer(list(test_df['text']), truncation=True, padding=True)

# Create PyTorch datasets

class IMDbDataset(torch.utils.data.Dataset):

def __init__(self, encodings, labels):

self.encodings = encodings

self.labels = labels

def __getitem__(self, idx):

item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}

item['labels'] = torch.tensor(self.labels[idx])

return item

def __len__(self):

return len(self.labels)

train_dataset = IMDbDataset(train_encodings, list(train_df['label']))

test_dataset = IMDbDataset(test_encodings, list(test_df['label']))

This code loads the pre-trained BERT tokenizer and tokenizes the text from both the training and testing datasets while enabling truncation and padding. We define a custom class, IMDbDataset, to manage the PyTorch datasets and convert the tokens and labels into PyTorch tensors.

Fine-tuning the BERT Model

We will utilize the pre-trained BERT model for fine-tuning to conduct sentiment analysis. The model can be loaded using the BertForSequenceClassification class from the transformers library:

from transformers import BertForSequenceClassification

# Load the BERT model

model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

# Set up the optimizer

params = model.parameters()

optimizer = torch.optim.Adam(params, lr=1e-5)

This snippet demonstrates how to load the pre-trained BERT model and configure the Adam optimizer with a learning rate of 1e-5.

Now, we can train the model using the training dataset and validate it on the testing dataset:

from tqdm.auto import tqdm

from sklearn.metrics import accuracy_score, classification_report

# Set up the dataloaders

train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, shuffle=True)

test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=16)

# Set up training loop

def train(model, optimizer, train_loader):

model.train()

train_loss = 0

for batch in tqdm(train_loader):

optimizer.zero_grad()

input_ids = batch['input_ids'].to(device)

attention_mask = batch['attention_mask'].to(device)

labels = batch['labels'].to(device)

outputs = model(input_ids, attention_mask=attention_mask, labels=labels)

loss = outputs.loss

loss.backward()

optimizer.step()

train_loss += loss.item()

return train_loss / len(train_loader)

# Set up validation loop

def evaluate(model, test_loader):

model.eval()

predictions = []

true_labels = []

val_loss = 0

with torch.no_grad():

for batch in tqdm(test_loader):

input_ids = batch['input_ids'].to(device)

attention_mask = batch['attention_mask'].to(device)

labels = batch['labels'].to(device)

outputs = model(input_ids, attention_mask=attention_mask, labels=labels)

loss = outputs.loss

logits = outputs.logits

val_loss += loss.item()

preds = torch.argmax(logits, dim=1).flatten()

predictions.extend(preds.cpu().detach().numpy())

true_labels.extend(labels.cpu().detach().numpy())

return val_loss / len(test_loader), predictions, true_labels

# Train the model

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

model.to(device)

for epoch in range(3):

print(f"Epoch {epoch+1}")

print("Training...")

train_loss = train(model, optimizer, train_loader)

print(f"Train Loss: {train_loss:.4f}")

print("Evaluating...")

val_loss, predictions, true_labels = evaluate(model, test_loader)

accuracy = accuracy_score(true_labels, predictions)

print(f"Val Loss: {val_loss:.4f}, Accuracy: {accuracy:.4f}")

print(classification_report(true_labels, predictions))

This code trains the model over three epochs, iterating through the training dataset using the train function, while validating its performance on the test dataset via the evaluate function. The Adam optimizer and a default learning rate of 1e-5 are utilized.

Conclusion

In this article, we illustrated the process of conducting sentiment analysis using BERT in Python. We employed the IMDb Movie Reviews dataset to fine-tune a pre-trained BERT model for sentiment analysis tasks. By tokenizing the text with the BertTokenizer class and creating PyTorch datasets for both training and testing, we successfully fine-tuned the model and evaluated its performance. The trained model can now be applied to predict the sentiment of any given text.

In the first video, "Sentiment Analysis with BERT Neural Network and Python," viewers will learn how to implement sentiment analysis using BERT, including setup and model training.

The second video, "Sentiment Analysis with BERT using Hugging Face, PyTorch, and Python Tutorial," provides a comprehensive tutorial on utilizing BERT for sentiment analysis with practical coding examples.