Develop Deep Learning Models with PyTorch and Google Colab - A Practical Introduction.

PyTorch is an open-source machine learning library developed on Python. It provides a simple and intuitive programming environment for developing Deep Learning models.

The use of PyTorch is becoming increasingly important for developers because it has proven to be a very effective tool for artificial intelligence and machine learning development. It offers easy integration with other applications and tools, flexible data processing and strong GPU acceleration. Because of its ease of use and high performance, it has become one of the most widely used libraries in deep learning model development. PyTorch’s features can be summarised as follows:

1. Easy to learn and use: PyTorch uses a simple and intuitive programming environment that allows developers to quickly start developing deep learning models.
2. Flexible Data Processing: PyTorch provides flexible data processing that allows developers to process data in the formats that best suit their applications.
3. GPU Acceleration: PyTorch harnesses the power of modern GPUs for fast processing of data and development of complex deep learning models.
4. Integrability: PyTorch integrates easily with other applications and tools, allowing developers to integrate their deep learning models into existing workflows.
5. Active Community: PyTorch has an active community of developers who are constantly developing new features and enhancements, which constantly improves the performance and flexibility of the library.
6. Dynamic Computational Graph: PyTorch provides a dynamic computational graph that allows developers to change and improve their models as they run. 7th Transfer Learning Support: PyTorch provides easy support for transfer learning, allowing developers to use and improve existing deep learning models for new applications.

PyTorch is now the gold standard in machine learning in 2023 and is one of the two big frameworks for it, along with TensorFlow.

From Facebook to number 1

PyTorch was developed by Facebook and first released in 2017. The development of PyTorch began as a replacement for the machine learning framework Torch used at the time. The original Torch code was written in C (CUDA). An SDK based on LuaJIT served as the interface language. In practice, however, there were some hurdles in using Torch, especially in integrating it with other existing tools and libraries.

To address these issues, Facebook’s developers decided to write a new framework that built on the best features of Torch while taking advantage of Python. The result was PyTorch, an open-source machine learning library that provides a simple programming environment and relatively flexible data processing. Hot Stuff! 🤓

Jupyter Notebooks

Before jumping in with a PyTorch example, we need to understand what PyTorch is best used with, i.e. on which platform or machine the deep learning model is written, modelled, trained and evaluated. Of course, one could simply install PyTorch on one’s computer and use existing Python-enabled IDEs such as JetBrain’s PyCharm, JetBrain’s DataSpells, or even Microsoft’s VSCode.

The common thread that is at the forefront of our choices is the integrated nature of Jupyter Notebook. For those who read our TechUps regularly and still have little intersection with Data Science or Machine Learning, the term “Jupyter Notebook” might be new. At this point, it should be said that Jupyter Notebooks are the heart and basic tool of any Data Scientist or Data Engineer.

Jupyter Notebook is a web-based, interactive environment for creating documents that contain live code, descriptive prose, visualisations and other multimedia content. This allows the Data Scientist to write code, run it and see the results immediately. This makes it easy to use and keeps iteration cycles short. In addition, Jupyter Notebook, precisely because it is web-based, enables collaboration by allowing users to share documents and work on them together.

Although Jupyter supports a variety of programming languages, the lingua franca is Python. Also common are languages such as Julia, GNU Octave or R. Thus, Jupyter Notebook is the favoured environment to run Machine Learning with PyTorch.

As a web-based environment, Jupyter is based on a client-server architecture. You can install it on your local machine or embed it in a well-designed cloud architecture. For this TechUp, we’ll make it a little easier and use a free cloud-based platform, namely Google Colab.

Jupyter in the Cloud with Google Colab

Google Colab is a free online platform that allows users to run Jupyter notebooks in the cloud. It is an easy-to-use platform that provides users with easy access to all popular machine learning libraries and frameworks. Colab also offers GPU support that allows users to quickly and efficiently build and train deep-learning models.

To create deep learning models with Colab, you first need to create a new notebook and install the required libraries and frameworks. Colab comes with many libraries and frameworks pre-installed, so there is usually no need to install them manually. You can then write code, import data, and create and train models.

PyTorch Kickstart

Here is a simple example of Deep Learning with PyTorch.

1st Importing Dependencies: First we need to import the PyTorch libraries we need.

``python import torch import torch.nn as nn import torch.optim as optim

1
2
3
4
5
6

2nd **Data Preparation**: Next, we prepare our training data. Here we use a simple dataset consisting of two features (the `x_train` axis) and one target (the `y_train` axis).

`python
x_train = torch.tensor([[1.0, 1.0], [2.0, 2.0], [3.0, 3.0]])
y_train = torch.tensor([[2.0], [4.0], [6.0]])

3rd Model Definition: Next, we define our model using PyTorch. Here we use a simple, linearly layered model.

``python class LinearRegression(nn.Module): def init(self): super().init() self.linear = nn.Linear(2, 1)

def forward(self, x):
    return self.linear(x)

model = LinearRegression()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

4. **Model training**: Now we can train our model with the training data. Here we use the **Stochastic Gradient Descent (SGD)** as an optimiser and define a loss function.

``python
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.01)

for epoch in range(100):
    y_pred = model(x_train)
    loss = criterion(y_pred, y_train)

    optimizer.zero_degree()
    loss.backward()
    optimizer.step()

5. Prediction: Finally, we can use our model to make predictions on new data.

``python x_test = torch.tensor([[4.0, 4.0]]) y_test = model(x_test) print(y_test)

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

This example demonstrates the basic steps in using PyTorch to perform Deep Learning.

### Machine Learning Slang

Machine learning has its own vocabulary, which should be understood in the process. In the example above, we have found the terms of **loss**, **accuracy**, or **epoch**. Here follows a brief explanation of these terminologies:

- **Loss**: **The penalty for a bad prediction**. That is, loss is a number that indicates how bad the model's prediction was for a single example. If the model's prediction is perfect, the loss is zero; otherwise, the loss is greater.
- **Accuracy**: Accuracy is **a measure for evaluating classification models**. Unofficially, accuracy is the proportion of correct predictions made by our model.
- **Epidity**: Each time a dataset passes through an algorithm, it is said to have completed an epoch. So in machine learning, epoch refers to **the entire run of training data through the algorithm**. It is a hyperparameter that determines the training process of the machine learning model.

That's it? No. Now let's get an understanding of what exactly PyTorch is abstracting. 



## The magic behind PyTorch

Since we at b-nova always strive to understand what makes something tick at its core, we naturally need to take a closer look and shed some light on the abstraction solution behind PyTorch.

Above in the example we see that certain functions like `torch.nn` or `torch.optim` as well as data structures like `Dataset` or `DataLoader` are hidden. So let's start with `torch.nn` and see how we can decode the magic behind this call.

### `torch.nn`

**`torch.nn`** is an abstraction in PyTorch that allows developers to easily and efficiently define and train neural networks and their layers. It provides a set of classes and methods that abstracts and simplifies commonly used neural network types, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs) and Fully Connected Networks (FCNs).

With **`torch.nn`**, developers can create their own layers tailored to the needs of their applications, or use predefined layers such as Convolutional Layers, Max Pooling Layers and Activation Functions. Layers can be chained together to define a complete neural network, and training and evaluation of models can then be performed using **`torch.optim`** and other PyTorch tools.

In short, **`torch.nn`** provides a simple and efficient way to define and train neural networks and their layers by abstracting and simplifying commonly used constructs.

### `torch.optim`

**`torch.optim`** is an abstraction in PyTorch that allows developers to easily and efficiently implement optimisation algorithms for training neural networks. It provides a set of classes that abstracts and simplifies commonly used optimisation techniques, such as Stochastic Gradient Descent (SGD), Adagrad, Adadelta, Adam and many others.

With **`torch.optim`**, developers can choose an optimiser that fits their needs and requirements, and then pass in the training parameters and loss function value for the neural network. The optimiser class then automatically calculates the gradients and updates the weights to improve the model.

In summary, `torch.nn` and **`torch.optim`** provide a simple and efficient way to define and train neural networks and their layers, as well as implement optimisation algorithms for training neural networks, by abstracting both quasi-commonly used constructs.



## Real-World with MNIST dataset

Here we would like to run through a real-world example where we deal with a familiar problem: reading handwritten digits. This is a commonly used introductory example, as it is well suited to assess the outcome of a solution.

The dataset we use for this comes from the MNIST database. MNIST stands for *Modified [National Institute of Standards and Technology](https://en.wikipedia.org/wiki/National_Institute_of_Standards_and_Technology)*, a US federal agency. The MNIST dataset contains 70,000 handwritten digits (0-9) in 28x28 pixel format. It is often used for training optical character recognition models.

In this example, the model **`MNISTModel`** is defined. It consists of three linear layers, each connected by a ReLU activation function. The model is trained with the Adam optimiser and the cross entropy loss function. The training and test dataset are loaded in batches using **`DataLoader`** to speed up the training. The model is trained for 10 epochs. After each epoch, the model is tested on the test data set and the test loss and accuracy are output.

### The finished notebook

``python
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision.datasets import MNIST
from torchvision.transforms import ToTensor
from torch.utils.data import DataLoader

# load the training and test dataset
train_data = MNIST(root='data', train=True, download=True, transform=ToTensor())
test_data = MNIST(root='data', train=False, download=True, transform=ToTensor())

# Define the model
class MNISTModel(nn.Module):
    def __init__(self):
        super(MNISTModel, self).__init__()
        self.fc1 = nn.Linear(28*28, 64)
        self.fc2 = nn.Linear(64, 32)
        self.fc3 = nn.Linear(32, 10)

    def forward(self, x):
        x = x.view(-1, 28*28)
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        x = self.fc3(x)
        return x

# Initialise the model, the loss function and the optimiser
model = MNISTModel()
loss_fn = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# definition of the trainer
def train(model, train_loader, optimizer, loss_fn):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_degree()
        output = model(data)
        loss = loss_fn(output, target)
        loss.backward()
        optimizer.step()

# definition of the tester
def test(model, test_loader, loss_fn):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            output = model(data)
            test_loss += loss_fn(output, target).item()
            pred = output.argmax(dim=1, keepdim=True)
            correct += pred.eq(target.view_as(pred)).sum().item()
    test_loss /= len(test_loader.dataset)
    accuracy = 100. * correct / len(test_loader.dataset)
    return test_loss, accuracy

# Train and test the model
train_loader = DataLoader(train_data, batch_size=64, shuffle=True)
test_loader = DataLoader(test_data, batch_size=64, shuffle=True)
for epoch in range(10):
    train(model, train_loader, optimizer, loss_fn)
    test_loss, accuracy = test(model, test_loader, loss_fn)
    print(f'Epoch {epoch}: Test Loss: {test_loss:.4f}, Accuracy: {accuracy:.2f}%')

What happens here can be summarised in the following steps

1. first, the required libraries are imported, including torch, torch.nn, torch.optim, MNIST and ToTensor.
2. the training and testing libraries are imported. The training and test data set are loaded and stored in train_data and test_data. ToTensor is used to convert the images into tensors.
3. The model MNISTModel is defined. It consists of three linear layers with ReLU activation between them. The first layer has 784 inputs (28 x 28 images), the second layer has 64 neurons and the third layer has 32 neurons and returns an output of 10 classes.
4. the model is initialised and the adam optimiser and the cross entropy loss function are defined.
5. the function train is defined to train the model on the training dataset. model.train() is called to put the model into training mode. The data and targets are loaded by train_loader. The optimiser is reset to zero (optimizer.zero_grad()), forward propagation is performed (output = model(data)), loss is calculated (loss = loss_fn(output, target)) and backward propagation is performed (loss.backward()). Finally, the optimiser is updated (optimizer.step()). The function test is defined to test the model on the test data set. model.eval() is called to put the model into evaluation mode. The test losses and accuracy are calculated by comparing the output of the model to the target (pred.eq(target.view_as(pred)).sum().item()).
6. train_loader and test_loader are defined to load the data in batches.
7. the model is trained for 10 epochs. In each epoch, the model is tested on the test data set and the test losses and accuracy are output. The train and test functions are used to train and test the model on the training and test data set.

Conclusion

Dealing with PyTorch, Jupyter Notebooks and Deep Learning is extremely future proof. PyTorch is a powerful library for Deep Learning and is used by many well-known companies and research institutions. It plays a crucial role in the machine learning ecosystem and is expected to continue to play a significant role in the future.

Jupyter Notebooks are also an essential tool in the machine learning space and are used by many researchers and data scientists to document, share and make their work reproducible. They provide an effective way to combine code, text and visual representations into a single document.

Deep Learning is a rapidly growing discipline that has applications in numerous industries and use cases. The demand for professionals with Deep Learning skills is high and is expected to increase in the future. Therefore, exposure to PyTorch, Jupyter Notebooks and Deep Learning will continue to be extremely relevant and future-proof.

Resources and further links

PyTorch

Machine Learning Mastery

What is torch.nn really? - PyTorch Tutorials 1.13.1+cu117 documentation

MNIST handwritten digit database, Yann LeCun, Corinna Cortes and Chris Burges

MNIST Dataset

Top 15 Machine Learning Libraries in 2023

Top Machine Learning Trends for 2023

[Torch - Meta Research](