Python is a general purpose, high-level programming language widely used in data science, making it an intuitive choice for data scientists extending their work into actively modeling deep learning networks. Python’s simple syntax is easy to read, takes relatively little time to learn and can run on any operating system, including Windows, macOS, Linux or Unix. Python has been the second most used programming language on GitHub for over three years, having overtaken Java in 2019. It continues to grow in popularity, with a 22.5 percent increase in 2022¹.

 This flexibility and simplicity has helped foster a robust online community of Python developers, collaborating on a wide array of Python libraries and APIs—like Numerical Python (NumPy) for mathematical operations, Pandas for data manipulation or matplotlib for data visualization—and educational resources. This community has also produced a great volume of Pytorch libraries that reduce the monotony and guesswork of coding for machine learning, freeing up developers and data scientists to focus on innovation rather than rote task writing.

In any machine learning algorithm, even those applied to ostensibly non-numerical information like sounds or images, data must be represented numerically. In PyTorch, this is achieved through tensors, which serve as the fundamental units of data used for computation on the platform.

In the context of machine learning, a tensor is a multi-dimensional array of numbers that functions like a mathematical bookkeeping device. Linguistically, “tensor” functions as a generic term inclusive of some more familiar mathematical entities:

• A scalar is a zero-dimensional tensor, containing a single number.
• A vector is a one-dimensional tensor, containing multiple scalars of the same type. A tuple is a one-dimensional tensor containing different data types.
• A matrix is a two-dimensional tensor, containing multiple vectors of the same type.
• Tensors with three or more dimensions, like the three-dimensional tensors used to represent RGB images in computer vision algorithms, are collectively referred to as N-dimensional tensors.

PyTorch tensors function similarly to the ndarrays used in NumPy—but unlike ndarrays, which can only run on central processing units (CPUs), tensors can also run on graphics processing units (GPUs). GPUs enable dramatically faster computation than CPUs, which is a major advantage given the massive volumes of data and parallel processing typical to deep learning.

In addition to encoding a model’s inputs and outputs, PyTorch tensors also encode model parameters: the weights, biases and gradients that are “learned” in machine learning. This property of tensors enables automatic differentiation, which is one of PyTorch’s most important features.

PyTorch uses modules as the building blocks of deep learning models, which allows for the quick and straightforward construction of neural networks without the tedious work of manually coding each algorithm.

 Modules can—and often do—contain other nested modules. In addition to enabling the creation of more elaborate multi-layer neural networks, this also allows these complex deep learning models to be easily saved as a single named module and transferred between different machines, CPUs or GPUs. PyTorch models can even be run in non-Python environments, like C++, using Torchscript (link resides outside ibm.com), helping bridge the gap between research prototypes and production deployment.

 Broadly speaking, there are three primary classes of modules used to build and optimize deep learning models in PyTorch:

• nn modules are deployed as the layers of a neural network. The torch.nn package contains a large library of modules that perform common operations like convolutions, pooling and regression. For example, torch.nn.Linear(n,m) calls a linear regression algorithm with n inputs and m outputs (whose initial inputs and parameters are then established in subsequent lines of code).

• The autograd module provides a simple way to automatically compute gradients, used to optimize model parameters via gradient descent, for any function operated within a neural network. Appending any tensor with requires_grad=True signals to autograd that every operation on that tensor should be tracked, which enables automatic differentiation.

• Optim modules apply optimization algorithms to those gradients. Torch.optim provides modules for various optimization methods, like stochastic gradient descent (SGD) or root mean square propagation (RMSprop), to suit specific optimization needs.

Dynamic computation graphs (DCGs) are how deep learning models are represented in PyTorch. Abstractly speaking, computation graphs map the flow of data between the different operations in a mathematical system: in the context of deep learning, they essentially translate a neural network’s code into a flowchart indicating the operations performed at each node and the dependencies between different layers in the network—the arrangement of steps and sequences that transform input data into output data.

What differentiates dynamic computation graphs (like those used in PyTorch) from static computation graphs (like those used in TensorFlow) is that DCGs defer the exact specification of computations and relationships between them until run time. In other words, whereas a static computation graph requires the architecture of the entire neural network to be fully determined and compiled in order to run, DCGs can be iterated and modified on the fly.

This makes DCGs particularly useful for debugging and prototyping, as specific portions of a model’s code can be altered or run in isolation without having to reset the entire model—which, for the very large deep learning models used for sophisticated computer vision and NLP tasks, can be a waste of both time and computational resources. The benefits of this flexibility extend to model training, as dynamic computation graphs are easily generated in reverse during backpropagation.

While their fixed structure can empower greater computational efficiency, static computational graphs have limited flexibility: for example, building a model that uses a varying number of layers depending on the input data—like a convolutional neural network (CNN) that can process images of different sizes—is prohibitively difficult with static graphs.

One extensively used method for training neural networks, particularly in supervised learning, is backpropagation. First, in a forward pass, a model is fed some inputs (x) and predicts some outputs (y); working backwards from that output, a loss function is used to measure the error of the model’s predictions at different values of x. By differentiating that loss function to find its derivative, gradient descent can be used to adjust weights in the neural network, one layer at a time.

PyTorch’s autograd module powers its automatic differentiation technique using a calculus formula called the chain rule, calculating complex derivatives by splitting them into simpler derivates and combining them later. Autograd automatically calculates and records gradients for all operations executed in a computational graph, greatly reducing the legwork of backpropagation. 

When running a model that has already been trained, autograd becomes an unnecessary use of computational resources. Appending any tensor operation with requires_grad=False will signal PyTorch to stop tracking gradients.

Working with the large datasets required to train deep learning models can be complex and computationally demanding. PyTorch provides two data primitives, datasets and dataloaders, to facilitate data loading and make code more easily readable.

• torch.utils.data.Dataset stores data samples and their corresponding labels
• torch.utils.data.Dataloader wraps an iterable—an object that can be operated upon—around the dataset to enable easy access to samples