Practice The building blocks of artificial neural networks: tensors

After you ssh to the training cluster, activate the virtual environment that you created so that you have access to the Python packages you installed in it:

$ source ~/env/bin/activate

For instance:

$ salloc --cpus-per-task=1 --mem=3G --time=0:30:0

You can now start an interactive Python session:

$ python

First, we need to load the torch package:

import torch

PyTorch's tensors are homogeneous multidimensional arrays.

You can create them with a variety of methods such as:

• torch.rand, for a tensor filled with random numbers from a uniform distribution on \([0, 1)\)
• torch.randn, for a tensor filled with numbers from the standard normal distribution
• torch.empty, for an uninitialized tensor
• torch.zeros, for a tensor filled with \(0\)
• torch.ones, for a tensor filled with \(1\)

Each element you pass to these methods represents the length of one dimension. Consequently, the number of elements determines the number of dimensions of the tensor.

Let's have a look at a few examples:

print(torch.rand(1))

This is a one-dimensional tensor. Its length in the unique dimesion is 1. So it is a tensor with a single element.

When a tensor has a unique element, that element can be returned as a number with the method item:

print(torch.rand(1).item())

print(torch.rand(2))

This is another one-dimensional tensor. Its length in the unique dimesion is 2.

print(torch.rand(3))

A one-dimensional tensor. Its length in the unique dimesion is 3.

print(torch.rand(1, 1))
print(torch.rand(1, 1).item())

A two-dimensional tensor. Its length in one dimesion is 1 and its length in the other dimesion is also 1. So this is also a tensor with a single element.

print(torch.rand(1, 1, 1))

A three-dimensional tensor with a single element.

print(torch.rand(3, 1))

A two-dimensional tensor. Its length in one dimension is 3 and in the other, 1.

print(torch.rand(2, 6))

A two-dimensional tensor. Its length in one dimension is 2 and in the other, 6.

print(torch.rand(2, 1, 5))

A three-dimensional tensor. Its length in one dimension is 2, in a second dimension it is 1, and in the third dimension it is 5.

Play with a few more examples until this all makes sense:

print(torch.rand(2, 2, 5))
print(torch.rand(1, 1, 5))
print(torch.rand(1, 1, 5, 1))
print(torch.rand(2, 3, 5, 2))
print(torch.rand(2, 3, 5, 2, 4))
print(torch.rand(3, 5, 4, 2, 1))

print(torch.rand(3, 5, 4, 2, 1).dim())

print(torch.rand(3, 5, 4, 2, 1).size())

x = torch.rand(2, 4)
print(x)

y = torch.zeros_like(x)
print(y)

x.size() == y.size()

Let's take the addition as an example:

Note: you need to have tensors of matching dimensions.

x = torch.rand(2)
y = torch.rand(2)

print(x)
print(y)

The addition can be done with either of:

print(x + y)
print(torch.add(x, y))

print(x)

x.add_(y)
print(x)

x.zero_()
print(x)

PyTorch has a dtype class similar to that of NumPy.

You can assign a data type to a tensor when you create it:

x = torch.rand(2, 4, dtype=torch.float64)

To check the data type of a tensor:

print(x.dtype)

You can also modify it with:

x = x.type(torch.float)
print(x.dtype)

Indexing works as it does in NumPy:

x = torch.rand(5, 4)
print(x)

print(x[:, 2])
print(x[3, :])
print(x[2, 3])

You can change the shape and size of a tensor with the method view:

Note: your new tensor needs to have the same number of elements as the old one!

print(x.view(4, 5))
print(x.view(1, 20))
print(x.view(20, 1))

You can even change the number of dimensions:

print(x.view(20))
print(x.view(20, 1, 1))
print(x.view(1, 20, 1, 1))

When you set the size in one dimension to -1, it is automatically calculated:

print(x.view(10, -1))
print(x.view(5, -1))
print(x.view(-1, 1))

Tensors can be sent to a device (CPU or GPU) with the to method:

x = torch.rand(5, 4)
x.to('cpu')
x.to('cuda')

Of course, if you don't have GPUs, that last one won't work.