Introduction to Tensors in Pytorch #1

A neural network is the bunch of nodes that take one or more input, processes them and returns one or more output. The data flow is done in form of tensors which we will learn in this post. Understanding tensors in PyTorch is an important and lengthy topic because while optimizing weights (or training), the backpropagation algorithm will apply some methods on these tensor objects to achieve its goal so I have divided it into two parts.  

I will keep this short and to the point discussing important things only. So let's dive deep into tensors

What are Tensors

Tensors are the building blocks of a neural network. A PyTorch tensor is the generalized form of arrays in \( n \) dimensions to run arbitrary computations on GPU. [READ MORE]. A tensor can be of any dimension. In this post, you will learn about the following dimensions only

• 0D -> Scalar numbers like 0, 10, 20 etc. are 0-Dimensional tensors
• 1D -> Array of numbers like [1, 2, 3] or [20, 30, 10] etc.
• 2D -> Matrix of numbers like [[1, 2, 3], [2, 3, 4]]

Before creating and using tensors in PyTorch, install it by following the instructions from here: https://pytorch.org/get-started/locally/

Creating Tensors

You can create the tensors by importing torch and using torch.tensor(data) method.

import torch

t0 = torch.tensor(10)                     # 0-D tensor
t1 = torch.tensor([1, 2, 3, 4])           # 1-D tensor
t2 = torch.tensor([[1, 2, 3], [2, 3, 4]]) # 2-D tensor

print(t0)
print(t1)
print(t2)


"""
Output:
tensor(10)
tensor([1, 2, 3, 4])
tensor([[1, 2, 3],
        [2, 3, 4]])
"""

To convert it into NumPy array, simply use Tensor.numpy() method and use it as you wish

print(t2.numpy())

"""
Output:
array([[1, 2, 3],
       [2, 3, 4]])
"""

Tensor Types and Their Conversions

PyTorch supports the following data types of the tensors

To get the data type used Tensor.dtype property and Tensor.type() to get the PyTorch tensor type

print(t2.dtype)
print(t2.type())

"""
Output:
torch.int64
torch.LongTensor
"""

The difference between dtype and type() is that the first one will return you the type of data in the tensor and the second one will return the type of tensor. In short, dtype will give you the same name of the type as you will see in NumPy whereas the type method will provide its class name developed by the original authors of PyTorch

To explicitly create a tensor of a specific type, either pass dtype while defining the tensor in the torch.tensor method or use a tensor constructor. Here I am passing the integer values with dtype=tf.float32. This will convert the integer data into floating numbers while creating tensors

tf0 = torch.tensor([1, 2, 3, 4], dtype=torch.float32)
print(tf0)
print(tf0.dtype)

tf1 = torch.FloatTensor([5, 6, 7, 8])
print(tf1)
print(tf1.dtype)


"""
Output:
tensor([1., 2., 3., 4.])
torch.float32
tensor([5., 6., 7., 8.])
torch.float32
"""

You can pass in the datatype in the Tensor.type(<new data type>) or Tensor.type_as(<new data type>) method, to change the data type of the existing tensor.

tf0 = tf0.type(torch.int32) # after changing, it returns a new tensor object of that type
print(tf0.dtype)

"""
Output:
torch.int32
"""

Shapes and Dimensions

While training the model, you will deal with higher dimensions and the neural network only accept the dimension which is defined at the input layer while architecting the model. The dimension tells whether the tensor is 0-D or 1-D or 2-D or even higher than that. The dimension of a tensor is also called the rank of the tensor.

To get the size you can use Tensor.size() method or Tensor.shape property and to get the dimension of the tensor, use Tensor.ndimension() method or Tensor.ndim property.

print(t1.size())
print(t1.ndim)

print(t2.shape)
print(t2.ndimension())


"""
Output:
torch.Size([4])
1
torch.Size([2, 3])
2
"""

Reshaping Tensors

Sometimes working with the neural network, you will face a situation when you are supposed to change the dimension of the tensor. This can be done by using Tensor.view(*shape) or Tensor.reshape(*shape) method of the tensor

For example, I have a tensor [[1, 2, 3], [2, 3, 4]] and I want to convert it to 2D with 1 row and 6 columns

tx = t2.view(1, 6)
print(tx)

"""
Output:
tensor([[1, 2, 3, 2, 3, 4]])
"""

There are special values for the reshape and view methods. Suppose you don't know how many rows would be good while resizing your matrix, you can set it to -1 it will automatically adjust the rows for you. The same goes for columns.

# Telling PyTorch to keep the max 2 elements in one column and make as many rows as you want
tr = t2.view(-1, 2)
print(tr)

# Telling PyTorch to keep max 2 rows and adjust columns automatically
tc = t2.view(2, -1)
tc


"""
Output
tensor([[1, 2],
        [3, 2],
        [3, 4]])
tensor([[1, 2, 3],
        [2, 3, 4]])
"""

Note – Both the rows and columns can't be set to -1 at the same time.