Complete Pytorch Tensor Tutorial (Initializing Tensors, Math, Indexing, Reshaping)

learning about the basic tensor

operation is an essential part of pi

torch and it's worth to spend some time

to learn it and it's probably the first

thing you should do before you do

anything related to deep learning what

is going on guys hope you're doing

awesome and in this video we're gonna go

through four parts so we're gonna start

with how to initialize a tensor and

there are many ways of doing that we're

gonna go through a lot of them and then

we're gonna do you know some tensor math

math and comparison operations we're

gonna go through tensor indexing and

lastly we're gonna go through tensor

reshaping and I just want to say that I

really encourage you to watch this video

to the end so you get a grasp on these

tensor operations even if you don't

memorize them after this video and

there's probably no way you can memorize

all of them you would at least know that

they exist and that will save you a lot

of time in the future so in this video I

will cover the basics but I will also go

a bit beyond that and show you a lot of

useful things or operations for

different scenarios so there's no way

I'm able to cover all of the tensor

operations there are a bunch of more

advanced ones that I don't even know

about yet but these are definitely

enough to give you a solid foundation so

I guess we'll just get started and the

first thing I'm going to show you is how

to create a tensor so what we're gonna

do is we're gonna do my tensor and we're

gonna do torch tensor and we're gonna do

the first thing we're gonna do is a list

and we're gonna do a list inside that

list we're gonna do one two three and

then let's do another list and we're

gonna do four five six so what this

means right here is that we're gonna do

two rows so this is the first row and

then this is the second row all right so

we're gonna have two rows in this case

and we're gonna have three columns

so we could do then is we can do print

my tensor and we can run that and we're

gonna just get in an in a nice format so

we're gonna get one for for the first

column two five and three six now what

we can do as well is we can set the type

of this tensor so we can do D type

equals torch dot float and we can do

let's say float 32 so then if we print

it again we're gonna get that those are

float values now another thing we can do

is we can set the device that this

tensor should be on either CUDA or the

CPU now if you have a CUDA enabled GPU

you should almost always have a sensor

on the on CUDA otherwise you're gonna

have to use the CPU but we can specify

this using the device so we can do

device equals then we can set this to

CUDA if you have that available and then

if we print my tensor again we can say

see that the device says CUDA right here

if you do not have a it could enable GPU

then you're gonna have to write CPU

right here I think that CPU is also the

default she don't have to write it but

it can help to just be specific now if

we run this the it doesn't the device

doesn't show which means that it's on

the CPU can also set other arguments

like requires gradient which is

important for auto grab which I'm not

going to cover in this video but

essentially for computing the gradients

which are used when we do the backward

propagation through the computational

graph to update our parameters through

gradient descent but anyways I'm not

gonna go in-depth on that one thing we

can do as well is that you're gonna see

a lot when in my videos and just if you

read PI torch code is that people often

write device equals CUDA if torch that

CUDA dot is available like that

else CPU all right and in this case what

happens is that if you have CUDA enabled

the device is going to be set to CUDA

and otherwise it's going to be set to

the CPU kind of that priority that if

you have enabled and you should use it

otherwise you're gonna be stuck with the

CPU but that's all you have so what we

can do instead of writing the string

here is that we can just write device

like this

and now the great thing about this is

that two people can run it and if one

has kuda it's going to run on the GPU if

they don't have it's gonna run in the

cpu but the code works no matter if you

have it or not

now let's look at some attributes of

tensors so what we can do is we can as I

said I print my tensor and we just I get

so we can do that again and we just get

some information about the tensor like

what device it's on and if it requires

gradient what we can also do is we can

do my tensor and we can do dot D type so

that would we'll just in this case print

towards up float32 and we can also do

print my tensor that device what that's

gonna do is gonna show us what there is

detention or zone so CUDA and then

you're gonna get this right here which

essentially if you have multiple GPUs

it's gonna say on which GPU it's on in

this case I only have one GPU so it's

gonna say 0 and 0 I think is the default

one if you don't specify and then what

we can do as well we can do print my

tensor dot shape so yeah this is pretty

straightforward it's just gonna print

the shape which is a two by three what

we can do as well is we can do print

might answer that requires grad which is

gonna tell us if that answer requires

gradient or not which in this case we've

set it to true alright so let's move on

to some other common initialization

methods

what we can do is if we don't have the

exact values that we want to write like

in this case we had one two three and

four five six we can do x equals torch

that empty and we can do size equals

let's say three by three and what this

is gonna do is it's gonna create a three

by three

tensor and our matrix I guess and it's

gonna be empty in that it's going to be

uninitialized data but these days this

data values that isn't gonna be in this

industry are gonna be just whatever is

in the memory at that moment so the

values can really be random so don't

think that this should be zeros or

anything like that

it's just gonna be unleash uninitialized

theta now if you would want to have

zeros you can do torched add zeros

and you don't have to specify the the

size equals since it's the first

argument we can just write three three

like that and that's gonna be so what we

can do actually we can print let's see

we can print X right here and we can see

what value is it gets and in this case

it actually got zero zeros but that's

that's not what it's gonna happen in

general and yeah if you print X after

this it's also going to be zeros now

what we can also do is we can do x

equals torch dot R and and we can do

three three again and and what this is

gonna do it it's gonna initialize a

three by three matrix with values from a

uniform distribution in the interval 0 &

1 another thing we could do is we could

do X equal torched at once I'm looking

again to I don't a by 3 and this is just

gonna be a three by three matrix with

all values of 1 another thing we can do

is torch that I and we can and this is

we're gonna send in five by five or

something like that and this is gonna

create an identity matrix so we're gonna

have ones on the diagonal and the rest

will be will be zeros so if you're

curious why it's called I is because I

like that is the identity matrix how you

write in mathematics and if you say I it

kind of sounds like I yeah that makes

sense

so anyways one more thing we can do is

we give you something like torch that a

range and we can do start we can do end

and we can do step so you know basically

the arrange function is exactly like the

range function in Python so this should

be nothing nothing weird one thing I

forgot to do is just print them so we

can see what they look like so if we

print that one as I said we're gonna

have once on the diagonal and the rest

will be 0 if we print X after the

arranged

it's going to start at zero it's going

to have a step of one and the end is a

non-inclusive v del v value or five so

we're going to have 0 1 2 3 4 so if we

print X we're gonna see exactly that 0

and then up to inclusive for another

thing we can do is we can do x equals

torsion linspace and we can specify

where it should start so we can do start

equals 0.1 and we could do something

like end equals 1 and we could also do

step equals 10 so what this is gonna do

is it's gonna write this should be steps

so what this is gonna do is it's gonna

start at 0.1 and then it's gonna end at

1 and it's gonna have 10 values in

between those so what's going to happen

in this case it's gonna take the first

value 0.1 then the next one it's going

to be a point to a point 3 0.4 etc up to

1 so just to make sure we can print X

and we see that that's exactly what

happens and if we calculate the number

of points so we have 1 2 3 4 5 6 7 8 9

10 right so we're gonna have it equal 2

steps amount of point so then we can do

also x equals torch that empty as we did

for the first one and we can set the

size and I don't know 1 and 5 or

something like that and what we can do

then is we can do dot normal and we can

do mean equals 0 and the standard

deviation equals 1 and essentially so

what this is gonna do it's gonna create

the initialized data of size 1 1 by 5

and then it's just gonna make those

values normally distribute normally

distributed with a mean of 0 and

standard deviation of 1 so we could also

do this with something like we can also

do something like with the uniform

distribution so we can do dot uniform

and then we can do 0 and 1 which would

also be similar to what we did up here

for the torch ran

but of course here you can specify

exactly what you want for the lower and

the upper of the uniform distribution

another thing we can do is to torch dot

d AG and then we can do torch dot ones

of some size let's say three it's going

to create a diagonal matrix of ones on

the diagonal it's going to be shape

three so essentially this is gonna

create a 3x3 diagonal matrix essentially

this is gonna create a an identity

matrix which is three by three so we

could adjust as well used I

but this diagonal function can be used

on on it on any matrix so that we

preserve those values across a diagonal

and in this case it's just simple to use

torched at once now one thing I want to

show as well is how to initialize tensor

to different types and how to convert

them to different types

so let's say we have some tensor and

we're just going to do a torch dot a

range of 4 so we have 0 1 2 3 and yea so

here I set to start the end and this

step similarly to Python the step will

be 1 by default and the start will be 0

by default so if you do a range for

you're just gonna do that's the end

value now I think this is this is

initialized as a 64 by default but let's

say we want to convert this into a

boolean operator so true or false what

we can do is we can do tensor dot bool

and that will just create false true

true true so the first one is 0 that's

gonna be false and the rest will be

through true now what's great about

these as I'm showing you now when you

dot boo and I'm also going to show you a

couple more is that they work no matter

if you're on CUDA or the CPU so no

matter which one you are these are great

to remember because they will always

work now the next thing is we can do

print tensor that's short and what this

is gonna do is it's gonna create two

int16

and I think both of these two are not

that often used but they're good to know

about and then we can also do tensor dot

loan and what this is gonna do is it's

gonna do it to in 64 and this one is

very important because this one is

almost always used we're going to print

tensor 1/2 and this is gonna make it to

float 16 this one is not used that often

either but if you have well if you have

newer GPUs in in the 2000 series you can

actually train your your networks on

float 16 and that's that that's when

this is used quite often but if you

don't have such a GPU I don't have that

that new of a GPU then it's not possible

to to Train networks using float 16 so

what's more common is to use tenser dot

float

so this will just be a float 32-bit and

this one is also super important this

one is used super often so it's good to

remember this one and then we also have

tensor dot double and this is gonna be

closed 64 now the next thing I'm gonna

show you is how to convert between

tensor and let's say you have a numpy

array so we'll say that we import numpy

as MP now let's say that we have some

numpy array we have numpy zeros and I'd

say we have a 5 by 5 matrix and then

let's say we want to convert this to a

tensor well this is quite easy we can do

tensor equals Torche dot from numpy and

we can just sending that numpy array

that's how we get it to a tensor now if

you want to convert it back so you have

a a back the number array we can just do

let's say numpy array back we can do

tensor dot numpy and this is gonna bring

back the number array perhaps there

might be some numerical roundoff errors

but they will be exactly identical

otherwise so that was some how to

initialize any tensor and some other

useful things like converting between

other types in float and double and also

how to convert between numpy arrays and

tensors now we're going to jump to

tensor math and comparison operations so

we're gonna first initialize two tensors

which we know exactly how to do at this

point we're going to torch that tensor

one two three and we're gonna do some

y2b torsa tensor and then I don't know

nine eight seven and

we're going to start real easy so we're

just going to start with addition now

there are multiple ways of doing

addition I'm going to show you a couple

of different points so we can do

something like is that one to be torched

empty of three values then we can do

torch that add and we can do X Y and

then we can do out equals Z one now if a

print said one we're going to get 10 10

and 10 because we've added these these

together and as we can see 1 plus 9 is

10 2 plus 8 and 3 plus 7 so this is one

way another way is to just do Z equals

torch dot add of x and y and we're gonna

get exactly the same result now another

way and this is my preferred way it's

just to do X Z equals X plus y so real

simple and real clean and you know these

are all identical so they will do

exactly the same operations and so in my

opinion there's really no way no reason

not to use just the normal addition for

subtraction there are again other ways

to do it as well but I recommend doing

it like this so we do Z equals X minus y

now for division this is a little bit

more clunky in my opinion but I think

they are doing some changes for future

versions of Pi torch but we can do Z

equals torch dot true divide and then we

can do x and y what's what's going to

happen here is that it's going to do

element wise division if they are of

equal shape so in this case it's going

to do 1/9 as its first element 2 divided

by 8 3 divided by 7 let's say that Y is

just an integer so Y is I don't know -

then what's gonna happen it's gonna

divide every element in X by 2 so it

would be 1/2 3/2 and 3/2 if Y would be

in

now another thing I'm gonna cover is in

place operations so let's say that we

have T equals towards that zeros of

three elements and let's say we want to

add X but we want to do it in place and

what that means it will mutate the

tensor in place so it doesn't create a

copy so we can do that by T dot ad

underscore X and whenever you see any

operation followed by an underscore

that's when you know that the operation

is done in place so doing these

operations in place are oftentimes more

computationally efficient another way to

do in place is by doing T plus equals x

so this will also do an in place and

ition similarly as ad underscore

although perhaps confusingly is if you

do T equals T plus X it's not going to

be in place that's going to create a

copy first and yeah I'm gonna expert on

this particular subject so that's just

what I know to be the case to move along

let's look at exponentiation so let's

say that we want to do element wise

exponentiation how we can do that is by

doing Z equals X dot power of two so

what that means is since X in this case

is one two and three and we're doing a

power of two this is going to be element

wise a power of two so it's going to

become one four and nine and we can

print that just to make sure so then we

get one four and nine and another way to

do this which is my preferred way of

doing it is doing is e equals x asterisk

asterisk two so this is going to do

exactly the same operation just without

dot power let's do some simple

comparison so let's say we want to know

which values of X are greater than 0 and

less than zero so how we can do that is

by doing Z equals x greater than zero so

we can just do print said

and this is going to again also be

elementwise comparison so this is just

going to be true since all of them are

greater than zero and if we do something

like that equals x and then less than

zero those are all going to be false

because all of the elements are greater

than zero so that's just how you do

simple comparison let's look at matrix

multiplication so

can do that is we if we if we initialize

two matrices so we have x1 torch that

Rand and then we do 2 and 5 and then we

do x2 and then we do to a tractor trend

and then 5 and 3 how we do the matrix

multiplication as we do something like X

3 equals torch mm X 1 and X 2 and then

the outer shape of this will just be 2x3

an equivalent way of doing the matrix

multiplication is you can do X 3 equals

x 1 dot mm and then X 2 so these are

really equivalent

eeeek you write out the torch dot and

then or you just do the tensor and then

it's a it has this operation as an

attribute to that tensor now let's say

that we want to do some matrix

exponentiation meaning so we don't want

to do element wise exponentiation but

rather we want to take the matrix and

raise the entire matrix so how we can do

that let's do matrix exponent and let's

just pick let's just initialize it to

torture brand of five and five

and then we can do something like matrix

dot X or underscore X and then matrix

underscore power and then we can send in

the amount of times to erase that matrix

so we can do something like three and

this would be equivalent of course to

doing matrix exponent and the matrix

multiply by itself and then matrix

multiply it again by itself if we're

curious how they look like we can just

print it and it's going to be the same

shape so it's going to be a five by five

and yeah these values don't really mean

anything because their values are all

random but at least we can see that it

sort of seems to make sense if we do the

matrix multiplication three times we

will probably get something that looks

like that now let's look at how to do

element wise multiplication so we have X

and we have Y so we have one two three

nine eight seven what we can do is we

can do Z equals x and then just star Y

so that would just be an element wise

multiplication and so if we print said

that should be you know 1 times 9 and

then 2 times 8 so and then 3 times 7 so

it should so it should be 9 16 and 21 so

if we print that we see that we get

exactly in 9/16 and 21 as we expect

another thing we can do is the dot

product so essentially that means taking

the element wise multiplication then

taking the sum of that but we can do

that instantly by doing torch dot dot

and then x and y so that would just be

the sum of 21 16 and 9 so we can just do

print that for that and we can see that

this sum is 46 something a little bit

more advanced is a batch matrix

multiplication and I'm just gonna

initialize so I'm just going to

initialize the matrices first or tensors

I guess so we're gonna do batch equals

32 and equals 10 M equals 20 P equals 30

so this doesn't make any sense yet but

we're gonna do tensor one and we're

gonna do torch dot Rand and then we're

gonna do batch and then N and then M so

we have three dimensions for this tensor

essentially that's what means with when

we have batch matrix multiplication if

we just have two dimensions of the

tensor then we can just do normal matrix

multiplication right but if we have this

additional additional dimension for the

batch then we're gonna have to use batch

matrix multiplication so let's define

the second tensor so that's gonna be

tortured Rand and it's gonna be badge

and it's gonna be M and it's gonna be P

if we have the tensors in structured in

this shape then we're gonna do out after

batch mates from application it's just

going to be torch that bmm of tensor 1

and tensor to and so what happens here

is that we can see that the dimension

that match there are equal are these

ones right here so it's gonna do matrix

multiplication across that dimension so

the resulting shape in this case is

going to be badge and MP so we're gonna

do we're gonna get batch and then N and

then P next I want to cover something

else which is some examples of a concept

called broadcasting that are you gonna

encounter a lot of times in both numpy

and PI torch so let's say we have two

tensors we have X 1

which is just a random uniformly random

5x5 matrix and then we have X 2 which is

torch dot R and and let's say it's 1 by

5 now let's say that we do Z equals x 1

minus X 2 and mathematically that

wouldn't make sense right we can't

subtract a matrix by a vector but this

makes sense in pi torch and numpy and

why this makes sense or how does it make

sense it makes sense in that what's

going to happen is that this row is

going to be expanded so that it matches

the row of the first one in this case so

this one is going to be expanded so we

have five rows where each column are

identical to each other and then the

subtraction works so in other words this

vector here is going to be subtracted by

each row of this matrix that's what we

refer to as broadcasting when it

automatically expands for one dimension

to match the other one so that it it can

actually do the operation that we're

asking it to do we can also do something

like X 1 and then element wise

exponentiation 2 X 2 so again

mathematically this doesn't make too

much sense we can't raise this matrix

element wise by something that doesn't

match in the shape but but it's going to

be the same thing here in that it's

gonna copy across all of those rows that

we wanted to element wise raise it to so

in this case it's gonna be again be a 5

by 5 and then it can element wise raised

from those elements now let's look at

some other useful tensor operations we

can do something like sum of X which is

going to be torched after some and then

we can do X and we can also specify the

dimension that it should do the

summation in this case X is just a

single vector so we can just do

dimension equal 0 although if you have a

like we did here in tensor one where we

have three dimensions of the tensor you

can

specify which dimension it should sum

over another thing we can do is we can

do torch that max so torch like Max is

gonna return the values and the indices

of the maximum values so we're gonna do

tours of Max of X and then we also

specify the dimension for which we want

to take the maximum again in this case

we just have a vector so the only thing

that makes sense here is dimension 0 you

can also do this same thing values

indices but you can do the opposite so

the torch type min instead of torch at

max then we can again do X dimension

equals 0 we can compute the absolute

value by doing torch that absolute

absolute value or torch with ABS of X

and that's gonna take the absolute value

element wise for each in X we could also

do something like torch dot arguments of

X and then we specify the dimension so

this would do the same thing as torch

dot max except it only returns the the

index of the one that is the maximum I

guess this would be a special case of

the max function we could also do the

opposite so we can do torch that

argument of X and then across some

dimension we could also compute and I

know these are a lot of operations but

so we can also do the mean so we can do

towards that mean but to compute the

mean Python requires it to be a float so

what we have to do then is X dot float

and then we specify the dimension in

this case again we only have dimension 0

if we would for example 1/2 element wise