Beginner's Guide to ControlNets in ComfyUI

so here we are in comfy again and we're

going to look at very briefly at another

aspect of comfy and you'll soon come

across I'm sure which are control Nets

and these are control Nets you'll I'll

put this workflow in and uh if these

come up with uh red then you go to

manager and install missing nodes you

will however also need to go and get the

control net models to get those you go

to manager install models and search for

control net so there are a lot of

different models and there are a lot of

different kinds of control net I'm going

to only deal with canny which is a line

extractor which essentially makes a

black line drawing of your image and the

depth once you have grasped how they

work the other ones all work the same

and and I think you'll be able to find

find your way through them but there's

there's poses and uh all sorts in there

there's quite a few different ones

there's too many to go through so I'm

I'm going over the underlying principle

of uh how they work they basically all

look at different aspects of an image

and use that to control uh what the

generation looks like so one aspect of

control Nets is that they need

pre-processing in now you can hook up

the pre-processor to feed directly into

it I don't I always I don't I always

pre-process

preprocess I going messed that and um

and then load the image in because in

the pre here you want to you might want

to mess around with uh these numbers

here which we'll we'll go into in a

second so it's not necessarily one shot

put it in and it comes out the other end

and then going to apply the control net

here so first of all uh this is a video

that comes after putting lauras in so we

have luras in here from the previous

video and the previous video you should

probably watch them in sequence but you

don't have to right so first of all uh

I'm assuming you've been to manager and

installed all the bits and pieces you

need so you'll be faced with this when

you've found the nose or when you

download my workflow so I'll just go

over how to Plum it in it is very simple

indeed so your input image your depth

map goes in there your control net model

goes in there and then you run your

positive from your you see the

conditioning here and the

positive and then the

positive back into the sampler and

exactly the same thing with the negative

negative back into the

S and that's it that's your control net

wi up so what we need to do we're going

to work with this image here but any

image so we're going to do a landscape

image so the first thing to do is to

turn off these so you go right click on

your group see group here and you can go

set group to never you see that set

group to never and then this group won't

do anything it'll just sit there and you

can do the same with this one though you

don't really need it so now we're just

going to work to make our depth M so we

choose our file there it is and we'll

just run that and see what comes out

there you go so that's what looks that's

what a depth map now you can alter these

numbers here this is the resolution it

looks at the image so it's breaking it

up into 512 squares you don't really

need to know about that and this is the

background the threshold so this is how

black it is and how white it is and so

on I would leave these settings uh alone

and you can mess with them afterwards

once you these images the thing is to

change these numbers and see what happen

so the way this works is pale stuff the

whitest stuff is the nearest stuff and

the darkest stuff is the further away

stuff okay so uh this image you can

right click it and go open image and

then you can drag your your image off in

into your uh wherever you're doing your

project so you can drag it off into your

depth maps whatever or this image will

be in your temp fold so if we turn this

off to never turn this on always so if

you go to your temp

folder there's your image in your temp

folder and there's your image loaded so

we're doing pretty well now we'll look

at the settings and we'll try everyone

and see what they do here you can use a

mask but I won't go into that I'm going

to deal with masking in a separate video

entirely as a subject by itself because

it's what masking is something that

people find quite difficult I think so

we doing pretty well we got our control

net loaded plumbed in and our map made

and loaded so next we'll look at what

these numbers here do so some of them

are fairly simple strength is obvious

higher the number the stronger the

influence start and end percent is a

little more complicated in the first

video we went through what all these

numbers so in the very first video you

can look at you'll find out what all

these numbers do so if you don't

understand what these numbers do go and

look at the image to image video and

it'll tell you what they are so I can't

go through everything on each video from

the start so I've broken this into

separate video so you can go back and

look at this video and then you'll know

what all these numbers do so we're using

a turbo so we've only got 10 steps so we

have 10 steps so this range here it's

going to start from the very beginning

at the first step and it's going to stop

its influence at the sixth step and so

you can alter that you can start at 5

end 10 you can play around with that and

each of those has a different result so

for this I'm going to turn off the land

Decker so you right click on the Laura

and go bypass and then that Laura won't

make anything any difference at all CU

that one's not working we'll bump the

other Laura up and you need to look at

my video on luras if you want to know

how luras work so we do the prompt and

I've done a video on prompting earlier

so you can go and look at that and

you'll learn something about prompt I'm

going to start by prompt with with a

simple description of what we have so

we'll put the image by The Prompt so

it's a fairy tale Castle we have a lake

or a river what do you think it is I

think it's a lake a still Lake fantasy

illustration is fine we don't have JC

ler anymore we have digital painting

that's the keyword for the Laura and we

have Morning Sun quite soft light and

probably initially that's all we want to

say about it should we say distant Hill

we can take that out of there so nothing

in in the uh in the negative we might

put something we'll put photograph as we

don't want to photograph if you want to

photograph you put photograph in here

obviously so there we go so we need a d

noise we want to change this quite a lot

so we go I think to 65 and there we are

all ready to go we got a fixed seed

seeds was dealt with in the first video

as well so that's it the whole lot is

ready to go so here we are with that

back and there's our image where which

as you see has taken all the color from

this and it's taken the structure from

this which is quite nice one thing I

will warn you is that control Nets are

quite computationally expensive so they

will slow you down if you have a monster

graphics card you'll be fine so this is

being influenced both by the control net

and I didn't go through choosing the

model but you have to choose a depth

model so there's all different models

here there's two depth Ms I think yeah

so there's depth of rank 1 to8 I think

1:8 is slightly less

expensive in Compu in computational time

but uh 256 is better in quality right I

think uh yeah 256 levels of gray and

that's 128 levels of gray I think

that'll be what it is I'm not

certain I guess there but that makes

sense so now we want to see what it does

uh without the influence of this so at

the moment you can see it's very much

taken the color for this but there's no

color information in here so we can turn

it off essentially by taking the D up to

one which means that this image will

have no influence whatsoever and we'll

run that again so here it is back and

here's the difference in the image got

see got a bit of cast in the sky there

so I think you should be able to see

that we still have the same composition

but it doesn't know anything about

what's happening in the background

really hence the hence the funny C

Castle so I'm putting it in at very

strong levels at present so that's uh

that's the result of uh putting things

at at high levels but I think you can

see let's bring the original uh image up

close so you can see we do have a

relationship you can see that this all

follows pretty much the same silhouette

here and we've just got this has all got

prettified I have no idea where that

castle look it really likes putting

Castles in the Sky but I suppose I put

fair I put fairy tale in so I've only

myself to blame so on the next part

we'll look at what canny does which um

makes outlines around things so we'll do

that next and with this you can you can

play with the strength of this against

the strength of this image here so you

can you can with this you control the

strength of this image and with this you

control the strength of the depth map so

you can balance the two together and uh

uh and get a result that you're like you

don't have to put the same image you

made the depth at from in here you could

put a a a mood image in here that was um

entirely different but just had lighting

or colors that you liked but if you just

the less like this image it is the more

the higher the D noise has to be cuz it

has to change it a lot it'll struggle uh

with a a different image and uh loading

noise cuz it won't CU it won't know what

to do so it won't work very well but um

if you use 75 which I did in the last

one and put a an image of of a different

landscape but with nice lighting that

you liked or nice colors then that would

be reflected and you'd get this Castle

lit different which is rather nice okay

so we'll move on to caddy which is quite

a different thing so here we are we're

going to do canny next and as see I've

moved my depth preprocessor over there

and we've got a canny there these will

all be in the workflow so you can stop

them over so we don't want this to run

so we turn to never and that one to

never as well and we just want to

process this image so we got a an image

uh of a lady sitting and it and it's a

bronze so it's quite hard to uh it's

quite hard to put that into image to

image and uh and get a uh you know a

normal colored illustration so run that

and there we have our cany map which is

just a series of lines that it will try

and fit the uh generation so this image

will now be in your temp folder so we

can turn this off turn this on go to

always and if we go to our temp folder

there it is so we've got our image

loaded in and we must change the control

or a model here to canny so you see

canny canny so this model understands

this sort of line work I'm going to uh

use this image as a as a color image but

first we'll run it uh without so

obviously Castle won't do anymore so

we'll say a uh a woman sitting and we'll

put her in a garden cuz we've got uh all

of this uh stuff in in here so we'll put

garden and we'll put Autumn and fantasy

illustration is fine I don't think we

want to still Lake Morning Sun yes

that's good soft light that's good

distant

Hill I will leave the distant Hill why

not so that should all be good and you

see G no is at one so it's at the moment

it's ignoring that image so and the

settings here are the same you know you

have to play with those to uh then

they'll all have different effects so

here we are she's back and there she is

sitting so you can see how it's stuck to

the relationship there we've got

everything we've asked for I think we

got a garden we got Autumn we got the

young lady it's fantasy Morning Sun yes

so now uh we can use this image to to

influence a little so we can lower our

do noise to what should we take it to

we'll take it to 80 so the this and the

prompt so the what what's called the

conditioning essentially is doing 80% of

the heavy lifting and this image is

doing only 20% and we ought to reflect

something about this in the prompt so as

this is a watercolor we might add

watercolor and that and that uh that

should help the influence of that

picture cuz we're using this as a style

image so we'll cue that and here she is

back and you see that this has

influenced the image we still got the

same pose and we' got a waterc fe now if

you want to take that image a little

further I would add another Laura and

this has uh the feel of um Al mut a

little bit so you know it could be so

what we could do is add add an alons so

we'll un bypass our Laura choose alons

Musha alons wants to low level so we

have to put the keyword for the Laura in

so that's the keyword that activates the

Laura okay and I think I think we can

afford um to go to 75 on the doo so this

SC this a little bit more influen and we

leave this the same this is doing pretty

well so we'll cue that and here we go

you see the influence of Al FAL buer and

because I went down to 75 here we've now

got our Castle in that position so we

can now clearly see we've got all of our

inferences going we've got the pose from

this lady we've got I think with a few

iterations you might get a more graceful

pose she's a bit uh she's okay I would

do a few iterations to get a slightly

different POS but um but for this

demonstration I think that shows it it

all works pretty well so this tree has

appeared so our Castle has appeared in

the same position as here all our

lighting and colors are coming from this

and all our pose information and subject

information is coming from here okay

that's it for uh control Nets there are

other control net they all work in

pretty much the same way that's an

animation one I don't know anything

about that open pose uh you get sort of

a stick figure there's a pre-processor

for that there's canny there's depth

there's a recolor one I've never used

that but I assume that's the same thing

there's sketch which is like a crude

form of canny and these are all

different types of control net I only

use canny and depth myself but uh I

occasionally use open pose but um anyway

they all work pretty much the same as's

a pre-processor and uh then you load the

appropriate model okay I hope that was

interesting and helpful thank you very

much