This Server CANNOT Lose Data…

when you make as many videos as we do

you need a lot of fast reliable storage

and our main editing server wanic has

checked all of those boxes for years

it's a great little server it's built

out of high quality components and it

even looks cool but as our team is grown

we've reached the point where even a

minute one single minute of downtime

costs over $50 and that's just in

payroll now practically speaking the way

to mitigate that is by adding redundant

now our drives are already redundant

we've got 20 drives in there with data

striping but the problem is they all sit

in one single server I'm sure you can

see where this is going it's been over a

year in the making but it's finally here

wck final form and I'm calling it wanic

10 because it's the last wever avability

W told you this like 10 times nobody

even knows what high availability means

it means it's lus just go ahead unplug

one do it go for it well okay I should

probably tell you the stakes before you

do that each of these two grand twin

boxes has four entire servers inside of

them that were provided by super micro

who sponsored this whole thing and

they're set up with WCA a redundant nvme

first file system in this config it

should sustain two entire servers

dropping out without anyone even

noticing except that we moved the entire

team onto it last night without telling

anyone and it's the middle of the work

day with a ton of high priority videos

in progress do you really want to test

it right now I like I haven't tried that

all right here we go okay what could go

wrong I mean a

[Applause]

lot naturally a huge part of a project

like this is the software the stuff

that's going to handle Distributing all

of ourish terabytes of video projects

Word documents and Linux isos to the

multiple machines that we just showed

you but we can't install any software

until we have some Hardware so why don't

we start there meet the super micro

Grand twin A+ server as- 2115 gt-

hntr despite its sort of ordinary

looking appearance and unexciting

sounding name it

is anything but ordinary and it is very

exciting because inside this 2u is four

independent Compu computers but for what

we're doing four nodes please we want

eight inside each of these is a

completely independent motherboard 384

gigs of memory an AMD epic Genoa

processor with 64 cores dual m.2 slots

for redundant boot drives six pcie Gen 5

2 and 1/2 in nvme slots up front and

we've got IO in the rear now this bit

here could be a little confusing at

first glance but that is because not

only do we have USB but we have two full

gen 5x6 pcie connections back here along

with display output and power for the

entire server this whole thing slides

into the chassis which holds a really

cool modular backplane assembly that

we'll take a look at in a minute and

then passes through thank you Jake ah to

the back at the server where you've got

a Management Port a single USB port for

each server nope it's two and they're

shared what the I was about to ask cuz

we've also got a single VGA you see the

button for two servers there no way this

button toggles

yeah and okay before we talk about that

a little bit more look at these power

supplies each of these is

2200 Watts 80 plus typ tianium which

sounds like a lot but when you're

potentially handling four 400 wat epic

Genoa CPUs along with a bunch of ram up

to 24 nvme drives and eight network

cards well it seems downright reasonable

doesn't it is it 24 drives can't be 6

yes 6 * 4 is

24 and of course that's just one of them

we've got two of those and that means

that in the event that one of these dies

the system should be able to continue to

operate uninterrupted which is a big

part of the high availability goal that

we have for this deployment speaking of

high availability let's move on to our

network cards each of those pcie gen 5x6

slots I showed you guys before

terminates in one of these ocp 3.0 small

form factor mezzanine slots and what

we're putting in them is these connectx

6 200 gbit cards from

melanox excuse me from Nvidia that okay

these are the older Gen 4 ones so

they're going to be limited by the slot

speed of around 250 gabit per second but

if we had newer cards that means that

each of these nodes could do 200 plus

another 200 400 up to

800 gigabit which would of course be a

complete waste for us a because our

workload can't take advantage of it and

B because our switch is only 100 gbit

sorry of course the two ports are still

helpful we do have redundant

switches except there's kind of a

problem here that's still a single point

of failure in a perfect world we would

have two single port Nicks so if a Nick

were to die it would still be okay but

because we have so many nodes we're not

really worried about an individual node

you know they could have one boot drive

and it die or one Nick and it die we

still have an extra backup how many

nines do you want I mean I don't know

like one would would be good 9% which

Jokes Aside is a really good point if we

were architecting this properly there

are so many more considerations that we

would need to make like the power coming

into the rack would have to come from

two independent backed up sources the

connectivity to our clients would have

to be redundant as well the connectivity

between all of the systems would have to

be architected in such a way that no

matter what fails everything will stay

up and realistically for us we're not

going to get that deep into it because

our goal is better than we had before

which was a single machine with its own

built-in redundancies but other than

that nothing now at least we should be

able to lose a full machine out of these

eight we can restart one of our core

switches totally fine two machines out

of these eight and we can still be

limping along I mean limping is a bit of

a stretch it's going to be very fast now

normally if you buy a super micro

machine they're going to pre-build it

for you they're going to validate it for

you you can even have them pre-build an

entire Rack or racks of these things and

then validate your application on it

before it ships to you in fact we've got

a whole video that we did about that

that was sponsored by super micro a

little while back of course this is LT

my friends so we will be assembling this

one ourselves do you like that spin of

the screwdriver above the server don't

worry I won't miss I'll never miss see I

could do this a hundred times and I

would never miss why no it's fine it's

good it's okay we have seven more any

who for our CPU we've gone with an epic

Genova

9534 this is a 64 core

128 thread monster of a CPU it'll do 3.7

GHz Max boost it has A4 gigabyte of

level three cache a 300 wat TDP it

supports ddr5 memory up to 12 channels

and it supports a whopping 128 Lanes of

pcie Gen 5 originally we were intending

to go with 32 core chips but they were

out of stock so free upgrade lucky us

compared to previous generation AMD epic

CPUs dooa is a big step up in terms of

IO performance which makes it perfect

for this application and in the long

term I mean if we've got all the extra

CPU cores and a whole bunch of ram

anyway why run WCA on the bare metal

when we could install prox Mox and then

use the other cores for I don't know

High

availability Plex server yeah Linux isos

more realistically it would be something

like active directory yeah which we

don't really want to do right now

because if you run active directory on

one server and it goes down you're going

to have a really really bad time but if

you run it on a bunch of servers yeah

it's good great so normally server CPU

coolers would come with their own

thermal paste pre-applied but since

we're doing this ourselves and uh if you

look carefully it's not the first time

that it's been installed we are going to

be using okay thank you for that a piece

of Honeywell PTM 7950 this stuff is

freaking awesome it has great thermal

transfer properties and it can handle

varying temperatures like seriously I

don't remember many not even just

varying but like a lot of huge cycles

for a very very long time now available

LTD store.com is that big enough does

that cover all of the ccds and

cxs oh there's a second piece of PL am I

stupid is there a second piece of

plastic no there isn't should I put one

in the fridge no no no it's totally fine

I've done this like a bunch of times

yeah oh she's Min look at that see all

right easy I would recommend putting it

in the fridge before you use it all

right to ensure we're making the

absolute most of our CPU especially in

this High throughput storage workload

we're going to be populating all 12 of

our memory Channels with 32 gig dims of

ddr5 ECC running at 4,800 megga

transitors per second that's a total

of 384 three terabytes of memory what

across all eight

oh each of the cables Jake removing

right now is a pcie by8 cable that feeds

two of the drive bays in the front but

the reason he's taking them out is that

we can install our boot drives these are

consumer grade each system is getting

two Sab 512 gig gen 3 rocket drives and

it's not because they're particularly

special in any meaningful way they're

not even that fast by modern standards

but what they are is from our experience

reliable enough and they are fast enough

for what we're going to be doing which

is just booting our operating system off

of them movie Magic all of the other

nodes are already built so what do you

mean movie Magic super micro built them

Oh I thought you buil them super micro

builds them for you I took it apart okay

fine I took that one apart no secrets

left anymore yep no Intrigue no mystery

you know what is still mysterious is

inside of here I've actually never

opened this before Oh okay let's have a

look woo holy oh that's power supplies

yeah this is so cool so the whole

computer is cooled by four fans no way

there's the two power supply fans and

then these fans in their what do they

call this like IO module I think is what

they call it look at the blades on this

thing counter rotating you're serious

that's what you're looking at not this

the most delicate of spaghet oh my God

there's not even connectors every one of

these wires is soldered directly to the

back of the ocp 3.0 what yeah for

storage we're installing ing two of

kokia Speedy cd6 Gen 4 and vme drives in

each node so we've got one that is 7

tabt and another one that is 15

terabytes they're kind of placeholders

for now and in the long term we're going

to switch to Something in the

neighborhood of about 4 15 tab drives

per node but the drives we want to use

are currently occupied by oh that

project by a top secret pastry related

project so that's going to have to wait

the good news is that when those drives

become available WCA supports live

upgrading and downgrading so we can just

pull these drives swap in the new ones

pull swap pull swap pull swap as long as

we uh don't do it all at once are we

ready to fire these things up okay

there's a lot going on here what is that

is that a switch y hey look you can see

the button now oh that's

cool what you're hearing so far is just

the Nvidia SN 3700 32 Port 200 gig

switch oh my God it even says melanox on

the front I know maybe it's an old like

review sample demo univ we got it with

the $1 million PC and I'm pretty sure

that that was already in video at that

point can you hear that you hear it

getting louder yeah

who well that one's just excited to see

this is the WKA dashboard maybe if I go

over here cluster servers we can see all

of our servers we have two drives per

and then course this is a very

interesting part of how wo works it's

not like trass let's say where it just

uses the whole CPU for whatever you're

trying to do they dedicate and like

fence off specific cores for specific

tasks for instance each Drive gets a

core so we've got two Drive containers

that means two a full core per Drive

yeah damn yeah you also have compute

cores which do like the par calculation

and intercluster communication and then

there's front end which you don't

necessarily always have frontend cores

managed connecting to a file system so

if you just had drives and Compu compute

you wouldn't be able to access the files

on this machine so you would have your

backend servers right those would run

drives and compute which is the cluster

and then on your like GPU box you would

run just the front end and that would

allow the GPU box to connect to the

backend cluster servers oh the back-end

cluster servers don't need to run a

front end unless you want to be able to

access the files on that machine or from

that machine which we want to cuz we're

using SMB we're using it as a a file

server stupid NZ for our stupid windows

machines yeah you can also have a

dedicated front end machine yes so if

you had like a 100 backend servers but

then that's adding a single point of

failure which is what we're trying to

avoid you could have multiple of them

okay you thought they thought of that

yeah I set it up so every single machine

in the cluster all eight of them are

part of our SMB cluster which means it

cannot go down realistically there are a

ton of other file systems out there that

you could use for something like this

traz has their scale out setup for

clustered ZFS which only requires three

nodes and is something we'd be quite

interested in trying out or if you're

looking for object storage there's a

million options but the main open-

source one Min iio requires only four

nodes though when we saw how nuts WCA

was when we set up the million dooll

server cluster I mean we had to try it

out for ourselves and try it out we did

so this is each not node holy

sh look okay the crazy thing is look at

the read latency now guys look look hold

on hold on hold on at 70 gabt a second

we've seen numbers like this before but

we're talking with in some cases double

the number of drives and no file system

without a file system like raw to each

drive this is with a file system with a

file system over a network and we're

only using 100 Gig ports like usually

with a WCA setup like this you'd

probably use 200 yeah cuz we oh my God

we didn't know cuz we didn't even have

networking as a factor last time all the

drives were in one box I know this is

networking too and the crazy part is

we're not using RDMA this is like um

some fancy uh what's it called dpdk I

think is the library this is wild yeah

look at that so read latency 131 microc

seconds that's 4 million read iops with

a latency of 1 millisecond average are

are we able to keep using W FS like this

is a trial okay this software is quite

expensive this is unreal 4 million iops

this is like it is unreal it's way more

than we could possibly ever need but

it's cool it's so cool don't they

support tearing and everything oh yeah

here I'll show you actually what that

looks like this is on mother vault which

I think right now has 400 Tippy bytes

left so let's say Max Capacity is 400

terabytes now once we run out of the 100

terab of SSD capacity which you can see

here it'll just it'll tear I mean it

automatically tear anyways and you do

need to make sure that your object store

is at least the same size as the flash

or bigger because they're going to

automatically tear everything to it that

makes sense so in theory we

move manually copy everything from Vault

one time to wo one time because it

stores in like 64 megabyte chunks and

then it just stays there forever stays

there forever and then we just have one

network share and when something needs

to get vaed you just you just move it

from allow it to Decay yeah you would

probably move it from pending projects

to like done or something like that we

make a folder for done yeah sure um and

then it will just do it automatically

wow or if it's a video that like

somebody was working on and then you

know it's been on hold for 3 months and

we shot you know a ter of footage it

will just and then when we're ready to

work on it it'll promote it back up holy

we K net boot off of this

followup video yeah I mean why not it's

so fast you literally could not we we

couldn't saturate this now a lot of you

at this point must be thinking gosh

Mister that's an awful lot of computers

for high availability couldn't you do

this with two and you're not that far

off the old school high availability net

app storage appliances like that one we

looked at recently did have just two

machines but those were both connected

to the same storage drives if each

system has its own drives when things

can get out of sync like let's say if

one machine has downtime you can run

into a situation where each system

believes with all the conviction in its

heart that it has the correct data and

then if all you have is two how will

they decide who's right this is

typically referred to as split brain and

that's why the majority of High

availability systems have at bare

minimum three servers this allows the

third system to be a tie breaker of

sorts in the case of a disagreement now

in our case WCA that stupid Ultra fast

file system that we're using which

unlike anything that we've used before

has been built specifically for nvme

drives not hard drives well it requires

a minimum of six nodes with a

recommendation of eight but running WKA

can still be an advantage video editing

with Adobe Premiere like we use is very

latency sensitive and even a small delay

when going to access a clip can be

enough to make the software crash so any

Improvement there is huge not to mention

that a pair of these Grand twins speced

out to the max with 128 car epic Berg

CPUs would get you just four rack units

with 1,000 CPU cores actually actually a

little more 24 terab of ddr5 and up to 3

pedabytes of ndme storage I mean h that

makes our setup seem downright

reasonable now the average W customers

are going to be a little more demanding

than us visual effect Studios AI

developers genomics Labs all the folks

out there that need Stupid Fast low

latency storage and WCA showed us

screenshots of clusters that were

reading in excess of 1 terabyte per

second consistently obviously that was a

bigger cluster but it shows you what can

be achieved with this kind of Hardware

running on I mean what used to be the

crappier option software raid man I feel

bad even calling it that these days I

had a interesting idea with the super

micro folks so you know how we have like

two pedabytes of 13 years worth of

footage thousands and thousands of hours

of footage thousands it's really cool

that we have it but it's really hard to

use unless you just happen to know what

video the thing you were looking for is

in well what if you could just like

search for something lonus Sebastian I

want every clip with lus Sebastian it

wow bam look at that shot up and let's

say you know there's this one that's uh

detected that it's you throughout the

entire clip yeah you're in a chair so

you could search for clips of lonus

sitting down with a keyboard yeah like

we're going to be able to actually find

stuff yeah right now there is a a finite

amount of objects that are trained I

mean chihuahua let me scroll through

this it's a lot eventually you'll be

able to train it and tell it hey this is

what a computer fan looks like or this

is what an SSD looks like oh my God that

is so cool so wait is this running on

these extra CPU cores or okay no not

right now faces and logos are running on

CPU yeah objects OCR and scenes run on

GPU got it but they're not running on

any of those machines they're running on

a GPU workstation that super micro sent

that's sitting at my desk um it was

Heavy anyways what is happening on that

new server is proxies because if we were

to analyze the original Clips oh AAL

formatting is a huge problem when you go

into an AI model it might not

necessarily support the Kodak that

you're filming in sure but also Clips

are like hundreds of megabytes a second

potentially that would take forever so

instead it generates proxies of

everything first first which we're

dumping to that new server and then we

can take advantage of the Lightning Fast

storage yeah you can we have 2.6 massive

compute and we can basically create like

a proxy map of what everything is in the

main archive right that is so cool so

far I've generated 2.6 terab of proxies

which might not sound like a lot but

they're only 5 megabit so it's actually

like a lot this is going to be a

flipping game Cher News

sports can you imagine your CNN you want

that person wearing a red tie yeah but

right now we've done 25,000 so 2.6 terab

is 25,000 Pro okay well let's try and

find something oh hold on once you've

generated a proxy you have to then

analyze it right ah so the analysis is

not done no not even close I've analyzed

22 Clips okay everything with Elijah

Elijah and this is the every clip that

Elijah's in and you can even see this is

so cool this is the actual ma'am as they

call it media asset manager the axle AI

guys built this before it was like AI as

far as I'm aware back when you would

have had to make comments like this

manually now it's just AI so all of the

data is in here now and we can see

here's Adam and Elijah oh that's so cool

here's all the different objects chair

flower pot microphone oh let me show you

the scene understanding thing cuz that

is so cool this is like brand new thing

they barely even worked it in but it

basic it basically takes a snapshot

every seconds two men are working on a

project in a room there is a speaker

stereo equipment there's a faucet

there's a tripod there's the tripod some

of these are a little less accurate two

men are working on a robot in a room it

kind of looks like a robot you I mean

yeah sure two men are in a workshop

looking at a laptop computer looking at

a machine there is person Alex Clark so

this is just running right now in real

time like more stuff is getting

processed as see here processing logos 9

there it is processing logos and faces

it's going to take a while yeah it's

going to take forever they're still

working on making it function on

multiple gpus so once we can get it

running on like four gpus say one GPU is

doing face detection one's doing scene

analysis one's doing object detection or

something like that we'll be able to go

a lot faster but right now it's just one

GPU got it but this is so cool all

that's left is to deploy it lonus had to

run away to do some other stuff so I've

hired some backup Cavalry Sean our

infrastructure administrator except

we've run into a bit of a problem lonus

and me and our Infinite Wisdom while we

were making this rack so much better ran

a bunch of cables right where we need to

put the server did we just start

unplugging no yeah how are we even going

to do this we have to like part the seas

exactly I started to try to move some of

the cables out of the way but they're

all twisted together so hopefully the

LTT Cable Management thing which you can

finally get at ltp store.com will save

us beautiful cable managed we can slide

a server in there now I hope you're in

yeah it's on ow ow ow ow ow ow ow okay

you're good just go that wasn't so bad

like made for it

next hey we're in now we just have to

run a million cables uhoh do you notice

anything different well it's loud most

of that's actually just the vent is on

one of the the air conditioners is

broken again but do you notice anything

different I mean the sticker's here that

that sticker's been there for years

seriously you haven't noticed anything

else well you guys uh screwed something

onto the oh did you put sauna Pan behind

it yeah but I thought this is supposed

to be a vented door my original plan was

to get rid of the vent that you put in

but that vent was there as a backup in

case the HVAC ever failed so that fan is

the exhaust and that's the intake you

see all the gaps F God there gaps but do

you notice the sound difference yeah

it's a big difference it's huge but that

server is so loud we basically ended up

where we

started yeah but that's okay I was just

trying to normalize I just mean I didn't

make it worse it's not that okay look at

that woo cute right God that's a lot of

metal if all goes to plan we could get

rid of this and this and just have these

so no more additional rack taken up