This Server CANNOT Lose Data…

when you make as many videos as we do

you need a lot of fast reliable storage

and our main editing server wanic has

checked all of those boxes for years

it's a great little server it's built

out of high quality components and it

even looks cool but as our team is grown

we've reached the point where even a

minute one single minute of downtime

costs over $50 and that's just in

payroll now practically speaking the way

to mitigate that is by adding redundant

now our drives are already redundant

we've got 20 drives in there with data

striping but the problem is they all sit

in one single server I'm sure you can

see where this is going it's been over a

year in the making but it's finally here

wck final form and I'm calling it wanic

10 because it's the last wever avability

W told you this like 10 times nobody

even knows what high availability means

it means it's lus just go ahead unplug

one do it go for it well okay I should

probably tell you the stakes before you

do that each of these two grand twin

boxes has four entire servers inside of

them that were provided by super micro

who sponsored this whole thing and

they're set up with WCA a redundant nvme

first file system in this config it

should sustain two entire servers

dropping out without anyone even

noticing except that we moved the entire

team onto it last night without telling

anyone and it's the middle of the work

day with a ton of high priority videos

in progress do you really want to test

it right now I like I haven't tried that

all right here we go okay what could go

wrong I mean a

[Applause]

lot naturally a huge part of a project

like this is the software the stuff

that's going to handle Distributing all

of ourish terabytes of video projects

Word documents and Linux isos to the

multiple machines that we just showed

you but we can't install any software

until we have some Hardware so why don't

we start there meet the super micro

Grand twin A+ server as- 2115 gt-

hntr despite its sort of ordinary

looking appearance and unexciting

sounding name it

is anything but ordinary and it is very

exciting because inside this 2u is four

independent Compu computers but for what

we're doing four nodes please we want

eight inside each of these is a

completely independent motherboard 384

gigs of memory an AMD epic Genoa

processor with 64 cores dual m.2 slots

for redundant boot drives six pcie Gen 5

2 and 1/2 in nvme slots up front and

we've got IO in the rear now this bit

here could be a little confusing at

first glance but that is because not

only do we have USB but we have two full

gen 5x6 pcie connections back here along

with display output and power for the

entire server this whole thing slides

into the chassis which holds a really

cool modular backplane assembly that

we'll take a look at in a minute and

then passes through thank you Jake ah to

the back at the server where you've got

a Management Port a single USB port for

each server nope it's two and they're

shared what the I was about to ask cuz

we've also got a single VGA you see the

button for two servers there no way this

button toggles

yeah and okay before we talk about that

a little bit more look at these power

supplies each of these is

2200 Watts 80 plus typ tianium which

sounds like a lot but when you're

potentially handling four 400 wat epic

Genoa CPUs along with a bunch of ram up

to 24 nvme drives and eight network

cards well it seems downright reasonable

doesn't it is it 24 drives can't be 6

yes 6 * 4 is

24 and of course that's just one of them

we've got two of those and that means

that in the event that one of these dies

the system should be able to continue to

operate uninterrupted which is a big

part of the high availability goal that

we have for this deployment speaking of

high availability let's move on to our

network cards each of those pcie gen 5x6

slots I showed you guys before

terminates in one of these ocp 3.0 small

form factor mezzanine slots and what

we're putting in them is these connectx

6 200 gbit cards from

melanox excuse me from Nvidia that okay

these are the older Gen 4 ones so

they're going to be limited by the slot

speed of around 250 gabit per second but

if we had newer cards that means that

each of these nodes could do 200 plus

another 200 400 up to

800 gigabit which would of course be a

complete waste for us a because our

workload can't take advantage of it and

B because our switch is only 100 gbit

sorry of course the two ports are still

helpful we do have redundant

switches except there's kind of a

problem here that's still a single point

of failure in a perfect world we would

have two single port Nicks so if a Nick

were to die it would still be okay but

because we have so many nodes we're not

really worried about an individual node

you know they could have one boot drive

and it die or one Nick and it die we

still have an extra backup how many

nines do you want I mean I don't know

like one would would be good 9% which

Jokes Aside is a really good point if we

were architecting this properly there

are so many more considerations that we

would need to make like the power coming

into the rack would have to come from

two independent backed up sources the

connectivity to our clients would have

to be redundant as well the connectivity

between all of the systems would have to

be architected in such a way that no

matter what fails everything will stay

up and realistically for us we're not

going to get that deep into it because

our goal is better than we had before

which was a single machine with its own

built-in redundancies but other than

that nothing now at least we should be

able to lose a full machine out of these

eight we can restart one of our core

switches totally fine two machines out

of these eight and we can still be

limping along I mean limping is a bit of

a stretch it's going to be very fast now

normally if you buy a super micro

machine they're going to pre-build it

for you they're going to validate it for

you you can even have them pre-build an

entire Rack or racks of these things and

then validate your application on it

before it ships to you in fact we've got

a whole video that we did about that

that was sponsored by super micro a

little while back of course this is LT

my friends so we will be assembling this

one ourselves do you like that spin of

the screwdriver above the server don't

worry I won't miss I'll never miss see I

could do this a hundred times and I

would never miss why no it's fine it's

good it's okay we have seven more any

who for our CPU we've gone with an epic

Genova

9534 this is a 64 core

128 thread monster of a CPU it'll do 3.7

GHz Max boost it has A4 gigabyte of

level three cache a 300 wat TDP it

supports ddr5 memory up to 12 channels

and it supports a whopping 128 Lanes of

pcie Gen 5 originally we were intending

to go with 32 core chips but they were

out of stock so free upgrade lucky us

compared to previous generation AMD epic

CPUs dooa is a big step up in terms of

IO performance which makes it perfect

for this application and in the long

term I mean if we've got all the extra

CPU cores and a whole bunch of ram

anyway why run WCA on the bare metal

when we could install prox Mox and then

use the other cores for I don't know

High

availability Plex server yeah Linux isos

more realistically it would be something

like active directory yeah which we

don't really want to do right now

because if you run active directory on

one server and it goes down you're going

to have a really really bad time but if

you run it on a bunch of servers yeah

it's good great so normally server CPU

coolers would come with their own

thermal paste pre-applied but since

we're doing this ourselves and uh if you

look carefully it's not the first time

that it's been installed we are going to

be using okay thank you for that a piece

of Honeywell PTM 7950 this stuff is

freaking awesome it has great thermal

transfer properties and it can handle

varying temperatures like seriously I

don't remember many not even just

varying but like a lot of huge cycles

for a very very long time now available

LTD store.com is that big enough does

that cover all of the ccds and

cxs oh there's a second piece of PL am I

stupid is there a second piece of

plastic no there isn't should I put one

in the fridge no no no it's totally fine

I've done this like a bunch of times

yeah oh she's Min look at that see all

right easy I would recommend putting it

in the fridge before you use it all

right to ensure we're making the

absolute most of our CPU especially in

this High throughput storage workload

we're going to be populating all 12 of

our memory Channels with 32 gig dims of

ddr5 ECC running at 4,800 megga

transitors per second that's a total

of 384 three terabytes of memory what

across all eight

oh each of the cables Jake removing

right now is a pcie by8 cable that feeds

two of the drive bays in the front but

the reason he's taking them out is that

we can install our boot drives these are

consumer grade each system is getting

two Sab 512 gig gen 3 rocket drives and

it's not because they're particularly

special in any meaningful way they're

not even that fast by modern standards

but what they are is from our experience

reliable enough and they are fast enough

for what we're going to be doing which

is just booting our operating system off

of them movie Magic all of the other

nodes are already built so what do you

mean movie Magic super micro built them

Oh I thought you buil them super micro

builds them for you I took it apart okay

fine I took that one apart no secrets

left anymore yep no Intrigue no mystery

you know what is still mysterious is

inside of here I've actually never

opened this before Oh okay let's have a

look woo holy oh that's power supplies

yeah this is so cool so the whole

computer is cooled by four fans no way

there's the two power supply fans and

then these fans in their what do they

call this like IO module I think is what

they call it look at the blades on this

thing counter rotating you're serious

that's what you're looking at not this

the most delicate of spaghet oh my God

there's not even connectors every one of

these wires is soldered directly to the

back of the ocp 3.0 what yeah for

storage we're installing ing two of

kokia Speedy cd6 Gen 4 and vme drives in

each node so we've got one that is 7

tabt and another one that is 15

terabytes they're kind of placeholders

for now and in the long term we're going

to switch to Something in the

neighborhood of about 4 15 tab drives

per node but the drives we want to use

are currently occupied by oh that

project by a top secret pastry related

project so that's going to have to wait

the good news is that when those drives

become available WCA supports live

upgrading and downgrading so we can just

pull these drives swap in the new ones

pull swap pull swap pull swap as long as

we uh don't do it all at once are we

ready to fire these things up okay

there's a lot going on here what is that

is that a switch y hey look you can see

the button now oh that's

cool what you're hearing so far is just

the Nvidia SN 3700 32 Port 200 gig

switch oh my God it even says melanox on

the front I know maybe it's an old like

review sample demo univ we got it with

the $1 million PC and I'm pretty sure

that that was already in video at that

point can you hear that you hear it

getting louder yeah

who well that one's just excited to see

this is the WKA dashboard maybe if I go

over here cluster servers we can see all

of our servers we have two drives per

and then course this is a very

interesting part of how wo works it's

not like trass let's say where it just

uses the whole CPU for whatever you're

trying to do they dedicate and like

fence off specific cores for specific

tasks for instance each Drive gets a

core so we've got two Drive containers

that means two a full core per Drive

yeah damn yeah you also have compute

cores which do like the par calculation

and intercluster communication and then

there's front end which you don't

necessarily always have frontend cores

managed connecting to a file system so

if you just had drives and Compu compute

you wouldn't be able to access the files

on this machine so you would have your

backend servers right those would run

drives and compute which is the cluster

and then on your like GPU box you would

run just the front end and that would

allow the GPU box to connect to the

backend cluster servers oh the back-end

cluster servers don't need to run a

front end unless you want to be able to

access the files on that machine or from

that machine which we want to cuz we're

using SMB we're using it as a a file

server stupid NZ for our stupid windows

machines yeah you can also have a

dedicated front end machine yes so if

you had like a 100 backend servers but

then that's adding a single point of

failure which is what we're trying to

avoid you could have multiple of them

okay you thought they thought of that

yeah I set it up so every single machine

in the cluster all eight of them are

part of our SMB cluster which means it

cannot go down realistically there are a

ton of other file systems out there that

you could use for something like this

traz has their scale out setup for

clustered ZFS which only requires three

nodes and is something we'd be quite

interested in trying out or if you're

looking for object storage there's a

million options but the main open-

source one Min iio requires only four

nodes though when we saw how nuts WCA

was when we set up the million dooll

server cluster I mean we had to try it

out for ourselves and try it out we did

so this is each not node holy

sh look okay the crazy thing is look at

the read latency now guys look look hold

on hold on hold on at 70 gabt a second

we've seen numbers like this before but

we're talking with in some cases double

the number of drives and no file system

without a file system like raw to each

drive this is with a file system with a

file system over a network and we're

only using 100 Gig ports like usually

with a WCA setup like this you'd

probably use 200 yeah cuz we oh my God

we didn't know cuz we didn't even have

networking as a factor last time all the

drives were in one box I know this is

networking too and the crazy part is

we're not using RDMA this is like um

some fancy uh what's it called dpdk I

think is the library this is wild yeah

look at that so read latency 131 microc

seconds that's 4 million read iops with

a latency of 1 millisecond average are

are we able to keep using W FS like this

is a trial okay this software is quite

expensive this is unreal 4 million iops

this is like it is unreal it's way more

than we could possibly ever need but

it's cool it's so cool don't they

support tearing and everything oh yeah

here I'll show you actually what that

looks like this is on mother vault which

I think right now has 400 Tippy bytes

left so let's say Max Capacity is 400

terabytes now once we run out of the 100

terab of SSD capacity which you can see

here it'll just it'll tear I mean it

automatically tear anyways and you do

need to make sure that your object store

is at least the same size as the flash

or bigger because they're going to

automatically tear everything to it that

makes sense so in theory we

move manually copy everything from Vault

one time to wo one time because it