This Server CANNOT Lose Dataβ¦
Summary
TLDRThe script details the process of upgrading a video production team's storage and editing server, named 'Wanic,' to improve reliability and performance. The team transitions from a single server setup to a high-availability configuration using Super Micro Grand Twin A+ servers, each housing four independent compute nodes with AMD EPYC Genoa processors, 384GB of memory, and NVMe drives. The new system is designed to handle the team's extensive video projects with minimal downtime, leveraging software like WCA for file distribution and Axle AI for media asset management, showcasing impressive read speeds and IOPS. The video also explores the potential of AI in video editing and the challenges of integrating new hardware into an existing infrastructure.
Takeaways
- π The team has upgraded their storage and editing server, named 'Wanic 10', to improve efficiency and reduce downtime costs.
- π The new server setup includes redundant drives and high availability features to ensure minimal disruption to the workflow.
- π» Super Micro Grand Twin A+ servers power the new system, each containing four independent computers within a 2U chassis.
- π Each server node boasts 384GB of memory, an AMD EPYC Genoa processor with 64 cores, and support for up to 24 NVMe drives.
- π‘ The servers are equipped with 2200W 80 PLUS Titanium power supplies to handle the high-performance components.
- π The system supports live drive upgrading and swapping without downtime, thanks to the flexibility of WCA (WekaIO Matrix)
- π The new storage solution demonstrated impressive read latency of 131 microseconds, achieving 4 million IOPS (Input/Output Operations Per Second).
- π The team tested the high availability by simulating server failures, which were handled smoothly without affecting ongoing editing tasks.
- π₯ The setup is designed to support video editing with Adobe Premiere, which is sensitive to latency and benefits from the system's low-latency storage.
- π€ AI integration allows for efficient search and organization of the extensive video archive, enabling quick retrieval of specific clips.
- π The infrastructure relies on a combination of CPU and GPU resources for different AI tasks, with a dedicated GPU workstation for object and scene analysis.
Q & A
What is the main issue with the current storage setup mentioned in the script?
-The main issue with the current storage setup is that it is all housed in a single server, which poses a significant risk of downtime and data loss if that server fails.
What is the term used in the script to describe the new server setup?
-The term used to describe the new server setup is 'high availability', which aims to minimize downtime and data loss by incorporating redundancy and fault tolerance.
Which company sponsored the new server setup with their servers?
-Super Micro sponsored the new server setup with their servers.
What are the key components of the Super Micro Grand Twin A+ server mentioned in the script?
-The key components of the Super Micro Grand Twin A+ server include four independent computers, each with its own motherboard, 384 GB of memory, an AMD EPYC Genoa processor with 64 cores, dual M.2 slots for redundant boot drives, six PCIe Gen 5 slots, and multiple NVMe drives.
How many NVMe drives are installed in each node of the server setup?
-Two NVMe drives are installed in each node of the server setup, with one being a 7 TB drive and the other a 15 TB drive.
What is the purpose of the OCP 3.0 small form factor mezzanine slots in the server setup?
-The OCP 3.0 small form factor mezzanine slots are used to install the ConnectX-6 200 Gbit cards from Nvidia, which provide high-speed network connectivity for the servers.
What is the significance of the 2200 Watts 80 Plus Titanium power supplies in the server setup?
-The 2200 Watts 80 Plus Titanium power supplies are necessary to handle the high power requirements of the server setup, which includes four 400 Watt EPYC Genoa CPUs, a large amount of RAM, up to 24 NVMe drives, and eight network cards.
How does the new server setup handle the potential failure of a machine?
-The new server setup is designed to be highly available, meaning it should be able to continue operating uninterrupted even if one or two machines fail, thanks to its redundancy and the fact that each system has its own drives.
What is the role of the WCA (WekaIO) file system in the server setup?
-The WCA file system is a high-performance, distributed file system designed specifically for NVMe drives. It is used to handle the distribution of terabytes of data to multiple machines and provides low-latency, high-throughput storage for video editing and other demanding applications.
What is the AI technology being used for in the script?
-The AI technology is being used for media asset management, which includes facial recognition, object detection, and scene understanding. This allows for efficient search and retrieval of specific video clips and content from a large archive of footage.
How does the new server setup affect the editing process?
-The new server setup provides a highly available and fast storage solution that minimizes downtime and improves the editing process by ensuring smooth access to large video files and projects, reducing the risk of software crashes due to latency.
Outlines
π Introduction to High Availability Storage Solution
The paragraph introduces the need for a reliable and fast storage solution due to the high volume of video production work. The team's main editing server, 'Wanic,' has been reliable over the years but as the team grows, even a minute of downtime costs significantly. The solution is to add redundancy, which is achieved through a new setup called 'Wanic 10,' emphasizing high availability (HA). The setup includes two grand twin boxes, each containing four independent servers provided by Super Micro, the sponsor of the project.
π οΈ Detailed Overview of the Super Micro Grand Twin A+ Server
This paragraph delves into the specifics of the Super Micro Grand Twin A+ server, highlighting its capabilities and components. Each 2U server contains four independent computers, each with its own motherboard, 384GB of memory, an AMD EPYC Genoa processor with 64 cores, dual M.2 slots for redundant boot drives, and multiple PCIe Gen 5 and NVMe slots. The server's power supply is discussed, with each unit providing 2200 Watts and being 80 PLUS Titanium certified. The paragraph also touches on the high availability aspect, mentioning the need for redundant network cards and switches, and the goal of the new setup to withstand the failure of individual components without affecting operations.
π§ Assembly and Configuration of the High Availability System
The paragraph describes the assembly process of the new high availability system. It discusses the installation of the CPUs, the application of thermal paste, and the populating of memory channels with 32GB DIMMs of DDR5 ECC RAM. The paragraph also covers the installation of boot drives and the storage configuration, which includes two types of drives in each node. The setup's dashboard and the way it allocates resources for different tasks, such as drive containers and compute cores, are also explained. The paragraph concludes with a test of the system's resilience by simulating a server failure.
π Impressive Performance Metrics and Future Plans
This paragraph focuses on the performance metrics of the new storage system, highlighting the impressive read latency and IOPS (Input/Output Operations Per Second) achieved. It discusses the system's ability to handle high throughput and the potential for future upgrades, such as increasing the number of drives per node. The paragraph also talks about the use of the system for various types of workloads, including video editing and AI development, and mentions the capabilities of the WCA file system. The potential for using the system as a file server for Windows machines is also discussed.
π₯ Utilizing AI for Media Asset Management and Searchability
The paragraph discusses the integration of AI for media asset management, allowing for the search and retrieval of specific clips based on content. It describes the process of generating proxies for the vast archive of footage and the use of a GPU workstation for AI analysis. The paragraph outlines the capabilities of the AI in identifying objects, scenes, and even specific people within video clips. It also touches on the potential for improving searchability and the challenges of managing the cables in the server rack.
π» Testing the Resilience of the High Availability System
The final paragraph demonstrates the resilience of the high availability system by intentionally removing servers from the network to simulate a catastrophic failure. The system continues to function smoothly despite the removal of critical components, showcasing its robustness. The paragraph concludes with acknowledgments to Super Micro for the servers, WCA for the software, and Axle for the AI detection, as well as a thank you to the viewers for their support.
Mindmap
Keywords
π‘High Availability
π‘WCK Final Form
π‘Redundant NVMe
π‘Supermicro Grand Twin A+ Server
π‘WCA (WekaIO)
π‘ConnectX-6 200Gb Cards
π‘Mother Vault
π‘AI Detection
π‘Latency
π‘Scalability
π‘Redundant Power Supplies
Highlights
The team has reached a point where a single minute of downtime costs over $50 in payroll, emphasizing the need for high availability storage solutions.
The main editing server, Wanic, has been reliable for years, but as the team grows, the need for redundancy becomes more critical.
Wanic 10 is introduced as the final form of the server, designed for high availability with the ability to handle unplugging without noticeable impact.
The new server setup includes two Grand Twin boxes, each containing four entire servers provided by Super Micro, the sponsor of the project.
Each server inside the Grand Twin boxes has 384GB of memory, an AMD EPYC Genoa processor with 64 cores, and dual M.2 slots for redundant boot drives.
The server setup features six PCIe Gen 5 slots, with 2.5 in NVMe slots upfront and additional slots in the rear for I/O.
The servers are equipped with 2200 Watts 80 Plus Titanium power supplies, capable of handling high-performance components.
The network cards installed are ConnectX-6 200 Gbit cards from Nvidia, providing high-speed connectivity despite being limited by the slot speed.
The server design allows for high availability, with the system able to continue operating even if one of the servers dies.
The new server setup is tested by moving the entire team onto it without notice during a busy workday, demonstrating its reliability and performance.
The software used for distributing terabytes of video projects and other data is WCA, a redundant NVMe first file system.
The CPU chosen for the servers is the AMD EPYC Genoa 9534, a 64-core, 128-thread processor with 4GB of L3 cache and a 300W TDP.
The memory installed is 32GB DIMMs of DDR5 ECC, totaling 384GB across all eight memory channels.
The storage solution includes two Kokisk Speedy CD6 Gen 4 and VME drives in each node, with plans to switch to larger drives in the future.
WCA supports live upgrading and downgrading of drives, allowing for easy maintenance and expansion of the storage system.
The WKA dashboard provides a clear overview of the cluster servers, showing the allocation of cores for specific tasks and the overall performance.
The system achieved 4 million read IOPS with a latency of 1 millisecond average, demonstrating exceptional performance for a file system over a network.
The server setup is designed to avoid single points of failure, with each machine in the cluster being part of the SMB cluster for uninterrupted operation.
The use of AI for media asset management allows for efficient searching and organization of the vast amount of footage, enhancing the usability of the storage system.
The server's cooling system includes four fans, with a unique counter-rotating design in the IO module for efficient heat dissipation.
Transcripts
when you make as many videos as we do
you need a lot of fast reliable storage
and our main editing server wanic has
checked all of those boxes for years
it's a great little server it's built
out of high quality components and it
even looks cool but as our team is grown
we've reached the point where even a
minute one single minute of downtime
costs over $50 and that's just in
payroll now practically speaking the way
to mitigate that is by adding redundant
now our drives are already redundant
we've got 20 drives in there with data
striping but the problem is they all sit
in one single server I'm sure you can
see where this is going it's been over a
year in the making but it's finally here
wck final form and I'm calling it wanic
10 because it's the last wever avability
W told you this like 10 times nobody
even knows what high availability means
it means it's lus just go ahead unplug
one do it go for it well okay I should
probably tell you the stakes before you
do that each of these two grand twin
boxes has four entire servers inside of
them that were provided by super micro
who sponsored this whole thing and
they're set up with WCA a redundant nvme
first file system in this config it
should sustain two entire servers
dropping out without anyone even
noticing except that we moved the entire
team onto it last night without telling
anyone and it's the middle of the work
day with a ton of high priority videos
in progress do you really want to test
it right now I like I haven't tried that
all right here we go okay what could go
wrong I mean a
[Applause]
lot naturally a huge part of a project
like this is the software the stuff
that's going to handle Distributing all
of ourish terabytes of video projects
Word documents and Linux isos to the
multiple machines that we just showed
you but we can't install any software
until we have some Hardware so why don't
we start there meet the super micro
Grand twin A+ server as- 2115 gt-
hntr despite its sort of ordinary
looking appearance and unexciting
sounding name it
is anything but ordinary and it is very
exciting because inside this 2u is four
independent Compu computers but for what
we're doing four nodes please we want
eight inside each of these is a
completely independent motherboard 384
gigs of memory an AMD epic Genoa
processor with 64 cores dual m.2 slots
for redundant boot drives six pcie Gen 5
2 and 1/2 in nvme slots up front and
we've got IO in the rear now this bit
here could be a little confusing at
first glance but that is because not
only do we have USB but we have two full
gen 5x6 pcie connections back here along
with display output and power for the
entire server this whole thing slides
into the chassis which holds a really
cool modular backplane assembly that
we'll take a look at in a minute and
then passes through thank you Jake ah to
the back at the server where you've got
a Management Port a single USB port for
each server nope it's two and they're
shared what the I was about to ask cuz
we've also got a single VGA you see the
button for two servers there no way this
button toggles
yeah and okay before we talk about that
a little bit more look at these power
supplies each of these is
2200 Watts 80 plus typ tianium which
sounds like a lot but when you're
potentially handling four 400 wat epic
Genoa CPUs along with a bunch of ram up
to 24 nvme drives and eight network
cards well it seems downright reasonable
doesn't it is it 24 drives can't be 6
yes 6 * 4 is
24 and of course that's just one of them
we've got two of those and that means
that in the event that one of these dies
the system should be able to continue to
operate uninterrupted which is a big
part of the high availability goal that
we have for this deployment speaking of
high availability let's move on to our
network cards each of those pcie gen 5x6
slots I showed you guys before
terminates in one of these ocp 3.0 small
form factor mezzanine slots and what
we're putting in them is these connectx
6 200 gbit cards from
melanox excuse me from Nvidia that okay
these are the older Gen 4 ones so
they're going to be limited by the slot
speed of around 250 gabit per second but
if we had newer cards that means that
each of these nodes could do 200 plus
another 200 400 up to
800 gigabit which would of course be a
complete waste for us a because our
workload can't take advantage of it and
B because our switch is only 100 gbit
sorry of course the two ports are still
helpful we do have redundant
switches except there's kind of a
problem here that's still a single point
of failure in a perfect world we would
have two single port Nicks so if a Nick
were to die it would still be okay but
because we have so many nodes we're not
really worried about an individual node
you know they could have one boot drive
and it die or one Nick and it die we
still have an extra backup how many
nines do you want I mean I don't know
like one would would be good 9% which
Jokes Aside is a really good point if we
were architecting this properly there
are so many more considerations that we
would need to make like the power coming
into the rack would have to come from
two independent backed up sources the
connectivity to our clients would have
to be redundant as well the connectivity
between all of the systems would have to
be architected in such a way that no
matter what fails everything will stay
up and realistically for us we're not
going to get that deep into it because
our goal is better than we had before
which was a single machine with its own
built-in redundancies but other than
that nothing now at least we should be
able to lose a full machine out of these
eight we can restart one of our core
switches totally fine two machines out
of these eight and we can still be
limping along I mean limping is a bit of
a stretch it's going to be very fast now
normally if you buy a super micro
machine they're going to pre-build it
for you they're going to validate it for
you you can even have them pre-build an
entire Rack or racks of these things and
then validate your application on it
before it ships to you in fact we've got
a whole video that we did about that
that was sponsored by super micro a
little while back of course this is LT
my friends so we will be assembling this
one ourselves do you like that spin of
the screwdriver above the server don't
worry I won't miss I'll never miss see I
could do this a hundred times and I
would never miss why no it's fine it's
good it's okay we have seven more any
who for our CPU we've gone with an epic
Genova
9534 this is a 64 core
128 thread monster of a CPU it'll do 3.7
GHz Max boost it has A4 gigabyte of
level three cache a 300 wat TDP it
supports ddr5 memory up to 12 channels
and it supports a whopping 128 Lanes of
pcie Gen 5 originally we were intending
to go with 32 core chips but they were
out of stock so free upgrade lucky us
compared to previous generation AMD epic
CPUs dooa is a big step up in terms of
IO performance which makes it perfect
for this application and in the long
term I mean if we've got all the extra
CPU cores and a whole bunch of ram
anyway why run WCA on the bare metal
when we could install prox Mox and then
use the other cores for I don't know
High
availability Plex server yeah Linux isos
more realistically it would be something
like active directory yeah which we
don't really want to do right now
because if you run active directory on
one server and it goes down you're going
to have a really really bad time but if
you run it on a bunch of servers yeah
it's good great so normally server CPU
coolers would come with their own
thermal paste pre-applied but since
we're doing this ourselves and uh if you
look carefully it's not the first time
that it's been installed we are going to
be using okay thank you for that a piece
of Honeywell PTM 7950 this stuff is
freaking awesome it has great thermal
transfer properties and it can handle
varying temperatures like seriously I
don't remember many not even just
varying but like a lot of huge cycles
for a very very long time now available
LTD store.com is that big enough does
that cover all of the ccds and
cxs oh there's a second piece of PL am I
stupid is there a second piece of
plastic no there isn't should I put one
in the fridge no no no it's totally fine
I've done this like a bunch of times
yeah oh she's Min look at that see all
right easy I would recommend putting it
in the fridge before you use it all
right to ensure we're making the
absolute most of our CPU especially in
this High throughput storage workload
we're going to be populating all 12 of
our memory Channels with 32 gig dims of
ddr5 ECC running at 4,800 megga
transitors per second that's a total
of 384 three terabytes of memory what
across all eight
oh each of the cables Jake removing
right now is a pcie by8 cable that feeds
two of the drive bays in the front but
the reason he's taking them out is that
we can install our boot drives these are
consumer grade each system is getting
two Sab 512 gig gen 3 rocket drives and
it's not because they're particularly
special in any meaningful way they're
not even that fast by modern standards
but what they are is from our experience
reliable enough and they are fast enough
for what we're going to be doing which
is just booting our operating system off
of them movie Magic all of the other
nodes are already built so what do you
mean movie Magic super micro built them
Oh I thought you buil them super micro
builds them for you I took it apart okay
fine I took that one apart no secrets
left anymore yep no Intrigue no mystery
you know what is still mysterious is
inside of here I've actually never
opened this before Oh okay let's have a
look woo holy oh that's power supplies
yeah this is so cool so the whole
computer is cooled by four fans no way
there's the two power supply fans and
then these fans in their what do they
call this like IO module I think is what
they call it look at the blades on this
thing counter rotating you're serious
that's what you're looking at not this
the most delicate of spaghet oh my God
there's not even connectors every one of
these wires is soldered directly to the
back of the ocp 3.0 what yeah for
storage we're installing ing two of
kokia Speedy cd6 Gen 4 and vme drives in
each node so we've got one that is 7
tabt and another one that is 15
terabytes they're kind of placeholders
for now and in the long term we're going
to switch to Something in the
neighborhood of about 4 15 tab drives
per node but the drives we want to use
are currently occupied by oh that
project by a top secret pastry related
project so that's going to have to wait
the good news is that when those drives
become available WCA supports live
upgrading and downgrading so we can just
pull these drives swap in the new ones
pull swap pull swap pull swap as long as
we uh don't do it all at once are we
ready to fire these things up okay
there's a lot going on here what is that
is that a switch y hey look you can see
the button now oh that's
cool what you're hearing so far is just
the Nvidia SN 3700 32 Port 200 gig
switch oh my God it even says melanox on
the front I know maybe it's an old like
review sample demo univ we got it with
the $1 million PC and I'm pretty sure
that that was already in video at that
point can you hear that you hear it
getting louder yeah
who well that one's just excited to see
this is the WKA dashboard maybe if I go
over here cluster servers we can see all
of our servers we have two drives per
and then course this is a very
interesting part of how wo works it's
not like trass let's say where it just
uses the whole CPU for whatever you're
trying to do they dedicate and like
fence off specific cores for specific
tasks for instance each Drive gets a
core so we've got two Drive containers
that means two a full core per Drive
yeah damn yeah you also have compute
cores which do like the par calculation
and intercluster communication and then
there's front end which you don't
necessarily always have frontend cores
managed connecting to a file system so
if you just had drives and Compu compute
you wouldn't be able to access the files
on this machine so you would have your
backend servers right those would run
drives and compute which is the cluster
and then on your like GPU box you would
run just the front end and that would
allow the GPU box to connect to the
backend cluster servers oh the back-end
cluster servers don't need to run a
front end unless you want to be able to
access the files on that machine or from
that machine which we want to cuz we're
using SMB we're using it as a a file
server stupid NZ for our stupid windows
machines yeah you can also have a
dedicated front end machine yes so if
you had like a 100 backend servers but
then that's adding a single point of
failure which is what we're trying to
avoid you could have multiple of them
okay you thought they thought of that
yeah I set it up so every single machine
in the cluster all eight of them are
part of our SMB cluster which means it
cannot go down realistically there are a
ton of other file systems out there that
you could use for something like this
traz has their scale out setup for
clustered ZFS which only requires three
nodes and is something we'd be quite
interested in trying out or if you're
looking for object storage there's a
million options but the main open-
source one Min iio requires only four
nodes though when we saw how nuts WCA
was when we set up the million dooll
server cluster I mean we had to try it
out for ourselves and try it out we did
so this is each not node holy
sh look okay the crazy thing is look at
the read latency now guys look look hold
on hold on hold on at 70 gabt a second
we've seen numbers like this before but
we're talking with in some cases double
the number of drives and no file system
without a file system like raw to each
drive this is with a file system with a
file system over a network and we're
only using 100 Gig ports like usually
with a WCA setup like this you'd
probably use 200 yeah cuz we oh my God
we didn't know cuz we didn't even have
networking as a factor last time all the
drives were in one box I know this is
networking too and the crazy part is
we're not using RDMA this is like um
some fancy uh what's it called dpdk I
think is the library this is wild yeah
look at that so read latency 131 microc
seconds that's 4 million read iops with
a latency of 1 millisecond average are
are we able to keep using W FS like this
is a trial okay this software is quite
expensive this is unreal 4 million iops
this is like it is unreal it's way more
than we could possibly ever need but
it's cool it's so cool don't they
support tearing and everything oh yeah
here I'll show you actually what that
looks like this is on mother vault which
I think right now has 400 Tippy bytes
left so let's say Max Capacity is 400
terabytes now once we run out of the 100
terab of SSD capacity which you can see
here it'll just it'll tear I mean it
automatically tear anyways and you do
need to make sure that your object store
is at least the same size as the flash
or bigger because they're going to
automatically tear everything to it that
makes sense so in theory we
move manually copy everything from Vault
one time to wo one time because it