Run your own AI (but private)
Summary
TLDRThe video discusses the concept of private AI, showcasing how to set up a personal AI model on your computer, ensuring data privacy and security. The host explores the potential of private AI in enhancing job efficiency, especially in sectors with strict privacy requirements. Highlighting VMware's role in enabling private AI through their comprehensive solutions, the video emphasizes the ease of fine-tuning AI models with the right tools and infrastructure. The host also demonstrates the integration of personal knowledge bases with AI for tailored assistance, illustrating the transformative impact of private AI on personal and professional lives.
Takeaways
- π The concept of private AI is highlighted, emphasizing running AI models locally on one's own computer, ensuring data privacy and security.
- π The ease and speed of setting up a private AI model on personal devices are discussed, with the process being free and taking only about five minutes.
- π The video introduces the idea of connecting personal knowledge bases, such as notes and documents, to a private AI for personalized assistance.
- π‘ Private AI can be particularly beneficial for jobs where using public AI models is restricted due to privacy and security concerns.
- π The role of companies like VMware in enabling on-premises AI solutions is explored, showcasing their contribution to the advancement of private AI.
- π The script mentions Hugging Face's platform, which hosts a community and a vast collection of AI models, many of which are free and open for use.
- π» The process of downloading and using a powerful AI model like Llama2 is described, illustrating the accessibility of advanced AI technology to individuals.
- π The training of AI models is discussed, including the extensive data and resources required, highlighting the capabilities of entities like Meta AI in pre-training models.
- π οΈ The concept of fine-tuning AI models with proprietary data is introduced, explaining how it can be done withηΈε―ΉθΎε°η resources compared to initial training.
- π The combination of VMware's infrastructure with NVIDIA's AI tools is presented as a comprehensive solution for companies to implement private AI.
- π The video concludes with a quiz for viewers, offering an incentive of free coffee for those who achieve a high score, reinforcing the engagement and educational aspect of the content.
Q & A
What is the primary advantage of running a private AI model on your own computer?
-The primary advantage is that it ensures data privacy and security as the AI model runs locally without sharing data with external companies or servers.
How long does it typically take to set up and run your own AI model on your laptop?
-It is mentioned that setting up and running your own AI model on a laptop computer can be achieved in about five minutes, making it a relatively quick and easy process.
What is Hugging Face and how does it relate to AI models?
-Hugging Face is a website and a community dedicated to providing and sharing a vast array of AI models. It hosts over 505,000 AI models, many of which are open and free to use and pre-trained.
What is an LLM and what is its significance?
-LLM stands for Large Language Model, which is an artificial intelligence model pre-trained on a large dataset of text. These models are significant because they can understand and generate human-like text, making them useful for various applications such as chatbots, text summarization, and more.
How was the Llama two model trained and what does its training entail?
-The Llama two model was trained by Meta (Facebook) using over 2 trillion tokens of data from publicly available sources and over a million human annotated examples. The training process involved a supercluster of over 6,000 GPUs and took 1.7 million GPU hours, with an estimated cost of around $20 million.
What is the role of O Lama in running private AI models?
-O Lama is a tool that allows users to run various LLMs, including Llama two and its uncensored versions, on their local machines. It simplifies the process of installing and utilizing these models without the need for extensive technical setup.
Why is fine-tuning an AI model important for businesses and individuals?
-Fine-tuning an AI model is important because it allows businesses and individuals to tailor the AI to their specific needs and data. This means the AI can understand and provide insights relevant to the user's unique context, such as company-specific knowledge bases or personal documents.
What is the difference between pre-training an AI model and fine-tuning it?
-Pre-training an AI model involves training the model on a large dataset to make it capable of understanding and generating human-like text. Fine-tuning, on the other hand, involves further training the pre-trained model on a smaller, more specific dataset to adapt it to a particular use case or to improve its performance in a specific domain.
How does VMware's private AI solution differ from other AI solutions?
-VMware's private AI solution provides an all-in-one package that includes the necessary hardware, software, and tools for fine-tuning and deploying AI models. It offers a robust infrastructure along with partnerships with companies like Nvidia and Intel, giving users a choice of tools and technologies to suit their needs.
What is RAG and how does it enhance the capabilities of an LLM?
-RAG stands for Retrieval-Augmented Generation. It is a technique that allows an LLM to consult a database or knowledge base for accurate information before generating a response. This enhances the LLM's ability to provide correct and relevant answers based on specific data that it may not have been fine-tuned on.
What is the significance of the quiz mentioned at the end of the video?
-The quiz is a test of the viewer's understanding of the video content. It provides an interactive way for viewers to engage with the material and assess their knowledge. The first five people to score 100% on the quiz receive free coffee from Network Chuck Coffee, adding an incentive for viewers to pay close attention to the video.
Outlines
π Introducing Private AI and its Benefits
The speaker introduces the concept of private AI, emphasizing its advantages over cloud-based AI models. They highlight the importance of data privacy and the ability to run AI models locally on one's computer, without the need for internet connection. The speaker also mentions the ease and speed of setting up private AI, and teases a more advanced demonstration involving connecting personal documents and knowledge bases to the AI for personalized assistance. The discussion extends to the professional applications of private AI, especially in jobs where the use of public AI models is restricted due to privacy and security concerns. The role of companies like VMware in enabling on-premise AI deployment is also acknowledged.
π» Setting Up Private AI and WSL Installation
The speaker provides a step-by-step guide on setting up private AI on a personal computer. They discuss the process of installing Windows Subsystem for Linux (WSL) on Windows, which simplifies the setup of a Linux environment. The speaker then demonstrates the installation of a tool called 'O Lama' and how it enables the running of various large language models (LLMs), including Llama two. The segment also covers the importance of having an Nvidia GPU for enhanced performance and provides a brief comparison of running LLMs on different platforms, such as Linux and Mac with M1 processors.
π Fine-Tuning AI Models and VMware's Role
The speaker delves into the concept of fine-tuning AI models, explaining how it allows customization of models with proprietary data for specific use cases. They discuss the resource-intensive process of pre-training models, as exemplified by Meta's training of their LLM, and contrast it with the more accessible fine-tuning process. The speaker then introduces VMware's contribution to private AI, highlighting their private AI offering in partnership with Nvidia. This includes tools and infrastructure that simplify the fine-tuning process, making it more accessible for companies to develop customized AI solutions.
π Implementing RAG for Personalized AI Experience
The speaker explores the use of RAG (Retrieval-Augmented Generation) for enhancing AI models with personalized data. They demonstrate how RAG can connect an AI model to a database of personal notes and journal entries, enabling the model to provide accurate and personalized responses. The speaker also discusses the potential of fine-tuning and RAG in enhancing the utility of AI in various scenarios, such as troubleshooting company code or creating customer-facing chatbots. The segment concludes with an overview of how VMware and its partners, including Nvidia and Intel, provide the necessary tools and infrastructure for deploying customized AI models.
π₯ Live Demonstration of Private GPT with Personal Knowledge Base
The speaker presents a live demonstration of a private GPT model connected to a personal knowledge base. They walk through the process of uploading a VMware article and asking the AI model questions based on the content. The speaker then shares their experience of connecting their personal journals to the AI and asking questions about their past experiences. The segment highlights the potential and ease of use of such personalized AI models, while also acknowledging the complexity and effort required to set up such a system independently, as opposed to using a comprehensive solution like VMware's private AI.
Mindmap
Keywords
π‘Private AI
π‘Chat GPT
π‘Data Privacy
π‘VMware
π‘On-Prem
π‘Fine Tuning
π‘LLM (Large Language Model)
π‘Hugging Face
π‘Data Freshness
π‘Super Cluster
π‘WSL (Windows Subsystem for Linux)
π‘RAG (Retrieval-Augmented Generation)
Highlights
The transcript discusses the concept of private AI, which is an AI model running locally on a personal computer, ensuring data privacy and security.
The presenter shares a personal experience of setting up a private AI and emphasizes its ease of use, taking only about five minutes to run on a laptop.
An introduction to Hugging Face's platform, which hosts a community dedicated to sharing and providing AI models, with over 505,000 models available.
The mention of Llama, an LLM (Large Language Model) developed by Meta (Facebook), which was trained on over 2 trillion tokens of data and a million human-annotated examples.
The presenter demonstrates the process of downloading and using a pre-trained AI model, emphasizing the power and capabilities of such models.
The discussion of how private AI can benefit professionals who are restricted from using public AI models due to privacy and security concerns in their jobs.
VMware's role in enabling private AI is highlighted, showing how companies can run their own AI models on-premises within their data centers.
The presenter guides the audience on how to install a tool called O Lama, which facilitates the running of various LLMs on different operating systems, including Windows Subsystem for Linux (WSL).
The importance of GPUs in running AI models efficiently is discussed, with the presenter sharing their experience of running an LLM on a Linux virtual machine with an M1 processor.
The concept of fine-tuning AI models with proprietary data is introduced, allowing companies to train their AI models on their own data for specific use cases.
VMware's Private AI solution, in partnership with Nvidia, is presented as a comprehensive package that simplifies the process of fine-tuning AI models for companies.
The presenter explains the process of fine-tuning an LLM with a small dataset, changing only a small percentage of the model's parameters, making it a feasible task without massive resources.
The use of RAG (Retrieval-Augmented Generation) is described, which allows an LLM to consult a database of information before generating a response, ensuring accuracy.
Nvidia AI Enterprise and its tools for deploying custom LLMs are mentioned, emphasizing the ease of setup and customization provided by the platform.
Intel's partnership with VMware is highlighted, offering data scientists tools for analytics, generative AI, deep learning, and classic ML for private AI implementation.
The transcript concludes with a quiz for viewers, offering free coffee to the first five people who score 100%, encouraging engagement and knowledge retention.
Transcripts
I'm running something called private ai. It's kind of like chat GPT,
except it's not. Everything about it is running right here on my computer.
Am I even connected to the internet?
This is private contained and my data isn't being shared with some random
company. So in this video I want to do two things. First,
I want to show you how to set this up.
It is ridiculously easy and fast to run your own AI on your laptop computer or
whatever. It's this is free, it's amazing.
It'll take you about five minutes and if you stick around until the end,
I want to show you something even crazier, a bit more advanced.
I'll show you how you can connect your knowledge base, your notes,
your documents,
your journal entries to your own private GPT and then ask it questions
about your stuff. And then second,
I want to talk about how private AI is helping us in the area we need help Most.
Our jobs, you may not know this,
but not everyone can use chat GBT or something like it at their job.
Their companies won't let them mainly because of privacy and security reasons,
but if they could run their own private ai, that's a different story.
That's a whole different ballgame and VMware is a big reason. This is possible.
They're the sponsor of this video and they're enabling some amazing things that
companies can do on-Prem in their own data center to run their own ai.
And it's not just the cloud man, it's like in your data center.
The stuff they're doing is crazy. We're going to talk about it here in a bit,
but tell you what, go ahead and do this. There's a link in the description.
Just go ahead and open it and take a little glimpse at what they're doing.
We're going to dive deeper,
so just go ahead and have it open right in your second monitor or something or
on the side or minimize. I don't know what you're doing.
I dunno how many monitors you have. You have three Actually, Bob,
I can see before we get started, I have to show you this.
You can run your own private ai. That's kind of uncensored. I watch this,
So yeah, please don't do this to destroy me. Also,
make sure you're paying attention at the end of this video,
I'm doing a quiz and if you're one of the first five people to get a hundred
percent on this quiz, you're getting some free coffee network. Chuck Coffee.
So take some notes, study up. Let's do this
now real quick, before we install a private local AI model on your computer,
what does it even mean? What's an AI model? At its core,
an AI model is simply an artificial intelligence pre-trained on data we
provided. One you may have heard of is open AI's Chat GBT,
but it's not the only one out there. Let's take a field trip.
We're going to go to a website called hugging face.co.
Just an incredible brand name. I love it so much.
This is an entire community dedicated to providing and sharing AI models and
there are a ton. You're about to have your mind blown. Ready?
I'm going to click on models up here. Do you see that number? 505,000 AI models.
Many of these are open and free for you to use and pre-trained,
which is kind of a crazy thing. Let me show you this.
We're going to search for a model named Llama two,
one of the most popular models out there. We'll do LAMA two seven B. Again,
I love the branding.
LAMA two is an AI model known as an LLM or large language model,
open AI's Chat. GPT is also an LLM. Now this LLM,
this pre-trained AI model was made by meda,
AKA Facebook and what they did to pre-train.
This model is kind of insane and the fact that we're about to download this and
use it even crazier, check this out if you scroll down just a little bit,
here we go. Training data.
It was trained by over 2 trillion tokens of data from publicly available
sources. Instruction data sets over a million human annotated examples,
data freshness. We're talking in July, 2023. I love that term.
Data freshness and getting the data was just step one.
Step two is insane because this is where the training happens.
Mata to train this model put together what's called a super cluster.
It already sounds cool, right? This sucker is over 6,000 GPUs.
It took 1.7 million GPU hours to train this model and it's estimated it
costs around $20 million to train it and now made is just like,
here you go kid. Download this incredibly powerful thing.
I don't want to call it a being yet. I'm not ready for that,
but this intelligent source of information that you can just download on your
laptop and ask it questions,
no internet required and this is just one of the many models we could download.
They have special models like text to speech, image to image.
They even have uncensored ones. They have an uncensored version of a llama too.
This guy George Sung,
took this model and fine tuned it with a pretty hefty GPU,
took him 19 hours and made it to where you could pretty much ask this thing.
Anything you wanted, whatever question comes to mind,
it's not going to hold back. Okay,
so how did we get this fine tuned model onto your computer? Well,
actually I should warn you, this involves quite a bit of llamas,
more than you would expect. Our journey starts at a tool called O Lama.
Let's go ahead and take a field trip out there real quick.
We'll go to O lama.ai. All we'll have to do is install this little guy, Mr.
Alama,
and then we can run a ton of different LLMs Llama two Code Llama told you lots
of llamas and there's others that are pretty fun like Llama two Uncensored or
Llamas. Tdrl. I'll show you in a second. But first, what do we install alama on?
We can see right down here that we have it available on macOS and Linux,
but oh bummer, windows coming soon.
It's okay because we've got WSL, the Windows subsystem for Linux,
which is now really easy to set up.
So we'll go ahead and click on download right here from os.
You'll just simply download this and install like one of your regular
applications for Linux. We'll click on this.
We got to fun curl command that will copy and paste now because we're going to
install WSL on Windows. This will be the same step. So Mac OS folks,
go ahead and just run that installer. Linux and Windows folks, let's keep going.
Now, if you're on Windows,
all you have to do now to get WSL installed is launch your Windows terminal.
Just go to your search bar and search for terminal and with one command it'll
just happen. It used to be so much harder, which is WSL dash dash install.
It'll go through a few steps. It'll install Ubuntu as default.
I'll go ahead and let that do that. And boom, just like that.
I've got Ubuntu 22 0 4 3 lts installed and I'm actually inside of it right
now. So now at this point, Linux and Windows folks, we converged.
We're on the same path. Let's install alama.
I'm going to copy that curl command that alama gave us,
jump back into my terminal, paste that in there and press enter.
Fingers crossed, everything should be great. Like the way it is right now,
it'll ask for my pseudo password and that was it. Oh, LAMA is now installed.
Now this will directly apply to Linux people and Windows people.
See right here where it says Nvidia GPU installed. If you have that,
you're going to have a better time than other people who don't have that.
I'll show you here in a second. If you don't have it, that's fine.
We'll keep going. Now let's run an LLM. We'll start with llama two.
So we'll simply type in, oh Lama run,
and then we'll pick one llama two and that's it. Ready,
set go. It's going to pull the manifest.
It'll then start pulling down and downloading Llama two.
And I want you to just realize this, that powerful LAMA two pre-training,
we talked about all the money and hours spent. That's how big it is.
This is the 7 billion parameter model or the seven B.
It's pretty powerful and we're about to literally have this in the palm of our
hands in like 3, 2, 1. Oh, I thought I had it. Anyways,
it's almost done. And boom, it's done.
We've got a nice success message right here and it's ready for us.
We can ask you anything. Let's try what is a pug?
Now the reason this is going so fast, just like a side note,
is that I'm running A GPU and AI models love GPUs.
So lemme just show you real quick.
I did install alama on a Linux virtual machine and I'll just demo the
performance for you real quick. By the way, if you're running a Mac with an M1,
M two or M three processor, it actually works great. I forgot to install it.
I got to install it real quick and I'll ask you that same question.
What is a pug? It's going to take a minute, it'll still work,
but it's going to be slower on CPUs and there it goes. It didn't take too long,
but notice it is a bit slower.
Now if you're running WSL and you know have an Nvidia GPU and it didn't show up,
I'll show you in a minute how you can get those drivers installed. But anyways,
just sit back for a minute,
sip your coffee and think about how powerful this is.
The tinfoil hat version of me stinking loves this because let's say
the zombie apocalypse happens, right? The grid goes down, things are crazy,
but as long as I have my laptop and a solar panel,
I still have AI and it can help me survive the zombie apocalypse.
Let's actually see how that would work. It gives me next steps.
I could have it help me with the water filtration system. This is just cool,
right? It's amazing. But can I show you something funny?
You may have caught this earlier. Who is network? Chuck?
What? Dude, I've always wanted to be Rick Grimes.
That is so fun, but seriously, it kind of hallucinated there.
It didn't have the correct information.
It's so funny how it mixed the zombie apocalypse prompt with me.
I love that so much. Let's try a different model. I'll say bye.
I'll try a really fun one called mytral. And by the way,
if you want to know which ones you can run with Llama, which LLMs,
they get a page for their models right here and all the ones you can run,
including llama two, uncensored Wizard Math.
I might give that to my kids actually. Let's see what it says.
Now who is Network Chuck?
Now my name is not Chuck Davis and my YouTube channel is not called Network
Chuck on Tech.
So clearly the data this thing was trained on is either not up to date or just
plain wrong. So now the question is cool,
we've got this local private ai, this LLM, that's super powerful,
but how do we teach it the correct information for us?
How can I teach it to know that I'm network Chuck, Chuck Keith, not Chuck Davis,
and my channel is called Network Chuck.
Or maybe I'm a business and I want it to know more than just what's publicly
available because sure, right now if you downloaded this lm,
you could probably use it in your job,
but you can only go so far without it knowing more about your job. For example,
maybe you're on a help desk.
Imagine if you could take your help desk's knowledge base, your IT procedures,
your documentation. Not only that,
but maybe you have a database of closed tickets, open tickets.
If you could take all that data and feed it to this LLM and then ask it
questions about all of that, that would be crazy.
Or maybe you wanted to help troubleshoot code that your company's written.
You could even make this LM public facing for your customers.
You feed information about your product and the customer could interact with
that chat bot you make.
Maybe this is all possible with a process called fine tuning where we can train
this AI on our own proprietary secret private stuff about our
company or maybe our lives or whatever you want to use it for,
whatever use case is,
and this is fantastic because maybe before you couldn't use a public LLM because
you weren't allowed to share your company's data with that LLM,
whether it's compliance reasons or you just simply didn't want to share that
data because it's secret. Whatever the case,
it's possible now because this AI is private,
it's local and whatever data you feed to it,
it's going to stay right there in a company. It's not leaving the door.
That idea just makes me so excited because I think it is the future of AI and
how companies and individuals will approach it. It's going to be more private.
Back to our question though, fine tuning, that sounds cool.
Training and AI on your own data, but how does that work?
Because as we saw before with pre-training a model with mata,
it took them 6,000 GPUs over 1.7 million GPU hours.
Do we have to have this massive data center to make this happen? No.
Check this out, and this is such a fun example, VMware, they asked chat GPT,
what's the latest version of VMware vSphere?
Now the latest chat GPT knew about was vSphere 7.0,
but that wasn't helpful to VMware because their latest version they were working
on chat hadn't been released yet.
So it wasn't public knowledge was vSphere eight update too.
And they wanted information like this internal information not yet released to
the public.
They wanted this to be available to their internal team so they could ask
something like chat GBT, Hey, what's the latest version of vSphere?
And they could answer correctly.
So to do what VMware is trying to do to fine tune a model or train it on new
data, it does require a lot. First of all,
you would need some hardware servers with GPUs.
Then you would also need a bunch of tools and libraries and SDKs like PyTorch
and TensorFlow, pandas, MPI side kit, learn transformers and fast ai.
The list goes on.
You need lots of tools and resources in order to fine tune an LLM.
That's why I'm a massive fan of what VMware is doing right here.
They have something called the VMware private AI with Nvidia,
the gajillion things I just listed off. They include in one package,
one combo meal, a recipe of ai, fine tuning goodness.
So as a company it becomes a bit easier to do this stuff yourself locally.
For the system engineer you have on staff who knows VMware and loves it,
they could do this stuff,
they could implement this and the data scientists they have on staff that will
actually do some of the fine tuning, all the tools are right there.
So here's what it looks like to fine tune and we're going to kind of peek behind
the curtain at what a data scientist actually does.
So first we have the infrastructure and we start here in vSphere, VMware.
Now if you don't know what vSphere is or VMware, think virtual machines,
you got one big physical server. The hardware, the stuff you can feel,
touch and smell. You haven't smelled the server, I dunno what you're doing.
And instead of installing one operating system on them like Windows or Linux,
you install VMware's, EA XI,
which will then allow you to virtualize or create a bunch of additional virtual
computers. So instead of one computer,
you've got a bunch of computers all using the same hardware resources.
And that's what we have right here. One of those virtual computers,
a virtual machine.
This by the way is one of their special deep learning VMs that has all the tools
I mentioned and many, many more pre-installed, ready to go.
Everything a data scientist could love.
It's kind of like a surgeon walking in to do some surgery and like their doctor
assistants or whatever have prepared all their tools.
It's all in the tray laid out nice and neat to the surgeon.
All he has to do is walk in and just go scalpel.
That's what we're doing here for the data scientist.
Now talking more about hardware,
this guy has a couple Nvidia GPUs assigned to it or pass through to it through
a technology called PCIE Passthrough. These are some beefy GPUs.
I notice they are V GPU for virtual GPU similar to what you do with the CPU,
cutting up the PU and assigning some of that to a virtual CPU on a virtual
machine. So here we are in data scientists world. This is a Jupiter notebook,
a common tool used by a data scientist,
and what you're going to see here is a lot of code that they're using to prepare
the data,
specifically the data that they're going to train or fine tune the existing
model on. Now we're not going to dive deep on that,
but I do want you to see this, check this out.
A lot of this code is all about getting the data ready. So in VMware's case,
it might be a bunch of the knowledge base product documentation and they're
getting it ready to be fed to the LLM. And here's what I wanted you to see.
Here's the dataset that we're training this model on. We're fine tuning.
We only have 9,800 examples that we're giving it or 9,800 new prompts or
pieces of data. And that data might look like this,
like a simple question or a prompt and then we feed it the correct answer and
that's how we essentially train ai. But again,
we're only giving it 9,800 examples,
which is not a lot at all and is extremely small compared to how the
model was originally trained.
And I point that out to say that we're not going to need a ton of hardware or a
ton of resources to fine tune this model.
We won't need the 6,000 GPUs we needed for MATA to originally create this model.
We're just adding to it,
changing some things or fine tuning it to what our use case is and looking at
what actually will be changed when we run this and we train it,
we're only changing 65 million parameters, which sounds like a lot, right?
But not in the grand scheme of things of like a 7 billion parameter model.
We're only changing 0.93% of the model.
And then we can actually run our fine tuning,
which this is a specific technique in fine tuning called prompt tuning where we
simply feed up additional prompts with answers to change how it'll react to
people asking you questions.
This process will take three to four minutes to fine tune it because again,
we're not changing a lot and that is just so super powerful and I think VMware
is leading the charge with private ai.
VMware and Nvidia take all the guesswork out of getting things set up to fine
tune an LLM. They've got deep learning VMs,
which are insane VMs that come pre-installed with everything you could want
everything a data scientist would need to find tune an LLM.
Then Nvidia has an entire suite of tools sensor around their GPUs,
taking advantage of some really exciting things to help you fine tune your lms.
Now there's one thing I didn't talk about because I wanted to save it for last.
For right now it's this right here, this vector database,
post gray SQL box here.
This is something called rag and it's what we're about to do with our own
personal GPT here in a bit. Retrieval, augment the generation. So scenario,
let's say you have a database of product information, internal docs,
whatever it is, and you haven't fine tuned your LLM on this just yet.
So it doesn't know about it. You don't have to do that with rag.
You can connect your LLM to this database of information,
this knowledge base and give it these instructions.
Say whenever I ask you a question about any of the things in this database,
before you answer, consult the database,
go look at it and make sure what you're saying is accurate.
We're not retraining the LLM, we're just saying, Hey, before you answer,
go check real quick in this database to make sure it's accurate to make sure you
got your stuff right. Isn't that cool? So yes,
fine tuning is cool and training an LLM on your own data is awesome,
but in between those moments of fine tuning,
you can have rag set up where it can consult your database,
your internal documentation and give correct answers based on what you have in
that database. That is so stinking cool.
So with VMware private AI foundation with nvidia,
they have those tools baked right in to where it just kind of works for what
would otherwise be a very complex setup. And by the way, this whole rag thing,
like I said earlier, we're about to do this,
I actually connected a lot of my notes and journal entries to a private GPT
using RAG and I was able to talk with it about me asking it about my
journal entries and answering questions about my past. That's so powerful. Now,
before we move on,
I just want to highlight the fact that Nvidia with their Nvidia AI enterprise
gives you some amazing and fantastic tools to pull the LLM of your choice and
then fine tune and customize and deploy that LLM. It's all built in right here.
So VMware Cloud Foundation,
they provide the robust infrastructure and NVIDIA provides all the amazing AI
tools you need to develop and deploy these custom LLMs.
Now it's not just Nvidia, they're partnering with Intel as well.
So VMware is covering all the tools that admins care about.
And then for the data scientists, this is for you.
Intel's got your back data analytics,
generative AI and deep learning tools and some classic ML or machine learning.
And they're also working with IBM, all you IBM fans. You can do this too. Again,
VMware has the admin's back. But for the data scientist, Watson,
one of the first AI things I ever heard about Red Hat and OpenShift,
and I love this because what VMware is doing is all about choice.
If you want to run your own local private ai, you can.
You're not just stuck with one of the big guys out there and you can choose to
run it with Nvidia and VMware, Intel and VMware, IBM and VMware.
You got options. So there's nothing stopping you.
It's not for some of the bonus section of this video and that's how to run your
own private GPT with your own knowledge base. Now, fair warning,
it is a bit more advanced, but if you stick with me,
you should be able to get this up and running. So take one more sip of coffee.
Let's get this going. Now, first of all, this will not be using a lama.
This will be a separate project called Private GPT. Now disclaimer,
this is kind of hard to do. Unlike VMware private ai,
which they do it all for you,
it's a complete solution for companies to run their own private local ai.
What I'm about to show you is not that at all. No affiliation with VMware.
It's a free side project.
You can try just to get a little taste of what running your own private GPT with
rag tastes like. Did I do that right? I don't know.
Now L Martinez has a great doc on how to install this. It's a lot,
but you can do it. And if you just want a quick start,
he does have a few lines of code for Linux and Mac users. Fair warning,
this is CPU only. You can't really take advantage of RAG without A GPU,
which is what I wanted to do. So here's my very specific scenario.
I've got a Windows PC with an NVIDIA 40 90. How do I run this?
Linux-based project. WSL, and I'm so thankful to this guy Emelia Lance a lot.
He put an entire guide together of how to set this up.
I'm not going to walk you through every step because he already did that link
below, but I seriously need to buy this guy a coffee. How do I do that?
I don't know, Emil, if you're watching this, reach out to me.
I'll send you some coffee. So anyways,
I went through every step from installing all the prereqs to installing NVIDIA
drivers and using poetry to handle dependencies, which poetry is pretty cool.
I landed here.
I've got a private local working private GPT that I can access through my web
browser and it's using my GPU, which is pretty cool. Now,
first I try a simple document upload,
got this VMware article that details a lot of what we talked about in this
video. I upload it and I start asking you questions about this article.
I tried something specific like show me something about VMware AI market growth.
Bam, it figured it out, it told me. Then I'm like,
what's the coolest thing about VMware private ai?
It told me I'm sitting here chatting with a document, but then I'm like,
let's try something bigger. I want to chat with my journals.
I've got a ton of journals on markdown format and I want to ask you questions
about me. Now this specific step is not covered in the article.
So here's how you do it. First,
you'll want to grab your folder of whatever documents you want to ask questions
about and throw it onto your machine.
So I copied over to my WSL machine and then I ingested it with this command once
complete and I ran private GPT. Again,
here's all my documents and I'm ready to ask it questions.
So let's test this out. I'm going to ask it what did I do in takayama?
So I went to Japan in November of 2023. Let's see if you can search my notes,
figure out when that was and what I did.
That's awesome. Oh my goodness.
Let's see, what did I eat in Tokyo?
How cool is that? Oh my gosh, that's so fun. No, it's not perfect,
but I can see the potential here. That's insane. I love this so much.
Private AI is the future and that's why we're seeing VMware bring products like
this to companies to run their own private local AI and then make it pretty
easy. If you actually did that private GPT thing, that little side project,
there's a lot to it. Lots of tools you have to install, it's kind of a pain.
But with VMware,
they kind of cover everything like that deep learning VM they offer as part of
their solution. It's got all the tools ready to go. Pre-baked again,