In this tutorial we'll walk through my local, private, self-hosted AI stack so that you can run it too.
If you're looking for the overview of this stack, you can out the video here:
Video Notes:
Products:
GPUs
12GB is probably "good enough" for small models.
3000 Series
👇 My value pick. 3060, 12GB RAM, affordable, can play games too.
– MSI Gaming GeForce RTX 3060 12GB:
– GIGABYTE GeForce RTX 3060 Gaming OC 12G:
– NVIDIA GeForce RTX 3090 Founders Edition Graphics Card Renewed 24GB:
– Zotac Gaming GeForce RTX 3090 Trinity OC, 24GB:
4000 Series
– ASUS TUF Gaming GeForce RTX™ 4090 OG OC Edition Gaming Graphics Card 24GB:
– MSI Gaming GeForce RTX 4060 8GB:
– ASUS Dual GeForce RTX™ 4070 White OC Edition 12GB:
CPUs
You'll want a modern CPU, if you are going desktop class here are a few I would choose
– Intel Core i7-12700K Gaming Desktop Processor:
– Intel Core i7-13700K Gaming Desktop Processor:
– Intel Core i7-14700K Gaming Desktop Processor:
Storage
For flash storage, I always go with these SSDs
– Samsung 870 EVO SATA:
– SAMSUNG 990 PRO SSD NVMe:
(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)
Support me on Patreon:
Sponsor me on GitHub:
Subscribe on Twitch:
Merch Shop 🛍️:
Gear Recommendations:
Get Help in Our Discord Community:
Main channel:
Talks Channel:
00:00 – Intro
01:14 – Hardware Specs
02:23 – GPU Considerations
03:10 – GPU Perspective
04:54 – Proxmox
07:05 – Server OS
07:25 – Drivers
08:31 – NVIDIA Container Toolkit
09:44 – Docker
10:25 – Folder Layout
11:20 – Stack Overview
12:15 – Traefik & Docker Networking Considerations
12:55 – Ollama
17:29 – Open WebUI
20:55 – Starting the Stack
23:08 – Ollama Models
25:50 – Chatting with Ollama & Performance
32:05 – searXNG
33:33 – Stable Diffusion & ComfyUI
39:54 – Stable Diffusion Models
44:42 – ComfyUI Workflows
47:00 – Verifying Model Checksum
48:48 – Integrating ComfyUI into Open WebUI
51:59 – Whisper
56:22 – Home Assistant Assist – Chat, Voice, and Text to Voice
01:04:15 – Code Suggestions and Chat Assistant in VSCode (Free Co-pilot)
Thank you for watching!