The video provides a comprehensive guide to setting up a local AI server using Proxmox 9 with LXC containers and GPU passthrough. It begins with preparing the hardware and BIOS settings, emphasizing the use of an AMD Epic 7702 processor, NVMe drives configured in a ZFS RAID 1 mirror for reliability, and enabling key BIOS features like SVM mode and UEFI boot. The presenter highlights the advantages of using a server with built-in remote keyboard, video, and mouse (IKVM) capabilities and explains the importance of configuring NUMA nodes per socket for optimal performance, especially in single-socket systems.
Next, the installation of Proxmox 9 is demonstrated, including selecting the ZFS RAID 1 option, setting up network interfaces, and configuring the hostname and static IP address. After installation, the video covers disabling the default enterprise repositories in favor of the no-subscription repository to access more up-to-date packages suitable for AI workloads. The presenter advises caution with system upgrades, recommending updates only when necessary to avoid breaking changes, particularly with Nvidia GPU drivers, which are essential for AI tasks due to their superior performance in image, video, and text generation.
The guide then moves on to installing the Open Web UI within an unprivileged LXC container, configured with appropriate CPU cores and disk size. The presenter explains the benefits of LXC containers over virtual machines for resource sharing and performance. Following this, the video details the installation of Nvidia drivers on both the Proxmox host and inside the container, including setting power limits on GPUs to optimize energy consumption without significant performance loss. The process of passing through multiple GPUs to the container is also covered, ensuring the container can fully utilize the hardware.
Once the environment is set up, the video demonstrates downloading and running AI models like Olama within the Open Web UI, showing GPU utilization and performance metrics. The presenter stresses the importance of managing disk space and resizing container storage as needed. Additionally, the video highlights the powerful snapshot and backup features of Proxmox, allowing users to quickly save and restore container states, which is invaluable for experimenting with AI models and configurations without risking data loss.
Finally, the presenter encourages viewers to explore further guides on installing and configuring other AI frameworks such as llama.c++ and VLLM, which offer improved performance and more up-to-date model support. The video concludes by emphasizing the advantages of using Proxmox for AI workloads due to its ease of backup, restore, and resource management, and thanks the community and supporters who make the content possible. Viewers are invited to engage with the channel for more advanced tutorials and updates.
