# 70B on a 4GB GPU? I Tested It (2 Minutes Per Token) **URL:** \ **Category:** Content Creators\ **Tags:** gpu, prompt-engineer-48, machine-learning, hardware, running-locally, benchmarking\ **Created:** [12 October 2026 18:48 UTC](https://www.artofsm.art/t/70b-on-a-4gb-gpu-i-tested-it-2-minutes-per-token/25757 "2026-10-12T18:48:16Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![merefield](https://www.artofsm.art/user_avatar/www.artofsm.art/merefield/32/74_2.png) [@merefield](https://www.artofsm.art/u/merefield)\ **Post date:** [12 October 2026 18:48 UTC](https://www.artofsm.art/t/70b-on-a-4gb-gpu-i-tested-it-2-minutes-per-token/25757/1 "2026-10-12T18:48:16Z")
[![](https://www.artofsm.art/uploads/default/original/3X/3/f/3fda61701975dbb81aea83393b0b21de8112c05b.jpeg "70B on a 4GB GPU? I Tested It (2 Minutes Per Token)") ](https://www.youtube.com/watch?v=eH79dNaVHls) The video demonstrates that while the ARLM project can run a 70 billion parameter model on a 4 GB GPU as claimed, the extremely slow token generation speed—about two minutes per token due to heavy disk I/O—makes it impractical for interactive use. Although 4-bit quantization improves speed somewhat, more optimized solutions like llama.cpp offer significantly faster performance by using more system RAM and efficient quantization. ---
**Author:** ![merefield](https://www.artofsm.art/user_avatar/www.artofsm.art/merefield/32/74_2.png) [@merefield](https://www.artofsm.art/u/merefield)\ **Post date:** [12 October 2026 19:12 UTC](https://www.artofsm.art/t/70b-on-a-4gb-gpu-i-tested-it-2-minutes-per-token/25757/3 "2026-10-12T19:12:52Z")
The video investigates the claim that a 72 billion parameter model can run on a single 4 GB GPU using the Air LLM (ARLM) project without quantization, distillation, or pruning. The presenter tests the model on an RTX A5000 GPU with 24 GB memory but focuses on the actual peak GPU memory used, which was about 2.4 GB, confirming the claim that the model fits within 4 GB GPU memory. However, the major bottleneck is disk speed because the model is split into 80 layers stored on disk, and each token generation requires loading all layers sequentially from disk, resulting in extremely slow token generation times—about two minutes per token. The ARLM project’s README highlights the memory efficiency but does not provide any speed benchmarks, which the presenter points out as a significant omission. The model files in 16-bit precision total around 136 GB, and each token generation involves reading this entire data from disk, making disk I/O the limiting factor rather than GPU memory. The presenter’s initial speed prediction was 30 to 60 seconds per token, but actual measurements showed about 121 seconds per token due to slower disk read speeds than initially assumed. This makes the model impractical for interactive chat applications. The presenter also tested the 4-bit quantized mode, which the README claims can speed up inference by up to three times. However, a bug in the code initially prevented the 4-bit mode from running. After bypassing the bug, the 4-bit mode did run and showed a speed improvement, generating tokens about five times faster than the 16-bit mode when the model files could fit into the operating system’s cache. This speedup is attributed mainly to reduced disk I/O rather than inherent computational efficiency, and the 4-bit mode still uses under 4 GB of GPU memory. For comparison, the presenter ran the same model using llama.cpp with a 4-bit quantized version, which runs about 35 times faster than ARLM in 16-bit mode and about seven times faster than ARLM in 4-bit mode with a warm cache. Llama.cpp requires more system RAM (about 44 GB) but achieves much better speed by keeping more layers in memory and leveraging efficient quantization. The trade-off is clear: ARLM can run on machines with limited RAM but at a severe speed cost, while llama.cpp requires more RAM but delivers practical performance. In conclusion, the video confirms that ARLM can indeed run a 70 billion parameter model on a 4 GB GPU as claimed, but the speed is prohibitively slow for interactive use, making it suitable only for batch jobs or one-off runs where waiting is acceptable. The 4-bit quantized mode offers some speed improvements but still falls far behind more optimized solutions like llama.cpp. The presenter encourages viewers to try the setup themselves and share their results, emphasizing that disk speed and caching significantly impact performance. The video provides a detailed, transparent benchmark that fills in the gaps left by the project’s README. ## Useful Links - [AirLLM GitHub Repository](https://github.com/lyogavin/airllm) — Directly related to the main claim and tested software. - [AirLLM utils.py Source Code (Lines 447-468)](https://github.com/lyogavin/airllm/blob/main/air_llm/airllm/utils.py) — Explains a critical bug affecting the 4-bit mode speed claim. - [llama.cpp GitHub Repository](https://github.com/ggml-org/llama.cpp) — Provides a performance baseline and alternative approach to running large models. - [Qwen2.5-72B-Instruct GGUF Quantized Model on Hugging Face](https://huggingface.co/bartowski/Qwen2.5-72B-Instruct-GGUF) — Quantized model used in the comparative benchmark.