# \#quantisation

**URL:** https://www.artofsm.art/tag/quantisation/184.md

[Latest](https://www.artofsm.art/latest.md) · [Categories](https://www.artofsm.art/categories.md) · [Tags](https://www.artofsm.art/tags.md)

---

## [LLM Quantization Explained! Which AI Model Should You Download?](https://www.artofsm.art/t/llm-quantization-explained-which-ai-model-should-you-download/28878)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [20 November 2026 15:51 UTC](https://www.artofsm.art/t/llm-quantization-explained-which-ai-model-should-you-download/28878 "2026-11-20T15:51:24Z")

</div>

The video explains that AI model downloads involve different layers—file formats (like GGUF), engines (such as MLX), and number formats (like NVFP4 and Q4)—which affect performance based on hardware compati…

---

## [Qwen3.8 on 8GB VRAM? Unsloth Dynamic V3.1 Reality Check (RTX 4060 vs 4090)](https://www.artofsm.art/t/qwen3-8-on-8gb-vram-unsloth-dynamic-v3-1-reality-check-rtx-4060-vs-4090/28478)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [15 November 2026 15:52 UTC](https://www.artofsm.art/t/qwen3-8-on-8gb-vram-unsloth-dynamic-v3-1-reality-check-rtx-4060-vs-4090/28478 "2026-11-15T15:52:15Z")

</div>

The video evaluates running the Qwen 3.8 language model with Unsloth’s dynamic quantization on GPUs with 8 GB and 24 GB VRAM, finding that while the model can run on an 8 GB RTX 4060 with limited speed and …

---

## [Deepseek V4.1 Flash Runs With No GPU Just 64GB RAM](https://www.artofsm.art/t/deepseek-v4-1-flash-runs-with-no-gpu-just-64gb-ram/28361)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [13 November 2026 15:51 UTC](https://www.artofsm.art/t/deepseek-v4-1-flash-runs-with-no-gpu-just-64gb-ram/28361 "2026-11-13T15:51:18Z")

</div>

The video showcases running the 284 billion parameter Deepseek V4.1 Flash AI model on consumer hardware with just 64GB RAM and no GPU using advanced compression techniques, but highlights significant qualit…

---

## [HyperQwen: I Ran a 27B AI on One GPU! (Qwen3.8-27B vs Stock vLLM, Real Benchmarks)](https://www.artofsm.art/t/hyperqwen-i-ran-a-27b-ai-on-one-gpu-qwen3-8-27b-vs-stock-vllm-real-benchmarks/28465)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [11 November 2026 03:42 UTC](https://www.artofsm.art/t/hyperqwen-i-ran-a-27b-ai-on-one-gpu-qwen3-8-27b-vs-stock-vllm-real-benchmarks/28465 "2026-11-11T03:42:46Z")

</div>

The video demonstrates that HyperQwen, a patched version of VLLM using 4-bit quantization and optimizations, can run the 27 billion parameter Qwen 3.8-27B model efficiently on a single consumer GPU, achievi…

---

## [NVIDIA's Secret 2 Bit Model](https://www.artofsm.art/t/nvidias-secret-2-bit-model/27960)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [9 November 2026 14:24 UTC](https://www.artofsm.art/t/nvidias-secret-2-bit-model/27960 "2026-11-09T14:24:27Z")

</div>

NVIDIA’s Nemotron 3.5 Bolt 74B A9B model leverages advanced techniques like pruning, distillation, and quantization-aware training to achieve efficient 2-bit quantization, enabling a large 74 billion parame…

---

## [I Run Full DeepSeek v4.1 Flash on a 12GB GPU](https://www.artofsm.art/t/i-run-full-deepseek-v4-1-flash-on-a-12gb-gpu/27808)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [6 November 2026 15:50 UTC](https://www.artofsm.art/t/i-run-full-deepseek-v4-1-flash-on-a-12gb-gpu/27808 "2026-11-06T15:50:41Z")

</div>

The video demonstrates that running the massive 510 GB DeepSeek V4.1 Flash model on a 12 GB GPU is feasible by focusing on the critical metric of bytes per word processed per token, highlighting trade-offs …

---

## [Every Way to Get 256 GB of VRAM for Local AI at Full Context](https://www.artofsm.art/t/every-way-to-get-256-gb-of-vram-for-local-ai-at-full-context/27108)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [29 October 2026 15:51 UTC](https://www.artofsm.art/t/every-way-to-get-256-gb-of-vram-for-local-ai-at-full-context/27108 "2026-10-29T15:51:21Z")

</div>

The video examines various hardware options for achieving 256 GB of VRAM to run large AI models like DeepSeek V4.1 Flash locally, highlighting the trade-offs between quantization levels, cost, performance, …

---

## [What Is Quantization? How AI Models Get Smaller & Faster](https://www.artofsm.art/t/what-is-quantization-how-ai-models-get-smaller-faster/27090)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [29 October 2026 11:20 UTC](https://www.artofsm.art/t/what-is-quantization-how-ai-models-get-smaller-faster/27090 "2026-10-29T11:20:25Z")

</div>

Quantization is a technique that reduces the precision of AI model parameters to lower memory usage and improve computational efficiency, enabling large language models to run faster and on less powerful ha…

---

## [Qwen 3.8 N-Gram Streaming: Max Quants on Low RAM (vs Kimi 3.1 & Qwen 4) 🤯](https://www.artofsm.art/t/qwen-3-8-n-gram-streaming-max-quants-on-low-ram-vs-kimi-3-1-qwen-4/26733)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [23 October 2026 16:46 UTC](https://www.artofsm.art/t/qwen-3-8-n-gram-streaming-max-quants-on-low-ram-vs-kimi-3-1-qwen-4/26733 "2026-10-23T16:46:42Z")

</div>

The video demonstrates how streaming engrams from storage enables running large language models like Qwen 3.8 with high-quality quantization on machines with limited RAM, significantly reducing memory usage…

---

## [I Tested Every Best Local AI For GPU (8GB TO 24GB VRAM)](https://www.artofsm.art/t/i-tested-every-best-local-ai-for-gpu-8gb-to-24gb-vram/26541)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [21 October 2026 10:51 UTC](https://www.artofsm.art/t/i-tested-every-best-local-ai-for-gpu-8gb-to-24gb-vram/26541 "2026-10-21T10:51:00Z")

</div>

The video explains that the performance of local AI models depends not just on GPU VRAM size but critically on the specific model file used, highlighting differences between dense and mixture of experts (Mo…

---

## [I Ran IBM's New 8B Model in 6GB. Here's the Catch](https://www.artofsm.art/t/i-ran-ibms-new-8b-model-in-6gb-heres-the-catch/26102)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [17 October 2026 01:53 UTC](https://www.artofsm.art/t/i-ran-ibms-new-8b-model-in-6gb-heres-the-catch/26102 "2026-10-17T01:53:45Z")

</div>

IBM’s Granite Switch 4.1 is an 8 billion parameter model integrating chat, safety, decision-making, and speech transcription via small activated adapters, allowing it to run efficiently in under 6 GB of VRA…

---

## [Baidu's Unlimited OCR on an 8GB Laptop: 25 Tests](https://www.artofsm.art/t/baidus-unlimited-ocr-on-an-8gb-laptop-25-tests/25842)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [13 October 2026 17:54 UTC](https://www.artofsm.art/t/baidus-unlimited-ocr-on-an-8gb-laptop-25-tests/25842 "2026-10-13T17:54:42Z")

</div>

The video evaluates Baidu’s Unlimited OCR model running on an 8GB laptop using a 4-bit quantized version, demonstrating strong accuracy on clean and structured documents but showing limitations with complex…

---

## [I Ran Nace.AI's Drex v1.5 Locally (llama.cpp, Q4\_K\_M, Real Demos)](https://www.artofsm.art/t/i-ran-nace-ais-drex-v1-5-locally-llama-cpp-q4-k-m-real-demos/25647)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [11 October 2026 09:54 UTC](https://www.artofsm.art/t/i-ran-nace-ais-drex-v1-5-locally-llama-cpp-q4-k-m-real-demos/25647 "2026-10-11T09:54:24Z")

</div>

The video demonstrates running NACE AI’s Drex V1.5, a 9-billion parameter decision model that outputs probabilities for decision-making tasks, locally on an 8GB GPU laptop by using llama.cpp with Q4\_K\_M qua…

---

## [Aleph Alpha's Kolibri-1: 78B Parameter Open-Weight AI Model Now Runs Locally](https://www.artofsm.art/t/aleph-alphas-kolibri-1-78b-parameter-open-weight-ai-model-now-runs-locally/25316)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 0\
**Last updated:** [8 October 2026 21:01 UTC](https://www.artofsm.art/t/aleph-alphas-kolibri-1-78b-parameter-open-weight-ai-model-now-runs-locally/25316 "2026-10-08T21:01:23Z")

</div>

German AI company Aleph Alpha has released Kolibri-1, a 78 billion parameter open-weight language model designed for both English and German, marking a significant step for organizations seeking data sovereignty and o…

---

## [Microsoft Joins the Local AI Push](https://www.artofsm.art/t/microsoft-joins-the-local-ai-push/25283)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [8 October 2026 15:20 UTC](https://www.artofsm.art/t/microsoft-joins-the-local-ai-push/25283 "2026-10-08T15:20:46Z")

</div>

Microsoft showcased advancements in local AI at their recent Windows and Surface event, highlighting hybrid intelligence with “intelligent routing” that dynamically balances on-device and cloud processing, …

---

## [Forget Claude, Your Gaming PC Run This 78B Local AI](https://www.artofsm.art/t/forget-claude-your-gaming-pc-run-this-78b-local-ai/25176)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [7 October 2026 04:24 UTC](https://www.artofsm.art/t/forget-claude-your-gaming-pc-run-this-78b-local-ai/25176 "2026-10-07T04:24:44Z")

</div>

Aleph Alpha’s Kolibri 1 is a massive 78 billion parameter AI model that, thanks to its mixture of experts design and four-bit quantization, can be run on high-memory gaming PCs without GPUs, though it requi…

---

## [You Can Run A 27B AI On A 16GB GPU At 100K Tokens/S \[NInfer\]](https://www.artofsm.art/t/you-can-run-a-27b-ai-on-a-16gb-gpu-at-100k-tokens-s-ninfer/25157)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [6 October 2026 21:37 UTC](https://www.artofsm.art/t/you-can-run-a-27b-ai-on-a-16gb-gpu-at-100k-tokens-s-ninfer/25157 "2026-10-06T21:37:15Z")

</div>

Developer Roofkid’s Ninfer 4080 engine enables running a 27 billion parameter AI model with a 100,000 token context on a single 16 GB RTX 4080 GPU by compressing model weights to 3 bits and optimizing memor…

---

## [DeepSeek V4.1: Which Version To Run Locally](https://www.artofsm.art/t/deepseek-v4-1-which-version-to-run-locally/25130)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [6 October 2026 14:44 UTC](https://www.artofsm.art/t/deepseek-v4-1-which-version-to-run-locally/25130 "2026-10-06T14:44:04Z")

</div>

DeepSeek V4.1 offers two local versions: Q4, which requires very high memory (around 512 GB) and closely matches the original model’s performance, and Q2, which uses less memory (about 152 GB) with some spe…

---

## [This 12B Model Rewrites AI Text, and It Fits on an 8GB Laptop (Q4 test)](https://www.artofsm.art/t/this-12b-model-rewrites-ai-text-and-it-fits-on-an-8gb-laptop-q4-test/25111)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [6 October 2026 07:00 UTC](https://www.artofsm.art/t/this-12b-model-rewrites-ai-text-and-it-fits-on-an-8gb-laptop-q4-test/25111 "2026-10-06T07:00:02Z")

</div>

The video showcases Humanizer, a 12 billion parameter AI model that rewrites AI-generated text to sound more human-like while preserving factual details, and can run efficiently on an 8 GB laptop GPU using …

---

## [Kolibri-1 Just Launched: I Converted the 78B Model to GGUF (Step by Step)](https://www.artofsm.art/t/kolibri-1-just-launched-i-converted-the-78b-model-to-gguf-step-by-step/25021)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [4 October 2026 23:54 UTC](https://www.artofsm.art/t/kolibri-1-just-launched-i-converted-the-78b-model-to-gguf-step-by-step/25021 "2026-10-04T23:54:55Z")

</div>

The video details the step-by-step conversion of Aleph Alpha’s newly released Colibri-1 78B parameter language model from its original FP8 safe tensor format into a 4-bit quantized GGUF format compatible wi…

---

## [Qwen 27B: Which Quant Should You Run?](https://www.artofsm.art/t/qwen-27b-which-quant-should-you-run/24961)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [3 October 2026 19:20 UTC](https://www.artofsm.art/t/qwen-27b-which-quant-should-you-run/24961 "2026-10-03T19:20:31Z")

</div>

The video recommends starting with the Q4K MQAT quant for running the Qwen 27B model on a 24 GB GPU due to its lower memory usage and sufficient coding performance, while suggesting users conduct controlled…

---

## [1.58 bit LLMs Are BETTER, But Why No One Uses Them?](https://www.artofsm.art/t/1-58-bit-llms-are-better-but-why-no-one-uses-them/24935)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [3 October 2026 15:07 UTC](https://www.artofsm.art/t/1-58-bit-llms-are-better-but-why-no-one-uses-them/24935 "2026-10-03T15:07:26Z")

</div>

In 2024, Microsoft introduced 1.58-bit ternary quantization for large language models, significantly reducing computational costs and memory usage while maintaining performance comparable to 16-bit models, …

---

## [NVIDIA’s Trick To Make Local AI 2X Faster On Your GPU](https://www.artofsm.art/t/nvidia-s-trick-to-make-local-ai-2x-faster-on-your-gpu/24702)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [1 October 2026 00:26 UTC](https://www.artofsm.art/t/nvidia-s-trick-to-make-local-ai-2x-faster-on-your-gpu/24702 "2026-10-01T00:26:16Z")

</div>

NVIDIA’s new 4-bit floating-point format, NVFP4, enhances AI model efficiency on Blackwell GPUs by speeding up prompt processing and reducing memory usage, though actual performance gains vary based on hard…

---

## [GSQ Explained: Smaller Quants, Same Great Performance](https://www.artofsm.art/t/gsq-explained-smaller-quants-same-great-performance/24658)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [30 September 2026 14:24 UTC](https://www.artofsm.art/t/gsq-explained-smaller-quants-same-great-performance/24658 "2026-09-30T14:24:37Z")

</div>

GSQ is an advanced quantization method for large language models that uses probabilistic training with Gumbel Softmax sampling to compress weights into very low bit levels, achieving smaller model sizes wit…

---

## [What It Actually Costs to Run DeepSeek V4.1 Flash Locally?](https://www.artofsm.art/t/what-it-actually-costs-to-run-deepseek-v4-1-flash-locally/23887)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [13 September 2026 15:50 UTC](https://www.artofsm.art/t/what-it-actually-costs-to-run-deepseek-v4-1-flash-locally/23887 "2026-09-13T15:50:30Z")

</div>

DeepSeek V4.1 Flash, despite its impressive performance and permissive MIT license, is impractical to run locally due to massive storage requirements (510 GB), slow generation speeds on modest hardware, and…

---

## [I Ran A 27B Model On A Hand-Sized PC… Didn't Expect This](https://www.artofsm.art/t/i-ran-a-27b-model-on-a-hand-sized-pc-didnt-expect-this/23885)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [13 September 2026 15:02 UTC](https://www.artofsm.art/t/i-ran-a-27b-model-on-a-hand-sized-pc-didnt-expect-this/23885 "2026-09-13T15:02:17Z")

</div>

The video demonstrates running a large 27 billion parameter AI model on a compact hand-sized PC, revealing that while feasible, performance is limited by memory bandwidth and shared system memory between CP…

---

## [EXL3 Explained: Smaller Local Models, Less Quality Loss](https://www.artofsm.art/t/exl3-explained-smaller-local-models-less-quality-loss/23993)

<div class="topic-metadata">

**Author:** [@artesia](https://www.artofsm.art/u/artesia)\
**Replies:** 1\
**Last updated:** [8 September 2026 19:45 UTC](https://www.artofsm.art/t/exl3-explained-smaller-local-models-less-quality-loss/23993 "2026-09-08T19:45:46Z")

</div>

The video introduces EXL3, an advanced quantization method that reduces AI model size by grouping weight rounding to minimize errors, enabling smaller local models like Qwen 3.8 27B to maintain high quality…

---

## [LLM Quantization Explained: The Q4 Quality Trap](https://www.artofsm.art/t/llm-quantization-explained-the-q4-quality-trap/23640)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [7 September 2026 07:20 UTC](https://www.artofsm.art/t/llm-quantization-explained-the-q4-quality-trap/23640 "2026-09-07T07:20:54Z")

</div>

The video explains that quantization compresses large language models to run efficiently on local hardware by reducing numerical precision, but the quality and method of quantization—especially at lower bit…

---

## [I Quantized Qwen3.8-27B. “4-Bit” Doesn’t Mean What You Think](https://www.artofsm.art/t/i-quantized-qwen3-8-27b-4-bit-doesn-t-mean-what-you-think/23494)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [3 September 2026 18:11 UTC](https://www.artofsm.art/t/i-quantized-qwen3-8-27b-4-bit-doesn-t-mean-what-you-think/23494 "2026-09-03T18:11:34Z")

</div>

The video explains that quantizing the Qwen 3.8-27B model as “4-bit” is a nuanced process involving varied precision levels across different model parts, resulting in a lossy but optimized compression that …

---

## [I Tested Every Qwen3.8-27B Quant: Here’s the Best One For Your GPU](https://www.artofsm.art/t/i-tested-every-qwen3-8-27b-quant-here-s-the-best-one-for-your-gpu/23398)

<div class="topic-metadata">

**Author:** [@merefield](https://www.artofsm.art/u/merefield)\
**Replies:** 1\
**Last updated:** [1 September 2026 15:50 UTC](https://www.artofsm.art/t/i-tested-every-qwen3-8-27b-quant-here-s-the-best-one-for-your-gpu/23398 "2026-09-01T15:50:57Z")

</div>

The video explores the challenges of running quantized versions of Alibaba’s Qwen 3.8 27B model, revealing that improper quantization—especially of critical components like the KV cache—can silently degrade…

[Next page](https://www.artofsm.art/tag/quantisation/184.md?match_all_tags=true&page=1&tags%5B%5D=quantisation)
