Rachel Lee Neighbors advocates for using smaller, local AI models (SLMs) over large foundation models to reduce costs, enhance security, and improve user experience by enabling offline capabilities and lower latency. She recommends a “prototype big, deploy small” approach, using tools like Arize’s Phoenix to benchmark and optimize smaller models with prompt engineering and post-processing, ultimately promoting sustainable and efficient AI deployment.
Rachel Lee Neighbors discusses the benefits of using local, smaller AI models (SLMs) instead of large foundation models like GPT-5 or Claude to reduce costs, improve security, and enhance user experience. She highlights the drawbacks of relying on large cloud-based models, including security risks, latency issues, uncontrollable inference costs, and dependency on internet connectivity. By contrast, smaller models can run efficiently on devices, offering offline capabilities, lower latency, and no ongoing API fees, making them more sustainable and user-friendly.
Rachel explains the differences between task-specific models, smaller language models (SLMs), and large language models (LLMs), emphasizing that many applications do not require the vast knowledge or multimodal capabilities of large models. SLMs, which range from millions to billions of parameters, are significantly smaller and more energy-efficient, making them suitable for on-device deployment. She notes that these models can handle common tasks like summarization or sentiment detection effectively, consuming less power and providing faster responses.
To select the best model for a given task, Rachel recommends a framework of “prototype big, deploy small.” This involves initially testing the feasibility of a task using a large foundation model and then benchmarking smaller models against a curated dataset of inputs and expected outputs. She demonstrates this approach using Arize’s open-source Phoenix tool to evaluate models like Claude Sonnet, Qwen, Llama 3.2, and Gemma 4, ultimately choosing Llama 3.2 for its balance of accuracy and latency.
Rachel also discusses how prompt engineering can improve the performance of smaller models to close the accuracy gap with larger models. By experimenting with different prompt styles—such as few-shot examples and chain-of-thought reasoning—she was able to enhance Llama 3.2’s factual consistency and reduce latency, achieving results comparable to or better than larger models like Claude Sonnet. Post-processing techniques further improved output validity and consistency, making the smaller model a viable production choice.
In conclusion, Rachel encourages developers to evaluate their current AI usage and consider transitioning to smaller, local models to save costs, improve privacy, and reduce environmental impact. She advocates for continuous evaluation and prompt tuning to maintain quality and adapt to evolving needs. Resources like Phoenix for model evaluation and Mima for social network summarization are available for those interested in experimenting with local AI models. Rachel’s message is clear: prototype with large models but deploy efficient, smaller models to build sustainable, user-friendly AI applications.