OMLX is an open-source caching solution for local AI coding agents on Macs that significantly reduces response latency during long sessions by preserving computation across similar prompts and server restarts, addressing inefficiencies in how conversation context is handled. While it doesn’t speed up token generation itself, OMLX enhances user experience by minimizing repeated full recomputations, making extended interactions feel as responsive as initial ones.
Many Mac users running local AI coding agents experience a frustrating slowdown after initial smooth interactions. At first, the local AI model responds quickly, promising a seamless, private, and cost-free coding assistant. However, by around the 12th interaction, response times start to lag significantly, sometimes taking up to 90 seconds by the 20th turn. This slowdown isn’t due to hardware limitations or model degradation but stems from a caching issue where the system recomputes the entire conversation from scratch each time, discarding previously cached data due to slight changes in the prompt.
The root cause lies in how local inference servers, including Apple’s MLX framework, handle conversation context. Coding agents repeatedly send prompts with nearly identical prefixes but appended new tool results, causing the servers to treat these as completely new inputs and invalidate the key-value (KV) cache. This leads to expensive recomputations for every turn, especially in long sessions with extensive context. OMLX, an open-source project by Jun Kim with over 18,600 stars, addresses this specific caching inefficiency by implementing a two-tier cache system that preserves computation across similar prompts and even server restarts.
OMLX’s caching system uses a hot in-memory tier for frequently accessed cache blocks and a cold SSD tier for less frequently used blocks, stored in a format optimized for quick retrieval. This approach drastically reduces the time to first token in long agentic sessions by reusing cached computations instead of starting from scratch. However, OMLX does not speed up the actual token generation once it begins; it only reduces the initial wait time before the model starts responding. This distinction is crucial for users to understand before adopting OMLX.
Setting up OMLX requires macOS 15 or later, Python 3.11 to 3.13, and an Apple Silicon chip (M1 to M4). Installation can be done via a user-friendly Mac app, Homebrew, or building from source, with the Mac app recommended for ease and full feature support. OMLX automatically detects models from existing caches and supports integration with popular coding agents like Claude Code, Cursor, Codex, and Pi Agent, each requiring specific configuration steps. The project also offers an admin dashboard for model management and monitoring cache performance during sessions.
In summary, OMLX is a targeted solution for the specific problem of caching inefficiency in local AI coding agents on Macs, making long sessions feel as responsive as the initial turns. It does not improve model intelligence or generation speed but significantly enhances user experience by reducing latency caused by repeated full recomputations. While it benefits users running extended agentic sessions, those using local models for simple one-off chats may see little difference. The project’s success and growing popularity highlight the importance of addressing software-level bottlenecks rather than just upgrading hardware.