FreeToken, a new edge-native mixture-of-experts (MoE) serving engine, has been introduced by researchers from UC Berkeley and UT Austin. This innovation allows the 753 billion parameter GLM-5.2 model to run on a single workstation GPU, addressing the accessibility issues surrounding powerful AI models. The team emphasizes that the key to unleashing this potential lies not in hardware but in an effective serving system that transforms personal machines into unified, elastic inference platforms.
FreeToken can deploy a 35 billion parameter model at interactive speeds on an 8 GB laptop GPU, and 284 billion parameters on a gaming desktop. The engine continuously maps computation and model state across the available GPU, CPU, memory, and interconnect bandwidth, enabling developers and small teams to leverage local inference capabilities without requiring expensive datacenter-class GPU clusters.
FreeToken is available under the Apache-2.0 license on GitHub and has been published on PyPI as freetoken v0.1.2. It can be installed via a one-click desktop app for Windows and Linux from flashml.ai. The command-line interface targets Linux x86_64 systems with NVIDIA GPUs and exposes endpoints compatible with OpenAI and Anthropic models.
This engine is particularly well-suited for solo developers, startups, and small to medium-sized business engineering teams, especially those facing high costs for agent token bills. Industries such as healthcare, legal, and defense will find FreeToken beneficial, as it allows data to remain on the local machine, ensuring privacy and compliance.
FreeToken’s architecture utilizes several mechanisms to optimize performance. It employs bandwidth-adaptive execution, semantic-aware caching, and elastic memory management to enhance efficiency. For example, the engine can sustain 77 to 83 tokens per second on the Qwen3.6-35B-A3B model utilizing an RTX 5090 GPU, significantly outperforming existing solutions.
Additionally, FreeToken addresses several limitations found in current engines. It overcomes issues such as prefill inefficiencies that compromise sparsity and static placement problems that lead to CPU bottlenecks. By isolating these failure modes, FreeToken is designed to maximize the utility of consumer hardware, providing a practical solution for running large AI models.
The research team has noted that while the engine’s performance is impressive, further independent verification of its claims is necessary. Nevertheless, FreeToken represents a significant advance in making powerful AI models accessible to a broader audience, particularly in environments where budget constraints limit access to high-performance computing resources.
FreeToken’s launch marks a pivotal moment in the AI landscape, as it opens up new possibilities for local inference on consumer-grade hardware. With its user-friendly deployment options and robust performance metrics, it positions itself as a valuable tool for developers and organizations aiming to harness the power of advanced AI models without the traditional infrastructure costs.
Featured image credit
