Large Mixture-of-Experts (MoE) models such as DeepSeek-V4, Qwen3.6 MoE, and GLM-5.2 are difficult to run on consumer hardware because their checkpoints exceed GPU memory. Common alternatives include datacenter GPUs, CPU/GPU offloading, and low-bit quantization, each with cost, speed, or quality tradeoffs.
UC Berkeley researcher Shuo Yang (@Andy_ShuoYang) shared a paper co-authored with researchers from UC Berkeley SkyLab and MIT, including Kurt Keutzer, Song Han, Matei Zaharia, and Ion Stoica. The team introduced FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution (arXiv:2608.16157).
The paper's claim: FreeToken serves selected MoE checkpoints on a gaming PC or laptop by coordinating GPU memory, host memory, and PCIe transfers. The reported speeds are fast enough for interactive use on the tested systems, with no per-token API charge.
Reported benchmark speeds
FreeToken treats VRAM, host DDR5 memory, and PCIe bandwidth as parts of one serving system. The following figures are reported by the authors, not independent measurements from this lab:
| Model | Total Parameters | Consumer Target Hardware | Inference Speed |
|---|---|---|---|
| Qwen3.6 35B (A3B) | 35B (3B active) |
8GB RTX 4060 Laptop GPU | 39 – 83 tok/s |
| DeepSeek-V4-Flash | 284B (16B active) |
RTX 5090 (32GB VRAM Desktop) | 22 – 25 tok/s |
| GLM-5.2 MoE | 753B (48B active) |
RTX PRO 6000 Workstation | 15 tok/s |
How FreeToken works
Choosing between GPU transfer and CPU execution
FreeToken measures PCIe bus bandwidth ($B_P$) and host memory bandwidth ($B_H$) rather than relying only on a static GPU/CPU split. It routes active experts either through PCIe DMA to GPU tensor cores or executes them on CPU host cores with vectorized AVX-512/AMX kernels.
Overlapping weight transfer and computation
During prefill, many experts in a layer may be active at once. FreeToken uses ping-pong double buffering: while the GPU computes layer $l$, the PCIe DMA controller streams layer $l+1$'s weights into an alternating buffer. This is intended to overlap transfer time with GPU matrix multiplication.
Reusing cached state in repeated tool loops
Agent harnesses such as Claude Code, Cursor, and Codex repeatedly extend conversation history with tool results and file edits. FreeToken identifies boundaries such as <think> segments, tool returns, and edits, then pins KV-cache checkpoints so a branch can reuse prior computation and process only changed tokens.
Implications for local agent workloads
Agent harnesses, discussed in the notes on Agent Harnesses and Codex as a Platform, often make many model calls inside tool loops.
Local inference avoids API rate limits and per-token charges, although it still has hardware and electricity costs. It also changes where data is processed:
- No API token charge: Multi-turn tool loops, test suites, and refactor runs execute on local hardware.
- Local data processing: Source code and internal data can remain on the workstation when the full workflow is configured locally.
- No provider rate limit: The local server is constrained by its own throughput rather than an external token bucket.
Lab evaluation plan
A useful local test would run the authors' implementation on an Intel Core i5-13600K, 96GB DDR5 RAM, and an NVIDIA GeForce RTX 4080 16GB. I have not measured FreeToken on this machine, so I am not publishing throughput or latency estimates for it.
Reproducible protocol
- Pin the software: Record the FreeToken commit, model checkpoint, quantization, drivers, kernel, and launch command.
- Fix the workload: Use the same saved prompts, context lengths, output-token cap, batch size, and concurrency for every comparison.
- Separate the timings: Report time to first token and decode throughput independently. Run warmups before repeated measured runs and retain every result rather than only the fastest one.
- Log the machine: Capture GPU memory, host memory, PCIe utilization, power, and temperatures during each run.
- Publish the evidence: Release the configuration, prompt set, raw logs, summary script, and any failed runs. Do not set a throughput target before collecting the measurements.