Perplexity built a custom inference engine for Apple silicon and one model: Qwen3.6-35B-A3B. Lily, as it is called, is a single Rust process. It loads the checkpoint, runs the generation loop, serves an OpenAI-compatible API, and executes the model with custom Metal kernels. A standalone demo is already publicly available on GitHub. Neither PyTorch nor MLX is in the execution path.

Why write your own engine when MLX-LM exists? Qwen3.6 is a hybrid: mixture-of-experts routing, Gated DeltaNet recurrent states, and ten full-attention layers. Each produces a different workload shape. A general framework has to serve whatever architecture comes along, so it picks kernels that work broadly. An engine dedicated to one model can specialize at the runtime level, coordinating kernels and data movement around Qwen's fixed structure.

The optimizations are phase-specific. Prefill keeps expert routing on the GPU, holds the recurrent scan in registers, and chunks long prompts. Decode minimizes bytes moved per token: packed grouped-query attention reads, and a separate attention layout for long contexts. Measured in isolation, the register-resident scan adds 5.6 percent at a 2K prompt, GQA packing adds 23.8 percent at 32K context, and the long-context layout adds 40.2 percent at 128K.

On one M5 Max (40-core GPU, 128 GB), Lily averages 1.23x MLX-LM's prefill throughput and 1.35x its decode throughput, from 256 to 128K tokens, same 4-bit checkpoint bytes, one request at a time. Output is nearly identical: 0.04 percent higher perplexity, same top token 96.35 percent of the time.

The most honest part of the post is the limits section. Speculative decoding made batch-1 decode 18 percent slower, so they dropped it. The MoE GEMMs run at 97.9 percent of the fastest sustained weight-read rate, and the GEMVs at 90.3 percent; removing arithmetic from the sparse GEMV changed throughput by 0.2 percent. They are close to the memory-bandwidth wall.

The caveat that matters is the baseline. Everything is measured against MLX-LM, Apple's general-purpose framework. llama.cpp with its Metal backend is the other engine people actually run on Macs, and it has historically been strong on decode. A specialized engine beating a general one by 20 to 35 percent is expected. What this post does not tell you is how Lily compares with llama.cpp, or how much of the 1.35x is real platform gain versus closing the gap to what llama.cpp already gets. That benchmark is the one I want to see.

Editorial

Where it leaves the question

Taken as a vendor post, this one is unusually good: specific ablations, honest failures, measured hardware ceilings. Taken as an answer to which engine you should run on a Mac, it is incomplete until the same benchmark is run against llama.cpp with Metal. The 1.35x is probably real. It is just not proven against the engine Mac users actually run.

Sources

Perplexity Engineering, "Optimizing On-Device Inference for Apple Silicon" (1 Sep 2026): The primary source. Source of all engine details, ablation numbers (register-resident scan +5.6%, GQA packing +23.8%, long-context layout +40.2% at 128K), the 1.23x prefill / 1.35x decode figures, the 97.9% hardware ceiling and 0.2% arithmetic-removal test, the speculative-decoding failure, and the 0.04% perplexity / 96.35% token agreement comparison.

huggingface.co/Qwen/Qwen3.6-35B-A3B: The model the engine is built for. Hybrid architecture: mixture-of-experts routing, Gated DeltaNet recurrent states, full attention.

github.com/ml-explore/mlx: Apple's MLX framework and MLX-LM, the baseline the benchmark compares against. The note's caveat concerns the absence of a comparable llama.cpp Metal benchmark, which the post does not provide.

github.com/perplexityai/pplx-garden/tree/main/lily: Perplexity's standalone Lily demo, already publicly available. A Rust runtime with custom Metal kernels for Qwen3.6-35B-A3B on Apple silicon.