Qwen3.8-27B-NVFP4-MTP-HIGHEST.gguf is a local model build I would test on a single RTX 5090. Its mixed-precision quantization and multi-token-prediction support make it an interesting candidate, but this note does not contain a controlled benchmark. Selection depends on model quality, prompt processing, generation speed, context length, power, and the agent workload.

What this build combines

The build combines four choices:

  1. NVFP4 quantization: The memory-heavy MLP layers use a low-bit format intended to reduce model memory and bandwidth. Actual VRAM use depends on the exact files, runtime, context, KV-cache format, and offload settings, so it must be measured with the selected release.
  2. MTP speculative decoding: If the chosen runtime and model files support the model’s multi-token-prediction head, speculative decoding may improve generation throughput. The gain is workload- and configuration-dependent and should be compared with MTP disabled.
  3. Q8attn mixed precision for long context: The MLP stays in NVFP4 to save VRAM, while the attention and DeltaNet projections remain at Q8_0 rather than taking the same 4-bit treatment. The higher-precision attention path trades memory for a possible quality benefit; long-context quality and memory use still need task-specific testing.
  4. Optional vision projector: A compatible mmproj file can add local image input, but compatibility and memory use depend on the exact model and runtime release.

The repository’s model card describes a large native context window, but a weight file fitting in VRAM does not show that the full context will fit. KV cache, the optional vision projector, runtime buffers, CUDA allocations, and the MTP path all consume additional memory. I would report only contexts that boot and complete a fixed long-context test while recording peak VRAM.

What my 4090 run does—and does not—show

I ran a version of this checkpoint on my local RTX 4090 and found it responsive enough for prototyping. I did not preserve the exact commit, command line, prompt length, context, batch settings, power limit, or separate prompt-processing and generation timings, so that observation is not a reproducible benchmark.

It also cannot be compared directly with third-party RTX 5090 numbers. A useful comparison would hold the model files, runtime commit, prompt, generated-token count, context, KV-cache, sampling, thermal state, and MTP settings constant, then report prompt-processing and generation throughput separately over repeated runs.

  • Record provenance: model repository and file hashes, runtime version/commit, driver, and command line.
  • Separate measurements: model load time, prompt processing, generation, peak VRAM, power, and output quality.
  • Use the same workload: identical prompts and output lengths, with warm-up runs and repeated samples on each card.

Checkpoint reference

esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF. The model card documents the available mixed-precision files, embedded MTP head, invocation flag, and author-run measurements. Those reported measurements are not my benchmark and are not used here as a controlled 4090/5090 comparison.