Before you start
What happened
After FreeToken Desktop updated to v0.2.0-beta.21 (bundling engine freetoken 0.1.3+gb332d3a97), serving an FP8 block-quantized MoE model fails at startup with:
KernelSelectionError: kernel 'triton' was requested but cannot run here:
has no CPU executor format; decode must run on the GPU
The scheduler process dies and the API server exits with code 15.
Environment: Windows 11, RTX 5060 Ti 16 GB (14.8 GiB free), Python 3.12, engine 0.1.3+gb332d3a97 (win_amd64 wheel from the beta channel).
Model: Qwen3.6-35B-A3B-FP8 — FP8 block quantization with weight_block_size: [128, 128] and bf16 scales.
Command:
ft serve --model F:\models\Qwen3.6-35B-A3B-FP8 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.85
Root cause analysis: Since free VRAM (14.8 GiB) cannot hold the model, the engine auto-selects moe_strategy='offload', which sets the decode target to CPU. However, for FP8-block-quantized MoE layers,
freetoken/layers/quantization/moe/fp8_block.py
declares only one candidate kernel, TritonFp8BlockMoEKernel, which has cpu_format = None and rejects CPU decode (cpu_ok=False: "has no CPU executor format; decode must run on the GPU"). The alternative dsv4 kernel is skipped because it only serves e8m0 block scales (this checkpoint uses bf16 scales). With zero usable kernels, the new kernel-selection check introduced in 0.1.3 raises KernelSelectionError.
Notably, fp8_block appears to be the only quantization format without a CPU executor format — nvfp4 (cpu_format="nvfp4"), mxfp4 (cpu_format="ds_fp4"), and unquantized (cpu_format="bf16") all have one.
Regression: The same model and command worked on engine 0.1.2 (which has no kernel-selection validation). Rolling back to 0.1.2 restores service.
Request: Please add a CPU executor format for fp8_block, or allow offload-strategy decode of fp8_block experts to run on the GPU.
Desktop app version
0.2.0-beta.21
OS
Windows 11
OS details
No response
GPU and driver
RTX 5060Ti 16GB
CPU and system RAM
i7-12700 64GB
Checkpoint
Qwen3.6-35B-A3B-FP8
Model settings
No response
Engine log
Anything else
No response
Before you start
What happened
After FreeToken Desktop updated to v0.2.0-beta.21 (bundling engine freetoken 0.1.3+gb332d3a97), serving an FP8 block-quantized MoE model fails at startup with:
KernelSelectionError: kernel 'triton' was requested but cannot run here:
has no CPU executor format; decode must run on the GPU
The scheduler process dies and the API server exits with code 15.
Environment: Windows 11, RTX 5060 Ti 16 GB (14.8 GiB free), Python 3.12, engine 0.1.3+gb332d3a97 (win_amd64 wheel from the beta channel).
Model: Qwen3.6-35B-A3B-FP8 — FP8 block quantization with weight_block_size: [128, 128] and bf16 scales.
Command:
ft serve --model F:\models\Qwen3.6-35B-A3B-FP8 --port 1919 --moe-backend auto --max-running-requests 4 --memory-ratio 0.85
Root cause analysis: Since free VRAM (14.8 GiB) cannot hold the model, the engine auto-selects moe_strategy='offload', which sets the decode target to CPU. However, for FP8-block-quantized MoE layers,
freetoken/layers/quantization/moe/fp8_block.py
declares only one candidate kernel, TritonFp8BlockMoEKernel, which has cpu_format = None and rejects CPU decode (cpu_ok=False: "has no CPU executor format; decode must run on the GPU"). The alternative dsv4 kernel is skipped because it only serves e8m0 block scales (this checkpoint uses bf16 scales). With zero usable kernels, the new kernel-selection check introduced in 0.1.3 raises KernelSelectionError.
Notably, fp8_block appears to be the only quantization format without a CPU executor format — nvfp4 (cpu_format="nvfp4"), mxfp4 (cpu_format="ds_fp4"), and unquantized (cpu_format="bf16") all have one.
Regression: The same model and command worked on engine 0.1.2 (which has no kernel-selection validation). Rolling back to 0.1.2 restores service.
Request: Please add a CPU executor format for fp8_block, or allow offload-strategy decode of fp8_block experts to run on the GPU.
Desktop app version
0.2.0-beta.21
OS
Windows 11
OS details
No response
GPU and driver
RTX 5060Ti 16GB
CPU and system RAM
i7-12700 64GB
Checkpoint
Qwen3.6-35B-A3B-FP8
Model settings
No response
Engine log
Anything else
No response