One use case is auto mode for your local coding agent (automatic permission approval).

There is a lot of room for optimizing the inference for specific models. For example, when I switch llamacpp to Strata, I get to 45ts (granted on a slightly poorer quant).

You don’t need to load the full model into VRAM. In my case it is offloaded into RAM. Some also get away with offloading to an NVME SSD.

Yup worth a try. I got from 13ts to 21ts for Qwen 3.8 Flash Next on a RX 9070 XT 16GB. As always, the additional memory demand requires some compromise. For me reducing context from full 262k to 200k and reducing kv cache down to q8_0 was the best option. moe-cache-mib at 6544. Cache hit rate at ~70%.