

FWIW I wouldnt recommend a 64GB strix halo for local LLM inference. I use a 128GB framework desktop for running qwen-flash-next which it just manages at Q4_K at a reasonable rake of 20-30 tk/s qwen27B is sloooow on it due to strix halo’s lower memory throughput than either a GPU or a integrated memory mac.
Though its possible this advice will change in the next few weeks with the release of Qwen4.



I think a sensible place for this would be the base models, those are the ones that are trained on the entire internet in order to get a good working model that “understands” things. Then leave the fine tunes on those base model commons as available for companies and organisations to bring their own value to.