I felt sorry for it and stopped it, 14min and the poor thing was just going around in circles.
I’m just looking at running some local llm on my m2 16GB mac and this was with the default settings for the qwen3.5 9b in LM Studio. Any pointers to get the best out of a M2?
You must log in or register to comment.
I like “I will assume ‘Arcane’ is a typo for ‘docker’”. I don’t have any suggestions for you, but damn that poor LLM was so confused.
Honestly OP should habe retried the prompt with the correct spelling
Try either orinth 1.5 9b, or if it will fit orinth 1.5 35b a3b. I run 35 a3b on my 7gb rx7600. I use the iq4_xs quant. They are based on qwen3.6 but with some post training. Stuff to make the think and tool call better.
What quant are you running? You might get better mileage from a gemma4 model of similar size and quants.
Try Ternary Bonsai. It’s a heavily quantized version of Qwen 3.8 27b. It won’t do wonders, but it shoud be better: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf-dev The page says you need a fork of llama.cpp, but the support was included in the latest release (0.7). You can read more about the mode on the main model page: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf
Oh god no. Bonsai is such a waste of time! Totally a dull version of Qwen. Keeps getting basic things wrong and isn’t agentic at all.
But I’d be curious to hear if you’ve used it and had a different experience.
i guess my advice is misplaced, given the community name, but in case the goal is to get LLM inference for free, not necessarily a local one, you could try opencode (their free plan gives you access to some decent models), or nvidia’s NIM if you don’t want a vendor lock-in to any specific agent/chat app.
I assume NIM only works with nvidia cards not AMD?
it’s a cloud service. It gives you access to an api. It couldn’t care less about your GPU. Can be used on anything that is capable of internet connection. Specifically NIM has a max requests per minute restriction tho. Due to this, running ai agents on it would be a pain in the ass, but technically possible (still not as painful as local inference tho). And it still works fine for plain chats.
What is arcane? I also don’t know
https://github.com/getarcaneapp/arcane
Just a Docker manager, I’ve moved from Portainer and slowly moving all my containers across.
Why? I don’t care for portainer’s commercial moves but it seems to work fine most of the time. How is arcane better?
Asking seriously. Not sarcastically.
Mostly because of their more and more restrictions on the free version.
Arcane seems to be the manager that is getting all the love at the moment, ie it was recommended a couple of times in the selfhosted community
