So Matt Pocock says models typically are smartest in the first 140k tokens. Hes using the big boy cloud models tho. Does that hold true for 27b-35b range? Or is it smaller and how much?
What would the ‘smart range’ be for tiny models that fit in 2gb? Would be it be like a sentence or two? I have an old 1050 was thinking of throwing in rig but im not sure what id even be able to do with such a small model
I was asking Kimi-K3 that very question and one of the possible things was a local model for VTT. I think looking at specialised models like that might be the best use case.
Another one I was thinking about last night was local semantic search through embeddings. I’m not sure if that’s actually feasible, but it would certainly be useful for when you have terabytes of PDF.