So Matt Pocock says models typically are smartest in the first 140k tokens. Hes using the big boy cloud models tho. Does that hold true for 27b-35b range? Or is it smaller and how much?
What would the ‘smart range’ be for tiny models that fit in 2gb? Would be it be like a sentence or two? I have an old 1050 was thinking of throwing in rig but im not sure what id even be able to do with such a small model
I’ve experimented with a lot of that range of models. The sweet spot is 35-50k of context. Anything more and the response quality degrades, or the model times out trying to answer, or just keeps churning till it becomes nonsense. I had a model respond with “SimiSimiSimi” for 3 pages of content because I overloaded the context to about 130k tokens. Was really disappointing to see.