I am running two Nvidia RTX A4000, ampere cards with 16gb VRAM for inference. One as PCIe 4.0 16x in CPU lane and one on pcie 4.0 4x on chipset lane.
I have the opportunity to install a PCI bifurcation 8x 8x to move both cards on CPU lane, at 4.0 8x each. Preload times should improve specially because I could use NCCL with P2P.
But when I do this, I could install in the chipset 4x lane a third GPU, an Nvidia Quadro RTX 6000 (touring, pcie 3.0, 24gb VRAM).
The PC is a Ryzen 9 5950x with 64gb ddr4 (4x16gb 2666) and a 850w PSu.
Question is… Would it be a good idea? I mean I would have 56gb VRAM… But with a split on two 8x and one 4x pcie lanes.
Currently I run qwen 3.8 27b Q5_K with 200k context for coding, agent work.
The idea of having those extra 24gb means bigger context and maybe a Q8 version of Qwen…
At the same time, I am afraid preload and inference speed would be even lower due to the touring card in the mix?
(Power wise, I cap the two a4000 at 100w, and would also cap the 6000 probably at 200w, so I don’t care too much about that)
You must log in or register to comment.
The PCI lane is only used to move weights from disk or ram to vram. If your usage is typical, the only time that matter is initial model load. Not the prompt processing at all.
Unless you’re using some newer framework juggling weights between ram and vram.
True… The advantage in vram probably outweighs the rest …
I guess a test is the only answer.
Need to wait for the raiser board to arrive and test …
Talking out of my ass:
The only reason you’d need lanes is more for moving weights around (storage tiering like Strata does)
If you fit everything in vram, the need for high speed bandwidth should be significantly lower
The bigger issue would be the processing on that card may be slower. But you can just prioritize faster cards for smaller models, and only use the third when you need it; even then it shouldn’t be to bad me thinks.
But again, ass.
True, but currently my prefill speed sucks: some 300-400 t/s. I guess going 8x 8x on the cpu lane will improve that, hopefully by quite some… Losing it again in favor of 24gb extra vram?
That’s the question …
Guess need to wait for the raiser board and test actual speed improvement for prefill.
