[FEATURE REQUEST] - Turbo Quant

[SabinStargem]

It is looking like a new method for handling KV Cache has arrived, one that improves over Q8 KV, both in speed and reducing the memory footprint. IMO, it seems likely to become a feature in LlamaCPP, as I have seen several people use Turbo Quant.

The thread below details the subject.

TurboQuant - Extreme KV Cache Quantization

I figured that I should draw attention to this, as it seems very promising.

[JohnLoveJoy]

It will likely be implemented in GGML, and eventually the optimizations will end up in Kobold, not the other way around.

[SabinStargem]

Presumably, some work would still be needed on KoboldCPP's end. The GUI, documentation, ect.

[birchamp]

This guy's got instructions for a llama.cpp port with turboquant in the readme. https://github.com/TheTom/turboquant_plus

[henk717]

This is not the kind of thing we typically add on our side, that would risk breaking compatibility with future versions of llamacpp if we did.

TurboQuant is having a lot of eyeballs on it upstream like you said so I am sure that even if they don't add it as TurboQuant in the end the improvements will make its way to us trough the llamacpp ecosystem itself.

One to watch is this one : ggml-org#21038 unlike the vibe coded PR's which won't land this is ggerganov himself implementing a bit of this for every quant type rather than just the new ones. And that kind of change we get automatically when we update llamacpp.