Hacker News 100 @hn100@social.lansky.name
Public

Show HN: KVSplit – Run 2-3x longer contexts on Apple Silicon

Link: github.com/dipampaul17/KVSplit
Discussion: news.ycombinator.com/item?id=4

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% ...

GitHub - dipampaul17/KVSplit: Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.

Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys &amp; 4-bit values, reducing memory by 59% with &lt;1% ...

github.com · GitHub

0
0
0

If you have a fediverse account, you can quote this note from your own instance. Search https://social.lansky.name/users/hn100/statuses/114519638651128669 on your instance and quote it. (Note that quoting is not supported in Mastodon.)