Benjamin Marie

Up to 4x faster than inference with fp16 parameters Benjamin Marie · Follow Published in Towards Data Science · 6 min read · 12 hours ago — Generated with DALL-E Large language models (LLMs) are often too large to be directly used on consumer hardware. To reduce their size, various techniques have been proposed to quantize LLMs and lower their memory consumption. While recent algorithms for 4-bit quantization are often released along with their own