LLM inference is often described as compute-bound. It is at least as often a data-movement problem: gigabytes of weights, a KV cache that grows with context, and intermediate tensors crossing PCIe, NVLink, memory interconnects, and the network. The arithmetic is fast; feeding it is not.
The obvious fix is quantization — store the tensors in a smaller format. It works, and it's everywhere, but it trades accuracy for bytes, and the trade isn't uniform. Accuracy can degrade, needs tuning per model and workload, and sometimes the side effects are strange rather than measurable: models that look fine on benchmarks and ramble more in production.
Lossless compression takes the other branch. It doesn't degrade anything; it just removes redundancy. A survey by Aditya Kamath and Simon Peter in ACM SIGARCH lays out the state of the art, and the numbers are better than "lossless" usually implies: around 1.4x average data reduction, with roughly 30% smaller model sizes in the systems they describe.
Where the redundancy lives
The interesting part is why it works at all.
BF16, the usual format for model weights, is one sign bit, eight exponent bits, and seven mantissa bits. Mantissa bits look close to random. Exponent bits do not: although eight are available, usually only about three carry useful information. The rest are low-entropy, which means they compress.
That's a property of the number format, not of any particular model, which is why the techniques generalize across architectures. It also explains why newer FP8 models are compressible in the same way — the exponent redundancy is structural.
Three ways to spend the savings
DFloat11 targets the case where weights don't fit in GPU memory. Tensors live in CPU memory and stream over PCIe as needed. It Huffman-codes the exponents offline, replaces the eight bits with shorter lookup keys, and keeps the lookup tables in GPU shared memory so decompression is cheap. The result: about 30% smaller models, a 48 GB model fitting on a 40 GB GPU, and 2–40x faster inference than HuggingFace Transformers — mostly because the data no longer has to cross PCIe.
ZipServ looks inside the GPU instead. Getting weights from GPU memory into tensor cores is itself a data-movement problem. It maps the eight exponent bits to three bits for the seven most frequent exponents, sends the rest to a side table, and fuses decompression into the matrix-multiply kernel so data is decompressed directly into the tensor cores. That's about 22% more throughput than vLLM.
Invariant Bit Packing, the authors' own work, generalizes: it discovers bits that are invariant across tensors, stores them as shared-memory metadata, and strips them at compression time. It supports runtime compression as well as decompression, which extends it to the KV cache, and reports about 24% faster inference than FlexGen and InfiniGen.
The real question is where to decompress
Every scheme here works, and every one moves the bottleneck rather than removing it. If decompression happens on the GPU, it steals cycles from compute. If it happens early — on a SmartNIC or a network switch — the data expands before it crosses PCIe or NVLink, and downstream links lose the benefit. NVIDIA's B200 includes a decompression engine that can expand data on the transfer path without using GPU multiprocessors, but it currently supports a limited set of algorithms (LZ4, Snappy, Deflate).
Processing-in-memory offers a third option: store compressed in memory, decompress as data leaves memory. That increases effective capacity without reserving cores, which is attractive because the capacity constraint is what forces offloading in the first place.
So the engineering question isn't "does compression help" but "which link in the chain pays for the expansion." That's a co-design problem, and it's why the same compression ratio can be a win in one deployment and a wash in another.
The hard case is training
All of this assumes static data. Inference reuses weights and KV entries across iterations, so the compression cost amortizes. Training breaks that assumption: the working set changes every step, so you'd be recompressing constantly. The interesting research direction is compression that supports updates in place — doing gradient descent on compressed data without a decompress-recompress loop.
That's a much harder problem, and it's the one that would matter most at scale, because training is where the data movement is worst.
What I take from it
Lossless compression is the unglamorous lever. It doesn't have a benchmark number that goes up, and it doesn't let you claim a smaller model in a headline. What it has is a structural argument: a large fraction of the bits in a BF16 tensor are doing no work, and moving them is pure cost.
The catch is that the win depends on where you put the decompressor, and that's a hardware and deployment decision, not an algorithm. The papers that will matter are the ones that co-design the format, the kernel, and the link — not the ones that report the best ratio in isolation.
The exponent bits were always low-entropy. It just took a while for the systems to notice.
Source: Compressing LLMs without Compromise — Aditya Kamath and Simon Peter, ACM SIGARCH.