Your 10GB Zip Is Now 3GB. Where Did the 7GB Go?

TL;DR — The 7GB didn't go anywhere, because it was never really information — it was repetition and predictability. Zip replaces repeated chunks with short "go back and copy" pointers, then gives the most common symbols the shortest codes. What's left is the file's actual information content. Random or already-compressed data won't shrink, and that's the whole story.

The magic trick, in one command

You zip a 10GB folder and get this:

$ du -sh project/
10G     project/

$ zip -r project.zip project/
$ du -sh project.zip
3.0G    project.zip

Seven gigabytes vanished, and the folder unzips back to a byte-for-byte identical 10GB. Nothing was lost. So what happened to those 7GB?

The honest answer: they were never 7GB of information. They were 7GB of stuff the file was repeating or that a computer could easily predict. Compression is just the art of not writing the same thing twice.

Files are boring, and boring is compressible

Here's the key intuition. Real files are nothing like random noise. They're full of patterns:

A log file says INFO and 2026-07-20 ten thousand times.
A CSV repeats the same column layout on every row.
An English text uses the, and, (space) constantly, and almost never uses qzx.
A bitmap of a blue sky is the same pixel, over and over and over.

Every one of those repetitions is redundancy — bytes you can reconstruct without storing them in full. Compression finds that redundancy and writes down the recipe instead of the result.

Two engines do the heavy lifting. Zip uses both, stacked.

Engine 1: don't repeat yourself (LZ77)

Imagine this silly string:

the cat sat on the mat, the cat sat on the rug

The phrase the cat sat on the shows up twice. The second time, instead of spelling it out again, we can say: "copy 19 characters starting from 27 positions back."

That's a back-reference — a (distance, length) pair:

the cat sat on the mat, <copy 19 chars from 27 back>rug

We replaced 19 bytes with a tiny pointer. This is the LZ77 algorithm (1977, still everywhere). As it reads the file, it keeps a sliding window of the recent past — for classic zip, the last 32KB — and constantly asks: "have I seen this run of bytes recently?" If yes, emit a pointer instead of the bytes.

literal bytes   → written as-is
repeated run    → (distance, length) pointer

Now picture that 10GB folder: duplicate files, boilerplate code, repeated headers, the same JSON keys on millions of records. LZ77 turns oceans of repetition into cheap pointers. This is where most of your 7GB goes.

Engine 2: short codes for common things (Huffman)

After LZ77 there's still a stream of symbols left — some common, some rare. Engine 2 attacks a different waste.

Normally every byte costs a fixed 8 bits, whether it's a super-common space character or a byte you'll only ever see once. That's dumb. Why spend 8 bits on something that appears half the time?

Huffman coding fixes this: frequent symbols get short bit-codes, rare symbols get long ones.

Fixed-width (8 bits each):
  ' ' → 00100000    'e' → 01100101    'q' → 01110001

Huffman (frequency-weighted):
  ' ' → 01          (2 bits — appears constantly)
  'e' → 001         (3 bits)
  'q' → 11010110    (8+ bits — barely shows up)

Because the common symbols dominate the file, the average bits-per-symbol drops well below 8. It's Morse code logic: E is a single dot, Q is dash-dash-dot-dash, because you type E far more often.

Zip = LZ77 + Huffman, stacked

The classic zip format uses an algorithm called DEFLATE, which is literally these two engines in sequence:

LZ77 — crush repetition into (distance, length) pointers.
Huffman — encode whatever's left (literals + pointers) with frequency-optimal bit codes.

raw bytes ──▶ [ LZ77: kill repeats ] ──▶ [ Huffman: short codes ] ──▶ .zip

That's the entire "magic." No data is thrown away — every step is perfectly reversible, which is why unzipping gives you the exact original back. This is lossless compression.

So where did the 7GB actually go?

It got described instead of stored.

✅ A 4KB block that appears 2,000 times → stored once, referenced 1,999 times.
✅ A byte that appears half the time → charged 1–2 bits instead of 8.
✅ Predictable structure (headers, whitespace, repeated keys) → folded into the recipe.

The 3GB that remains is roughly the file's real information content — the genuinely unpredictable part that can't be derived from anything else. Everything above that was redundancy, and redundancy is free to remove because you can always regenerate it.

Think of it like a recipe. "Add salt" repeated 500 times isn't 500 facts — it's one instruction and a count. You didn't lose 499 facts; you never had them.

The catch: some files refuse to shrink

If compression removes redundancy, then a file with no redundancy can't be compressed. This is why some things barely budge:

⚠️ JPEG, PNG, MP3, MP4, H.264 — already compressed. Zipping them again gains almost nothing (sometimes it gets bigger by a hair, thanks to zip's overhead).
⚠️ Encrypted / random data — by design it looks like noise, so there's no pattern to exploit. Truly random data is incompressible.
✅ Text, logs, CSVs, source code, uncompressed images (BMP), databases — hugely redundant, so they crush beautifully (often 5–10×).

There's a hard mathematical floor here, from Claude Shannon: a file's entropy is the minimum number of bits needed to represent it. No lossless compressor — zip, gzip, 7-Zip, anything — can ever go below it. Your 10GB file compressing to 3GB means its true entropy was about 3GB all along. The rest was noise in the accounting.

And a quick sanity check on why "just zip it again" doesn't keep shrinking:

$ zip -r once.zip project/      # 10G → 3.0G   ✅ removed the redundancy
$ zip -r twice.zip once.zip     # 3.0G → 3.0G   ❌ nothing left to remove

Once the redundancy is gone, it's gone. A compressed file is supposed to look random — that's the sign it worked.

Lossless vs lossy — a one-line aside

Zip is lossless: perfect reconstruction. That's mandatory for code, documents, and archives — flip one bit and a program breaks.

Formats like JPEG and MP3 are lossy: they throw away detail your eyes and ears won't miss (subtle colour shifts, inaudible frequencies) to hit far smaller sizes. That's a genuinely different trick — actual information is discarded — and it's why you never store your source code as a JPEG.

The takeaway

The 7GB didn't go anywhere, because it was never really there as information:

Compression describes redundancy instead of storing it.
Zip does this with LZ77 (kill repeats) + Huffman (short codes for common symbols).
It's lossless — the original comes back exactly.
What remains (~3GB) is the file's true information content; you literally cannot go below its entropy.
Already-compressed or random data has no redundancy left, so it won't shrink.

Compression isn't destroying data. It's realising your file was padded with predictability all along — and quietly declining to write the same thing twice.

Sources: DEFLATE — RFC 1951 · LZ77 (Wikipedia) · Huffman coding (Wikipedia) · A Mathematical Theory of Communication — Shannon, 1948