tarfile stream mode raises EOFError reading past the end of bz2/xz/zstd data #159643

Bug report

Since the _Stream._read rewrite for gh-121109, tarfile in stream mode with bz2, xz or zstd (r|bz2, r|xz, r|zst, and r|* when it detects one of them) raises a bare EOFError from the decompressor whenever it reads past the end of the compressed stream. That happens for an ordinary archive opened with ignore_zeros=True, for concatenated archives (the documented use of ignore_zeros), and for an archive that has no end-of-archive zero blocks. The r: modes and gzip in stream mode return the members.

import bz2
import gzip
import io
import lzma
import tarfile
from compression import zstd

COMPRESSORS = {"gz": gzip.compress, "bz2": bz2.compress,
               "xz": lzma.compress, "zst": zstd.compress}


def tar_bytes(name, data=b"hello"):
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w") as tf:
        info = tarfile.TarInfo(name)
        info.size = len(data)
        tf.addfile(info, io.BytesIO(data))
    return buf.getvalue()


def names(mode, raw, **kwargs):
    try:
        with tarfile.open(fileobj=io.BytesIO(raw), mode=mode, **kwargs) as tf:
            return [m.name for m in tf]
    except Exception as e:
        return f"{type(e).__name__}: {e}"


one = tar_bytes("a")
cases = [
    ("single archive, ignore_zeros=True", one, {"ignore_zeros": True}, ["a"]),
    ("two concatenated archives, ignore_zeros=True",
     one + tar_bytes("b"), {"ignore_zeros": True}, ["a", "b"]),
    ("archive without end-of-archive blocks", one[:1024], {}, ["a"]),
]
for label, tar, kwargs, expected in cases:
    print(f"{label} (expected {expected}):")
    for comp, compress in COMPRESSORS.items():
        raw = compress(tar)
        for mode in (f"r:{comp}", f"r|{comp}", "r|*"):
            print(f"  {mode:6} {names(mode, raw, **kwargs)}")

Output on main (3.16.0a1+dev, 9cb7a5b), first case shown, the other two fail the same way:

single archive, ignore_zeros=True (expected ['a']):
  r:gz   ['a']
  r|gz   ['a']
  r|*    ['a']
  r:bz2  ['a']
  r|bz2  EOFError: End of stream already reached
  r|*    EOFError: End of stream already reached
  r:xz   ['a']
  r|xz   EOFError: Already at end of stream
  r|*    EOFError: Already at end of stream
  r:zst  ['a']
  r|zst  EOFError: Already at the end of a Zstandard frame.
  r|*    EOFError: Already at the end of a Zstandard frame.

Expected: every mode returns ['a'], ['a', 'b'] and ['a'] for the three cases. That is what 3.14.4 prints for all 36 combinations with the same script.

The cause is the non-gzip branch of _Stream._read (Lib/tarfile.py lines 593-606 at 9cb7a5b). Once BZ2Decompressor, LZMADecompressor or ZstdDecompressor reaches eof, it reports needs_input == False, so the loop takes the else path, sets cbuf = b"" and calls self.cmp.decompress(b"", ...), which raises EOFError on an exhausted decompressor. EOFError is not self.exception, so it is not converted to ReadError and escapes from iteration. The code before the rewrite read fileobj first and stopped on empty input, so it never fed an exhausted decompressor. zlib keeps returning empty bytes after the end, which is why gzip is unaffected. The change was backported to 3.15 in GH-150604, and v3.15.0 has the same _read, so the regression is in a released version. The existing test_ignore_zeros only opens with mode="r", and no stream-mode test reads past the end of the compressed data.

Fix: in the non-gzip branch, break out of the loop when self.cmp.eof is true, before checking needs_input. I tried that two-line change locally: all 36 combinations above return the expected members and test_tarfile still passes. Stream-mode tests for bz2, xz and zstd with ignore_zeros=True on concatenated archives and on an archive without end-of-archive blocks would cover it.

This is a different path from gh-156057, which is about corrupt member data in the r: modes.

I'm happy to send a PR with the fix and the tests.

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-159643: Fix EOFError in tarfile stream mode with bz2, xz and zstd #159647