Bug report

Bug description:

Summary

The zlib module docs say that any data in unconsumed_tail "has not been seen by the zlib machinery, so you must feed it back to a subsequent decompress call". This is incorrect when Decompress.eof is true: calling d.decompress(d.unconsumed_tail) after eof will append a copy of the unconsumed tail to unused_data. This could be treated as a documentation bug (the interaction between unused_data, unconsumed_tail, and eof is subtle and the loop structure required to use them all correctly is not obvious), but I believe the right fix is to clear unconsumed_tail when eof is set.

Reproduction

The bug requires

• Input containing the zlib-compressed data followed by something else (which is expected to ultimately appear in Decompress.unused_data). In my case the "something else" is concatenated gzip data (This is valid per RFC 1952 and supported by the python gzip module, but not by zlib's interface which we use for streaming. See also Inconsistency between gzip and zlib: Handling concatenated .gz files #116457)
• decompress(max_length=x) is used to bound memory consumption

A naive read of the docs suggests a loop like the following:

d = zlib.decompressobj()
out = d.decompress(data, 100)
while d.unconsumed_tail:
    out += d.decompress(d.unconsumed_tail, 100)

However, if there is additional data after the zlib data, this loops forever and d.unused_data grows on each iteration:

import zlib

data = zlib.compress(bytes(range(256)) * 4) + b"extra"

d = zlib.decompressobj()
out = d.decompress(data, 100)  # stopped by max_length
while d.unconsumed_tail:
    out += d.decompress(d.unconsumed_tail, 100)
    print(d.eof, d.unconsumed_tail, d.unused_data)
    if len(d.unused_data) > 20:
        break  # otherwise this loops forever

Output (3.11, 3.12 and 3.13; the last lines):

False b'\xe4\xc9\xfe\x10extra' b''
True b'extra' b'extra'
True b'extra' b'extraextra'
True b'extra' b'extraextraextra'
True b'extra' b'extraextraextraextra'
True b'extra' b'extraextraextraextraextra'

The correct logic is to loop on while (not d.eof) and d.unconsumed_tail, but this is not documented. In my case I'm dealing with code that dates back to python 2.7 so d.eof didn't even exist. Even now the fact that .eof was added relatively recently in python 3.3 suggests that it is less essential than the others.

Suggested fix

The docs should at least document the intended loop structure. I argue that the implementation should also be changed so that unconsumed_tail is empty whenever eof is true, so that the naive loops works as well.

Potential cause

I found this issue during an AI-assisted development session; the above summary and analysis is my own. This is Claude Opus 5.5's analysis of the possible cause: In save_unconsumed_input() in Modules/zlibmodule.c, the Z_STREAM_END branch copies the leftover input into unused_data and sets self->zst.avail_in = 0, but doesn't advance self->zst.next_in. The second block runs because the previous unconsumed_tail is non-empty. It is meant to clear unconsumed_tail ("2. All input data was consumed. Clear unconsumed_tail."), but it computes the leftover size from next_in (data->buf + data->len - next_in), not avail_in, so it copies the same bytes into unconsumed_tail. A likely fix is to advance next_in by left_size in the first block, so that the second block sees zero bytes left. (This is untested; I haven't built CPython with it.)

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-158169: Clear Decompress.unconsumed_tail at EOF #158177