Skip to content

gh-107862: Fix infinite loop in email header folding found by a Hypothesis round-trip test - #159771

Draft
pganssle wants to merge 5 commits into
python:mainfrom
pganssle:email-fold-hypothesis
Draft

pganssle wants to merge 5 commits into
python:mainfrom
pganssle:email-fold-hypothesis

Conversation

@pganssle

@pganssle pganssle commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Stacked on #159770; only the top two commits are new here.

When an encoded word does not fit in the remaining space of a line, _fold_as_ew shrinks the chunk being encoded one character at a time until it fits. Nothing stops it from shrinking the chunk to nothing, and an empty chunk makes no progress: the loop appends an empty encoded word and a new continuation line forever.

>>> import email.policy
>>> email.policy.default.clone(max_line_length=20).fold('Subject', '😀')
# never returns

This is reachable whenever a single character's encoded form is wider than a fresh continuation line, e.g. any character outside the BMP at max_line_length=20 (the q-encoded word is 24 characters). The two earlier fixes in this area (bpo-33529, bpo-36564) handled lines too short for the RFC 2047 chrome, but not lines too short for the chrome plus one character.

The fix keeps at least one character in the chunk, so the loop always advances and the line runs over max_line_length instead. That matches what _refold_parse_tree already does when it cannot fold a part at all, and avoids raising for widths that work for every other input.

Tests, next to the two earlier infinite-loop regressions in test_policy.py:

  • An exact-output regression case, as the earlier two have.
  • A Hypothesis property: folding any generated unstructured value at any width in 1..120 (plus 0 and None) either raises HeaderParseError (only possible at widths of 13 or fewer, where no encoded word fits) or reparses to the same value modulo leading whitespace, which the parser drops. 100 examples take about 0.2 s; the explicit examples replay under the stubs.
  • A separate expectedFailure test for a second finding: a header value that is itself a well-formed encoded word is written verbatim, so reparsing decodes =?utf-8?q?x?= to x. As far as I can tell RFC 2047 §5 requires such text to be encoded. Recorded rather than worked around, so it starts failing when fixed; a separate issue for it is coming.

@bedevere-app bedevere-app Bot added the type-feature A feature request or enhancement label Oct 19, 2026
Comment thread Lib/test/support/_hypothesis_stubs/stateful.py
Comment thread Lib/test/support/_hypothesis_stubs/stateful.py Outdated
Comment thread Lib/test/support/hypothesis_helper.py Outdated
Comment thread Lib/test/test_hypothesis_stubs.py Outdated
…thesis stubs

The stubs in test.support._hypothesis_stubs only cover the strategies
and decorators the existing property tests happen to use, so a test
that wants hypothesis.stateful, or the timezones() and ip_addresses()
strategies, fails to import when Hypothesis is not installed.

This adds those names. The stateful API has no equivalent of @example,
so there is nothing for the stub to replay: a state machine's TestCase
is skipped and run_state_machine_as_test raises SkipTest. The
decorators are accepted so that state machine classes can be defined
at import time.

Real Hypothesis only exposes hypothesis.stateful once the submodule has
been imported, so hypothesis_helper now imports it, making the
attribute available in both modes.
@pganssle
pganssle force-pushed the email-fold-hypothesis branch from ad9482f to aeb677f Compare October 19, 2026 21:30
…he line

When an encoded word does not fit in the remaining space of a line,
_fold_as_ew shrinks the chunk being encoded one character at a time
until it fits. Nothing stops it from shrinking the chunk to nothing,
and an empty chunk makes no progress: the loop appends an empty
encoded word and a new continuation line forever.

This is reachable whenever a single character's encoded form is wider
than a fresh continuation line, e.g. any character outside the BMP
with max_line_length=20 (the q-encoded word is 24 characters). The two
earlier fixes in this area (bpo-33529, bpo-36564) handled lines too
short for the RFC 2047 chrome, but not lines too short for the chrome
plus one character.

This keeps at least one character in the chunk, so the loop always
advances and the line runs over max_line_length instead. That matches
what _refold_parse_tree already does when it cannot fold a part at
all, and avoids raising for widths that work for every other input.

The regression test pins the exact output, as the earlier two did. The
Hypothesis property checks that folding any unstructured value at any
width either raises HeaderParseError (only possible at widths of 13 or
fewer, where no encoded word fits) or reparses to the same value modulo
leading whitespace, which the parser drops.
…oded words

A header value that happens to be a well-formed RFC 2047 encoded word
is written out verbatim, so reparsing the message decodes it:
folding 'Subject' '=?utf-8?q?x?=' comes back as 'x'. As far as I can
tell RFC 2047 section 5 requires an encoder to encode such text.

This records the behavior as an expected failure rather than keeping
the round trip property away from it, so the test starts failing when
the generator is fixed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type-feature A feature request or enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant