gh-159184: Make the marshal order of set elements with equal dumps stable

[gpshead]

This is the second half of gh-159184. The first half, #159196 (merged), stopped marshal from comparing set elements, which fixed the TypeError for complex NaNs. This PR fixes the order such elements are written in.

Set elements with the same dump are NaNs that are distinct objects. marshal leaves them in set iteration order, which is the order of their addresses. That is invisible unless one of them is also referenced elsewhere in the dump; then the reference lands at a different place among them from run to run:

>>> nans = [float('nan') for _ in range(3)]
>>> len({marshal.dumps([n, set(nans)]) for n in nans})
3                             # the same value every time; 1 with this PR

Change

flowchart TB
  subgraph before["main"]
    direction TB
    b1["list of (dump, index, element)"] --> b2["PyList_Sort()"]
    b2 --> b3{"run of equal dumps?"}
    b3 -- no --> b4["ordered by dump"]
    b3 -- yes --> b5["left in set order<br>= address order"]
  end
  subgraph after["this PR"]
    direction TB
    a1["array of {element, dump}"] --> a2["qsort() on the dump bytes"]
    a2 --> a3{"run of equal dumps?"}
    a3 -- no --> a4["ordered by dump"]
    a3 -- yes --> a5["w_same_set_items()"]
    a5 --> a6["already written in this dump:<br>first, in the order written"]
    a5 --> a7["floats and complex not written yet:<br>interchangeable; which one is which<br>is decided when one is referenced again"]
  end

Loading

• For a run of elements with equal dumps, the order is a function of what the writer has already output, never of addresses. This uses the exact reference bookkeeping added by gh-129724: gh-156504: Make marshal output independent of reference counts #159192 (merged).
• The set code moves out of w_complex_object() into w_set_items(). It keeps the checks for a set whose size changes while it is marshalled (gh-159185: Fix marshalling of a set modified while its elements are marshalled #159195, merged).
• No tuples or list are allocated any more, so ordinary sets are marshalled faster. A set made only of distinct NaN objects is 10-24% slower.

Not covered

Elements that are containers with equal dumps, not written before the set (for example {(nan1,), (nan2,)} with one of the NaNs also used elsewhere), are still written in set order. A canonical order for that case amounts to graph canonisation. The compiler cannot produce it: every folded NaN is a new object.

Tests

ReferenceTestCase.test_nan_in_set_order checks exact bytes for sets of float and complex NaNs with references before, after and through other sets. test_nan_in_set also covers complex NaNs.

Authored using Claude, reviewed by me.

• Issue: marshal.dumps() raises TypeError for a set containing several complex NaNs #159184

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #35080091 | 📁 Comparing a4b8ab9 against main (e878e1e)

  🔍 Preview build  

2 files changed

± c-api/marshal.html
± whatsnew/changelog.html