gh-152754: Fix crash when an os.scandir iterator is shared between threads

[nascheme]

The issue was a race between ScandirIterator_iternext() and ScandirIterator.close(): one thread could be inside readdir() or FindNextFileW() while another closed the underlying directory stream or Windows handle.

Changes

• Add a PyMutex and atomic closed state to ScandirIterator in all builds.
• Serialize access to the native directory stream or handle.
• Make close() mark the iterator closed immediately without waiting for an in-progress directory read. The reading thread performs the native close afterward when necessary.
• Copy native directory-entry data while holding the mutex, then create Python objects and report errors after releasing it.
• Keep Py_BEGIN_ALLOW_THREADS around readdir(), FindNextFileW(), closedir(), and FindClose().
• Avoid incorrectly emitting ResourceWarning when opening the directory fails.
• Add regression tests for concurrent iteration, iteration racing with close(), and failed iterator creation.
• Document the behavior of sharing a scandir() iterator between threads.

Threading semantics

Sharing an os.scandir() iterator between threads will no longer corrupt the iterator or crash. Concurrent calls may consume entries in any order, so which thread receives each entry remains unspecified. Calling close() while another thread is iterating marks the iterator closed immediately and ends iteration early.

A PyMutex is used rather than a critical section because the protection must remain active across Py_BEGIN_ALLOW_THREADS while the native directory operation runs without the GIL.

• Issue: Segfault between os.scandir iterator's close() and __next__() on the dirp handle #152754

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34158953 | 📁 Comparing 730e655 against main (bc31217)

  🔍 Preview build  

34 files changed · ± 34 modified

± Modified

• c-api/import.html
• extending/extending.html
• library/annotationlib.html
• library/asyncio-task.html
• library/audit_events.html
• library/compression.zstd.html
• library/csv.html
• library/curses.html
• library/curses.panel.html
• library/dataclasses.html
• and 24 more...

[nascheme]

Note, I revised this PR to no longer try to be thread-safe with multiple threads using the same iterator. That seems a pretty unlikely usage scenario. Instead, this avoids crashing and/or memory corruption.

[miss-islington-app]

Thanks @nascheme for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

[miss-islington-app]

Thanks @nascheme for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @nascheme, I could not cleanly backport this to 3.14 due to a conflict.

Please backport manually with cherry_picker, see the devguide for more information.

cherry_picker 999a046b24cff4ba0e72b574196721f66bd08237 3.14

[bedevere-app]

GH-159365 is a backport of this pull request to the 3.15 branch.

[bedevere-app]

GH-159381 is a backport of this pull request to the 3.14 branch.