gh-155695: Remove resolved names from sys.lazy_modules consistently - #157714
brittanyrey wants to merge 5 commits into
Conversation
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase |
|
@pablogsal Addressed comments. |
|
cc @encukou |
|
Thanks! The initializing-module case is covered now. I left one follow-up comment about the new |
Names only left sys.lazy_modules through _imp._set_lazy_attributes(), which the import machinery calls from _find_and_load_unlocked(). Two cases never reached it, so their names were recorded and then kept forever: - A lazy import of a module already in sys.modules. _find_and_load() returns early, so nothing ever discards the name. Do not record it in the first place. - The "pkg.attr" entry for `lazy from pkg import attr`. The import machinery only discards module names, and attr is often not a module. Discard it when the lazy object is reified, where the name is already known and has been resolved either way. Submodules that are not yet loaded are still tracked: loading a package does not load its submodules, so those imports can still fire. Names whose reification failed also stay tracked, since the import can still happen.
A module is in sys.modules while its body runs, so lazy_modules_add() counted it as loaded and skipped recording the name. If the body then raises, the module is removed from sys.modules again and the lazy import is left pending under no name at all. Check __spec__._initializing so such a module does not count as loaded. A name added while a module initializes is still discarded once the import completes, since _set_lazy_attributes() runs after the body.
pablogsal
added
the
needs backport to 3.15
374d85f to
44ebbaa
Compare
|
I found a couple of issues and pushed fixes. Checking the spec could call an Importing an already loaded attribute could also leave a stale entry in |
Read stored initialization flags without invoking spec callbacks or materializing dictionaries. Keep unsupported spec state conservatively tracked. Skip cached exports without changing their import context or clearing independently pending modules.
83ac500 to
ad36b56
Compare
|
I spent a bunch of time reading the code, trying to understand the internal invariants. @pablogsal, your edit suggests that The Unsupported spec representations stay conservatively tracked part worries me a bit. This PR should be titled “Remove resolved names from sys.lazy_modules more consistently”. We can't really remove the caveats from the docs; in fact the “consumers [of The magic needed in |
The point of the direct lookup was to keep the tracking change from adding calls to user code. Reading a spec normally can invoke a property or There’s a cleaner way to handle this, though. I’d suggest building on #158282 and removing the cached
Agreed, that caveat should stay. I’d make the bookkeeping follow the operations we actually perform: add names when declaring the lazy import and remove them when resolution succeeds. For a from-import, successful module import and successful attribute lookup are separate steps. If the module imports but the attribute lookup fails, we can remove the module entry while keeping the attribute entry. This still isn’t a list of every unresolved binding. Multiple declarations share one name, and resolving one can remove that name while other placeholders remain. A previously loaded module can also be listed after a new, unused lazy declaration. The docs should explain those cases.
The reason for checking initialization was that a module can be in With cleanup at resolution, we don’t need to predict that outcome when recording the declaration. #158282 already brings the resolution code together, so I’d use that structure and drop the declaration-time filtering. Moving initialization state onto the module may still be useful elsewhere, but we wouldn’t need that change to maintain this set. |
|
So, do we leave this as is for 3.15, and go for a refactor in 3.16? |
|
No, the refactor should land in 3.15 otherwise it's going to be hell to keep both versions and backport any fixes. |
|
@hugovk I think this should mark as a release blocked and block 3.15 on landing the other commit as we are still on time to avoid quite a painful set of backports |
hugovk
added
the
🔨 test-with-buildbots
|
🤖 New build scheduled with the buildbot fleet by @hugovk for commit ad36b56 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F157714%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
bedevere-bot
removed
the
🔨 test-with-buildbots
5 participants
summary
Improve
sys.lazy_modulesto address the following issues:sys.lazy_modules.lazy from pkg import attris cleaned up after reification.perf
End-to-end (hyperfine,
--warmup 50, 1000 runs; 400 for reify-only)-X lazy_imports=allapp