gh-158445: Allocate memory in the heap in Py_GetVersion()

[vstinner]

Py_GetVersion() now allocates memory on the heap, instead of using a static buffer, to no longer truncate the version if it's longer than 299 bytes.

Update Py_GetCompiler() and Py_GetBuildInfo() tests: they are now always a part of sys.version.

• Issue: C API: Detect invalid UCS4 characters in debug mode #158445

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34907310 | 📁 Comparing ef0a101 against main (be87a85)

  🔍 Preview build  

2 files changed

± c-api/interp-lifecycle.html
± whatsnew/changelog.html

[vstinner]

On my Fedora 44, __clang_version__ is short: 22.1.8 (Fedora 22.1.8-4.fc44). But in issue gh-157368, way longer version has been seen: 22.1.3 (https://github.com/llvm/llvm-project e9846648fd6183ee6d8cbdb4502213fcf902a211). On Windows, PC/pyconfig.h no longer uses __clang_version__. But we may such long clang version on other platforms.

[itamaro]

hey @vstinner, we ran into a side effect of moving the buffer to the heap - some debuggers and core-dump tools find the interpreter version by reading the Py_GetVersion() buffer out of process memory (py-spy for example). With only the 20-byte static_version left, they lose the tag and the "free-threading build" marker. Getting those back means following heap_version into the heap, which is fragile and not always possible.

can we make the static buffer bigger, at least enough to hold the version, tag, and free-threading marker (~64 bytes)? let me know if I should file a separate issue

[vstinner]

py-spy:

    if let Some(&addr) = python_info.get_symbol("Py_GetVersion.version") {
        info!("Getting version from symbol address");
        if let Ok(bytes) = process.copy(addr as usize, 128) {
            if let Ok(version) = Version::scan_bytes(&bytes) {
                return Ok(version);
            }
        }
    }

I understand that in Python 3.15, py-spy copies 128 bytes from the static char version[300]; variable.

Would it be possible for py-spy to (1) call Py_GetVersion() function and (2) copy the return string? I suppose that it's better to not call functions in a debugger to leave the process state unchanged.

some debuggers and core-dump tools find the interpreter version by reading the Py_GetVersion() buffer out of process memory (py-spy for example)

I didn't think about these use cases. That's not how I expected the Python C API to be used. Well, they just inspect memory, they don't even call functions, if I understand correctly.

@itamaro:

can we make the static buffer bigger, at least enough to hold the version, tag, and free-threading marker (~64 bytes)? let me know if I should file a separate issue

Would you recommend to revert my change and add a comment explaining the debugger/crash reporter use cases?

Or just restore the static buffer to its Python 3.15 size (300 bytes)? I suppose that the static buffer name should be restored to version, so debuggers/crash reporters don't have to update their code.

Thanks for rising the issue!

[vstinner]

Or there is another approach, build strings at build time, to avoid the complex code building strings at runtime: #158951.

[itamaro]

I understand that in Python 3.15, py-spy copies 128 bytes from the static char version[300]; variable.

Would it be possible for py-spy to (1) call Py_GetVersion() function and (2) copy the return string? I suppose that it's better to not call functions in a debugger to leave the process state unchanged.

yeah, I think not calling function in the debugged process is the point. for core-dump tools, there isn't even a process around.

I didn't think about these use cases. That's not how I expected the Python C API to be used. Well, they just inspect memory, they don't even call functions, if I understand correctly.

sounds right. like you say, they're inspecting memories, not "using" the C API per se.

Would you recommend to revert my change and add a comment explaining the debugger/crash reporter use cases?

Or just restore the static buffer to its Python 3.15 size (300 bytes)? I suppose that the static buffer name should be restored to version, so debuggers/crash reporters don't have to update their code.

I don't think we need to revert, just restoring the version static buffer should be sufficient!

thanks for looking into this! I didn't look closely at your gh-158951, it looks like a bigger change than just restoring the static buffer, so if you think that's the better way to go, I may be able to test it in the next few days, or maybe next week.

[vstinner]

thanks for looking into this! I didn't look closely at your #158951, it looks like a bigger change than just restoring the static buffer, so if you think that's the better way to go, I may be able to test it in the next few days, or maybe next week.

I created https://discuss.python.org/t/generate-modules-getbuildinfo-h-when-building-python-to-get-build-information-statically/109401 to discuss my change. The backup plan is to revert my change and just make the static version buffer larger (350 bytes?) to support very long Clang version strings.

[itamaro]

thanks for making gh-158951 @vstinner ! I looked at it and did some testing, and here are my thoughts:

1. I don't have a strong opinion about generating the strings at build time, so I'll leave that to the issue/PR where there's already ongoing discussion.
2. if possible, I’d like to decouple fixing the regression from the rework. reinstating the version static buffer should be a quick, safe PR that doesn't need to wait on the bigger design discussion. happy to put a PR if it helps.
3. as implemented in gh-158948: Generate Modules/getbuildinfo.h #158951, the issue I raised isn't fixed. version is now a pointer rather than a buffer, and with gcc and clang at -O2 the variable is optimized out entirely, leaving only an unnamed string in .rodata.

to be more concrete about the use case: what broke for us is an lldb-based Python debugging extension that we use on live processes and on core dumps. on a core you can't call Py_GetVersion(), so it reads the string from memory. it can't rely on the symbol name either (with LTO the static becomes version.llvm.<hash>), so it scans .bss for the version pattern. a string in .rodata doesn't work for that, since it isn't in .bss, and default Linux core dumps don't include unmodified file-backed mappings like .rodata.

I tested this against our debugger test suite. with gh-158951 applied, it fails to detect the Python runtime. a named static const char version[] = VERSION; doesn't help either, since it's still in .rodata. a static char version[300] in .bss, filled at startup, passes everything.

what we actually use is the start of the string: version, local version suffix (3.16.0a0+meta), and the free-threading marker. a 20-byte fallback would cut off before the free-threading marker.

_Py_DebugOffsets covers most of this. I checked that _PyRuntime is in default core dumps with the cookie, version and free_threaded readable, and we already take free_threaded from there. what's missing is the suffix, which we use to pick build-specific struct offsets when debug info isn't available. if _Py_DebugOffsets exposed the PY_VERSION string, tools could move off the buffer for 3.16+, but that's a separate, bigger change, and older versions would still need the buffer.

so my concrete proposal is to keep static char version[~300] in .bss, filled at startup with the full/truncated string, as in 3.15. we only need roughly the first 64 bytes, but 300 matches what existing tools have always seen.