gettext re-checks the filesystem on every call when no catalog exists, which slows argparse parser construction

[aktanazat]

Proposal:

gettext.gettext() and ngettext() go through dgettext()/dngettext(), which call translation(), which calls find() every time (Lib/gettext.py:591-596 and :492-525 at main 7731da1fe749eafe0d97f80ab0021d6fd1b47c8d). find() has no cache. With LANG=en_US.UTF-8 each call runs os.path.exists on four candidate .mo paths, finds nothing, and translation() raises FileNotFoundError, which dgettext() catches. When there is no catalog the answer is the same every time, but it is worked out again on every call.

argparse makes this visible. Every ArgumentParser calls _() three times while it is built (Lib/argparse.py:2049, :2050 and :2063), so a CLI with 20 subcommands makes 63 find() calls and 252 os.path.exists calls just to build its parser.

import argparse
import gettext
import locale  # imported up front so its one-time import is not timed
import os
import statistics
import sys
import time


def build():
    parser = argparse.ArgumentParser(prog="tool")
    sub = parser.add_subparsers(dest="cmd")
    for i in range(20):
        cmd = sub.add_parser(f"cmd{i}", help=f"command {i}")
        cmd.add_argument("--flag", action="store_true", help="a flag")
        cmd.add_argument("path", help="a path")
    return parser


mode = sys.argv[1]
if mode == "identity":
    argparse._ = lambda message: message
    argparse.ngettext = lambda singular, plural, n: singular if n == 1 else plural

if mode == "count":
    counts = {"find": 0, "exists": 0}
    real_find, real_exists = gettext.find, os.path.exists

    def find(*args, **kwargs):
        counts["find"] += 1
        return real_find(*args, **kwargs)

    def exists(path):
        counts["exists"] += 1
        return real_exists(path)

    gettext.find, os.path.exists = find, exists
    build()
    print(f"one build: {counts['find']} gettext.find calls, {counts['exists']} os.path.exists calls")
    sys.exit()

start = time.perf_counter()
build()
first = (time.perf_counter() - start) * 1000

samples = []
for _ in range(200):
    start = time.perf_counter()
    build()
    samples.append((time.perf_counter() - start) * 1000)
print(f"{mode}: first build {first:.2f} ms, then median {statistics.median(samples):.2f} ms per build")

LANG=en_US.UTF-8 python bench.py real, the same with identity (argparse's _ and ngettext swapped for functions that return the message), and LC_ALL=C python bench.py real. macOS on Apple Silicon, five runs each (three for LC_ALL=C), range across runs:

	real gettext	identity	LC_ALL=C
3.15.0rc1, first build	2.74-2.90 ms	1.96-2.14 ms	2.06-2.43 ms
3.15.0rc1, median per build after that	0.98-1.01 ms	0.34-0.35 ms	0.44-0.45 ms
3.14.7, first build	11.09-12.71 ms	10.23-10.74 ms	10.33-11.37 ms
3.14.7, median per build after that	1.28-1.33 ms	0.42-0.43 ms	0.54-0.55 ms

find() and _expand_lang() on main are the same as in 3.15.0rc1 and 3.14.7.

Repeated lookups for a domain with no catalog shouldn't touch the filesystem every time. A cache has to stay correct if LANGUAGE, LC_ALL, LC_MESSAGES or LANG change while the process runs, and it would miss a .mo file installed after the first lookup. Would a cache keyed on the domain, localedir and resolved language list be acceptable, or would you rather keep find() uncached and fix this in argparse? I can write the PR either way.

Claude (via omp) found this and wrote the benchmark.

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-158298: Cache the .mo file search in gettext.gettext() and related functions #158376

[anbv29]

Hi maintainers — I’d like to contribute to “gettext re-checks the filesystem on every call when no catalog exists, which slows argparse parser construction”. I have Python experience and can take a focused approach: reproduce the report, trace the affected module, add a regression test for the stated behavior plus a nearby edge case, then implement the smallest compatible fix while keeping the public API and existing behavior stable. I’ll run the repository’s relevant test/lint/type-check commands and keep the PR limited to this issue. Would you be comfortable confirming this direction before I open a PR?

[aisk]

@anbv29 Hi, the reporter already said that he want to work on this, so please pick another one. Also, we usually don't assign issues to people.

[anbv29]

Okay, I'll see if I can work on some other issue.

[aktanazat]

went with a cache in gettext instead of argparse, since glibc already works this way: it remembers per domain, directory and locale whether a catalog exists and doesn't look again. it only covers gettext(), dgettext() and the other module-level functions, translation() and find() still search every time. the key is the domain, the absolute locale directory and the language list from LANGUAGE, LC_ALL, LC_MESSAGES and LANG, so changing any of those or calling bindtextdomain() with a new directory triggers a fresh search. the visible difference is that a .mo file added after the first lookup for the same key isn't seen until the process restarts or one of those settings changes, same as glibc.