Skip to content

"python -m pydoc -w" fails in nondecodable directory #69371

Closed
Closed
"python -m pydoc -w" fails in nondecodable directory#69371
Labels
3.7 (EOL)end of life3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@serhiy-storchaka
serhiy-storchaka
opened on Sep 19, 2015
BPO 25184
Nosy @orsenthil, @pitrou, @vstinner, @vadmium, @serhiy-storchaka
Files
  • pydoc_undecodabple_path.patch
  • pydoc_undecodabple_path_2.patch: Using pathlib
  • pydoc_iri.patch: IRI file: link
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2015-09-19.21:07:25.093>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = '"python -m pydoc -w" fails in nondecodable directory'
updated_at = <Date 2018-12-20.12:28:32.594>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2018-12-20.12:28:32.594>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2015-09-19.21:07:25.093>
creator = 'serhiy.storchaka'
dependencies = []
files = ['40534', '40600', '40636']
hgrepos = []
issue_num = 25184
keywords = ['patch']
message_count = 9.0
messages = ['251117', '251212', '251213', '251226', '251271', '251721', '251731', '251985', '332221']
nosy_count = 6.0
nosy_names = ['orsenthil', 'pitrou', 'vstinner', 'Arfrever', 'martin.panter', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25184'
versions = ['Python 3.7', 'Python 3.8']

    Linked PRs

    Activity

    1. serhiy-storchaka commented on Sep 19, 2015

      @serhiy-storchaka
      serhiy-storchaka
      on Sep 19, 2015
      MemberAuthor
      $ pwd
/home/serhiy/py/cpy�thon-3.5
$ ./python -m pydoc -w pydoc
Traceback (most recent call last):
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 2648, in <module>
    cli()
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 2611, in cli
    writedoc(arg)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 1642, in writedoc
    page = html.page(describe(object), html.document(object, name))
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 370, in document
    if inspect.ismodule(object): return self.docmodule(*args)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/pydoc.py", line 651, in docmodule
    url = urllib.parse.quote(path)
  File "/home/serhiy/py/cpy\udcffthon-3.5/Lib/urllib/parse.py", line 706, in quote
    string = string.encode(encoding, errors)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 19: surrogates not allowed
    2. serhiy-storchaka
      added
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Sep 19, 2015

    3. vadmium commented on Sep 21, 2015

      @vadmium
      vadmium
      on Sep 21, 2015
      Member

      Seems to be caused by the Python directory being non-decodable; the current working directory does not matter. What is going on is “pydoc” is trying to make a link to a module’s source code, such as

      <a href="file:/usr/lib/python3.4/pydoc.py">/usr/lib/python3.4/pydoc.py</a>

      For non-decodable paths, the following would work in Firefox:

      <a href="file:/home/serhiy/py/cpy%FFthon-3.5/Lib/pydoc.py">/home/serhiy/py/cpy�thon-3.5/Lib/pydoc.py</a>

      but since URL percent encoding already uses UTF-8, this scheme isn’t foolproof (e.g. a UTF-8 sequence when the locale is ASCII would be ambiguous). A simpler and more consistent way forward would be an error handler substituting something like this, decoding the surrogate escape code with the “replace” handler, with the HTML link suppressed:

      /home/serhiy/py/cpy�thon-3.5/Lib/pydoc.py (invalid filename encoding)

    4. vstinner commented on Sep 21, 2015

      @vstinner
      vstinner
      on Sep 21, 2015
      Member

      Technically, I think that it's possible to put bytes in an URL using %HH format. I didn't check if we can retrieve the "raw bytes".

    5. serhiy-storchaka commented on Sep 21, 2015

      @serhiy-storchaka
      serhiy-storchaka
      on Sep 21, 2015
      MemberAuthor

      We could use url = urllib.parse.quote_from_bytes(os.fsencode(path)) on Posix systems, but I heart that on Windows os.fsencode() can irreversible spoil file names (replace unencodable characters with '?'). On other side, I'm not sure that Windows unicode path can't contain lone surrogates. In this case we should use the 'surrogatepass' error handler (at least it allow to restore the path in principle).

      Here is a patch that tries to handle undecodable and unencodable paths. Need to test it on Windows.

    6. vadmium commented on Sep 21, 2015

      @vadmium
      vadmium
      on Sep 21, 2015
      Member

      Serhiy’s patch essentially uses the local filesystem encoding and then percent encoding, rather than the current behaviour of strict UTF-8 encoding and percent encoding. This is similar to what the “pathlib” make_uri() methods do, so maybe we could let “pathlib” do the work instead.

      This draft RFC discusses encoding “file:” URLs:

      https://tools.ietf.org/html/draft-ietf-appsawg-file-scheme-03#section-4

      It suggests leaving Unicode characters alone (in IRIs) if possible, or using UTF-8 and percent encoding even if the filesystem uses a non-UTF-8 encoding. Perhaps we could leave the filename in the HTML as Unicode characters without percent encoding, and only percent encode the undecodable (surrogate-escaped) bytes.

      This “IRI” scheme is also recommended by <http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx\>, which says on Windows, “in file URIs, percent-encoded octets are interpreted as a byte in the user’s current codepage”. This contradicts the draft RFC and the “pathlib” implementation, which both use UTF-8.

    7. serhiy-storchaka commented on Sep 27, 2015

      @serhiy-storchaka
      serhiy-storchaka
      on Sep 27, 2015
      MemberAuthor

      Yes, perhaps using pathlib is more correct. Updated patch uses pathlib. Can you please test it on Windows?

      Can filenames on Windows contain lone surrogates?

    8. vadmium commented on Sep 27, 2015

      @vadmium
      vadmium
      on Sep 27, 2015
      Member

      I don’t have much to do with Windows, but I understand we don’t support surrogate-escaped bytes there. E.g. see <https://docs.python.org/dev/library/os.html#os.fsdecode\> and sys.getfilesystemencoding(). However I suspect your first patch would have failed on Windows doing os.fsencode(TESTFN_UNENCODABLE); apparently it cannot represent all possible file names in bytes. The second patch doesn’t call fsencode() so this shouldn’t be a problem.

      Your tests do not test that the output is valid Unicode without surrogates. With your first patch applied, when pydoc wrote the HTML to a UTF-8 disk file, I got the error:

      File "/media/disk/home/proj/python/cpy\udcffthon/Lib/pydoc.py", line 2626, in cli
      writedoc(arg)
      File "/media/disk/home/proj/python/cpy\udcffthon/Lib/pydoc.py", line 1659, in writedoc
      file.write(page)
      UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 674: surrogates not allowed

      I have been working on an alternative patch using my IRI (Unicode URLs) proposal for “file:” links, and “surrogatepass” for HTTP links. But I am also trying to fix some related problems with the built-in HTTP server, and the unit tests are a bit tricky.

    9. vadmium commented on Oct 1, 2015

      @vadmium
      vadmium
      on Oct 1, 2015
      Member

      Here is a patch that implements my IRI (Unicode URL) proposal. Differences compared to Serhiy’s patches:

      • “file:” URLs use Unicode if possible. Percent encode encoding only used for reserved ASCII characters and undecodable bytes.

      • HTTP URLs use UTF-8 and retain surrogate escaping. This means that the links generated by “getobj” pages to “getfile” pages work with troublesome paths.

      • Displayed file names are get an ASCII question mark thanks to the “replace” error handler. This means that the generated HTML is now encodable to UTF-8. Serhiy’s path would be displayed something like:

      /home/serhiy/py/cpy?thon-3.5/Lib/pydoc.py (invalid filename encoding)

      • Added some missing html.escape() calls.

    10. serhiy-storchaka commented on Dec 20, 2018

      @serhiy-storchaka
      serhiy-storchaka
      on Dec 20, 2018
      MemberAuthor

      Could you please create a PR for your path Martin?

    11. serhiy-storchaka
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      on Dec 20, 2018
    12. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    13. serhiy-storchaka
      added a commit that references this issue on Aug 29, 2026

      gh-69371: Fix pydoc for modules whose path contains undecodable bytes (…

      c8ca336
    14. serhiy-storchaka
      closed this as completedon Aug 29, 2026
    15. serhiy-storchaka
      added a commit that references this issue on Aug 29, 2026

      [3.14] gh-69371: Fix pydoc for modules whose path contains undecodabl…

      d7339b4
    16. clin1234
      added a commit that references this issue on Sep 12, 2026

      pythongh-69371: Fix pydoc for modules whose path contains undecodable…

      06170da
    17. serhiy-storchaka
      added a commit that references this issue on Sep 13, 2026

      [3.13] gh-69371: Fix pydoc for modules whose path contains undecodabl…

      50f66bf
    18. serhiy-storchaka
      added a commit that references this issue on Oct 10, 2026

      [3.15] gh-69371: Fix pydoc for modules whose path contains undecodabl…

      3e758f2
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of life3.8 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions