Skip to content

tarfile: GNU sparse 1.0 pax tar header offset not properly computed #83869

Closed
Closed
tarfile: GNU sparse 1.0 pax tar header offset not properly computed#83869
Labels
3.13only security fixes3.14bugs and security fixes3.15bugs and security fixes3.16new features, bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@nitmir
nitmir
mannequin
opened on Feb 19, 2020
BPO 39688
Nosy @nitmir
PRs
  • gh-83869: tarfile: compute next header offset using pax size for sparse file #18562
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = None
created_at = <Date 2020-02-19.14:34:50.810>
labels = ['3.7', '3.8', 'type-bug', 'library', '3.9']
title = 'tarfile: GNU sparse 1.0 pax tar header offset not properly computed'
updated_at = <Date 2020-02-19.16:28:29.996>
user = 'https://github.com/nitmir'

    bugs.python.org fields:

    activity = <Date 2020-02-19.16:28:29.996>
actor = 'Nit'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-02-19.14:34:50.810>
creator = 'Nit'
dependencies = []
files = []
hgrepos = []
issue_num = 39688
keywords = ['patch']
message_count = 3.0
messages = ['362275', '362279', '362282']
nosy_count = 1.0
nosy_names = ['Nit']
pr_nums = ['18562']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39688'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

    Linked PRs

    Activity

    1. nitmir commented on Feb 19, 2020

      @nitmir
      nitmirmannequin
      on Feb 19, 2020
      MannequinAuthor

      When tarfile open a tar containing a sparse file where actual data is bigger than 0o77777777777 bytes (~8GB), it fails listing members after this file. As so, it is impossible to access/extract files located after a such file inside the archive.

      A tar file presenting the issue is available at https://genua.fr/sample.tar.xz
      Uncompressed the file is ~16GB.
      It containes two files:

      • disk.img a 50GB sparse file containing ~16GB of data
      • README.txt a simple text file containing "This last file is not properly listed"

      disk.img was generated using the folowing python script:

      GB = 1024**3
buf = b"\xFF" * 1024**2
with open('disk.img', 'wb') as f:
    f.seek(10 * GB)
    wrotten = 0
    while wrotten < 0o77777777777:
        wrotten += f.write(buf)
        f.flush()
        print(wrotten/0o77777777777 * 100, '%')
    f.seek(50 * GB - 1)
    f.write(b'\0')

      sample.tar was generated using GNU tar 1.30 on a Debian 10 with the following command:

      tar --format pax -cvSf sample.tar disk.img README.txt

      The following script expose the issue:

      import tarfile
t = tarfile.open('sample.tar')
print('members', t.getmembers())
print('offset', t.offset)

      Its output is:

      members [<TarInfo 'disk.img' at 0x7f5b14242b38>]
      offset 17179806208

      members should also list README.txt.

      I think I have found the root cause of the bug: Because the file is bigger than 0o77777777777, it's size cannot be specified inside the tar ustar header, so a "size" pax extented header is generated. This header contain the full size of the file block in the tar.

      As the file is sparse, as of sparse format 1.0, the file block contains first a sparse mapping, then the file data. So this block size is the size of the mapping added to the size of the data.

      Because the file is sparse, a GNU.sparse.realsize header is also added containing the full expanded file size (here 50GB).

      Here

      cpython/Lib/tarfile.py

      Line 1350 in 4dee92b

      elif keyword == "GNU.sparse.realsize":
      tarfile set the tarinfo size to GNU.sparse.realsize (50GB),then, in this block

      cpython/Lib/tarfile.py

      Line 1297 in 4dee92b

      if "size" in pax_headers:
      the file offset is moved forward from GNU.sparse.realsize (50GB) instead of pax_headers["size"]. Moreover, the move is done from next.offset_data which is set at https://github.com/python/cpython/blob/master/Lib/tarfile.py#L1338 to after the sparse mapping.
      The move forward in the sparse file should be made from next.offset + BLOCKSIZE.

    2. nitmir
      added
      3.7 (EOL)end of life
      3.8 (EOL)end of life
      3.9 (EOL)end of life
      stdlibStandard Library Python modules in the Lib/ directory
      type-bugAn unexpected behavior, bug, or error
      on Feb 19, 2020

    3. nitmir commented on Feb 19, 2020

      @nitmir
      nitmirmannequin
      on Feb 19, 2020
      MannequinAuthor

      This commit fix the issue nitmir@50c1f68

      I am currently unable to create pull request on github (error 500)

    4. nitmir commented on Feb 19, 2020

      @nitmir
      nitmirmannequin
      on Feb 19, 2020
      MannequinAuthor

      hum trying to be clever and I am doing mistakes.

      This commit is simpler and effectively fixes the issue nitmir@682138a

      tarinfo.offset_data is where tarfile start reading the file data and thus must be set to the beginning of the actual data for a sparse file even if the data block starts before.

    5. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    6. serhiy-storchaka
      added this to Tarfile issues 🗄️on May 20, 2022
    7. serhiy-storchaka
      added
      3.13only security fixes
      3.14bugs and security fixes
      3.15bugs and security fixes
      3.16new features, bugs and security fixes
      and removed
      3.9 (EOL)end of life
      3.8 (EOL)end of life
      3.7 (EOL)end of life
      on Aug 14, 2026
    8. serhiy-storchaka
      added a commit that references this issue on Aug 15, 2026

      gh-83869: tarfile: compute next header offset using pax size for spar…

      1a52eae
    9. serhiy-storchaka
      closed this as completedon Aug 15, 2026
    10. github-project-automation
      moved this to Done in Tarfile issues 🗄️on Aug 15, 2026
    11. serhiy-storchaka
      added 2 commits that reference this issue on Aug 15, 2026

      [3.13] gh-83869: tarfile: compute next header offset using pax size f…

      66861f1

      [3.14] gh-83869: tarfile: compute next header offset using pax size f…

      422e679
    12. serhiy-storchaka
      added a commit that references this issue on Oct 10, 2026

      [3.15] gh-83869: tarfile: compute next header offset using pax size f…

      5c6758a

    13. gpshead commented on Oct 14, 2026

      @gpshead
      gpshead
      on Oct 14, 2026
      Member

      This appears to have caused a regression FYI - #159435 @serhiy-storchaka

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.13only security fixes3.14bugs and security fixes3.15bugs and security fixes3.16new features, bugs and security fixesstdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions