unarchive
Post-fetch archive unpack for :class:~MEDS_extract.download.source.Source.
Two of the public ETLs (AUMCdb and HIRID) ship their raw data as a single archive that
the rest of the pipeline can’t read directly. Rather than pushing archive handling into
every ETL’s pre_MEDS.py, we expose an optional unarchive field on each
:class:~MEDS_extract.download.source.RemoteFile and unpack in the post-fetch hook
:meth:~MEDS_extract.download.source.Source._maybe_unarchive (invoked from
:meth:~MEDS_extract.download.source.Source._fetch_one whenever bytes newly land at a
dest) — so every transport picks it up for free without bespoke plumbing.
Why stdlib (zipfile / tarfile) and not a third-party library
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Our format set is narrow: zip (AUMCdb), tar.gz (HIRID), plus tar and tgz
for completeness. Both are first-class in stdlib. Alternatives we considered:
libarchive(viapython-libarchive/libarchive-c): native C performance and ~30 supported formats. Rejected: needs the systemlibarchiveshared lib installed (extra OS-level install friction), and our entire format requirement is two formats — neither win matters for one-shot extraction of CSVs that came over the wire.patool/pyunpack: shell out to7z,unrar, etc. Rejected: needs external binaries onPATH, runs subprocesses, and again only buys format breadth we don’t need.extractcode: kitchen-sink wrapper over libarchive + 7z + stdlib. Rejected: same dep-weight issues amplified.
So: import stdlib zipfile and tarfile directly. The code in this module is not
“we re-implemented zip extraction” — it is the path-validation wrapper around the
stdlib calls, surfacing a clear :class:ValueError that names the offending archive
member when a zip-slip / tar-slip / symlink-escape attempt is found.
Module shape: :class:ArchiveFormat (a :class:enum.StrEnum) is the single source of
truth for what we support. :func:safe_extract does format dispatch via a
{format → extractor function} table; per-format extractors share a common safety
pre-pass via :func:_validate_member. Adding a new format means: add an enum value,
write a one-screen extractor function, register it in _EXTRACTORS.
Safety: every member path is validated before any bytes are written — a traversal
attempt raises :class:ValueError while the target directory is still clean. Both
zip-slip and tar-slip variants are covered: absolute paths, .. components, and
symlinks / hardlinks that point outside the target.
The tarfile branch additionally uses Python 3.12+’s data_filter (PEP 706, the
official fix for CVE-2007-4559), which rejects special files (devices, fifos), strips
the setuid / setgid bits, and re-enforces the same path constraints. Our explicit
pre-pass is defense-in-depth: it surfaces a clear :class:ValueError naming the
offending member rather than tarfile’s generic :class:tarfile.FilterError.
ArchiveFormat
Bases: StrEnum
Recognized unarchive values.
AUTO defers the decision to :func:infer_format at extraction time and is the
one-arg way to opt into the full “fetch → extract → cleanup-archive” flow (see
:class:~MEDS_extract.download.source.RemoteFile cleanup_archive semantics).
Explicit values (ZIP, TAR, TAR_GZ) are used when the URL extension is
misleading — e.g. a content-negotiated download URL like
…/api/access/datafile/123 that returns a zip body without a .zip suffix.
TGZ is an alias that resolves to TAR_GZ; preserved as a distinct enum
value so users can write unarchive: tgz in YAML and it round-trips through
:func:resolve_format to the canonical "tar.gz" extractor.
Examples:
>>> ArchiveFormat.AUTO
<ArchiveFormat.AUTO: 'auto'>
>>> ArchiveFormat("zip")
<ArchiveFormat.ZIP: 'zip'>
>>> ArchiveFormat("tar.gz")
<ArchiveFormat.TAR_GZ: 'tar.gz'>
>>> ArchiveFormat("tgz")
<ArchiveFormat.TGZ: 'tgz'>
>>> ArchiveFormat("rar")
Traceback (most recent call last):
...
ValueError: 'rar' is not a valid ArchiveFormat
Source code in MEDS_extract/download/unarchive.py
_extract_tar(archive_path, target_root)
_extract_tar_gz(archive_path, target_root)
_extract_tar_with_mode(archive_path, target_root, *, mode)
Tar extraction body shared between :func:_extract_tar and :func:_extract_tar_gz.
Validates member names AND link targets (a safe-named tar entry can still escape
via a symlink whose linkname walks outside the target). PEP 706’s
data_filter is applied at extractall as belt-and-suspenders defense-in-depth.
Source code in MEDS_extract/download/unarchive.py
_extract_zip(archive_path, target_root)
Extract a .zip archive into target_root after validating every member.
Source code in MEDS_extract/download/unarchive.py
_is_safe_path(member_name, target_root)
Internal helper: True iff member_name resolves strictly inside target_root.
Archive members are POSIX-shaped by format, hence posix_member=True. The
containment check itself is shared with the fetch path’s rel_path validation —
see :mod:MEDS_extract.download._paths for why there is exactly one of it.
Source code in MEDS_extract/download/unarchive.py
_validate_member(member_name, target_root, *, archive_path, fmt_label, link_target=None)
Raise :class:ValueError if extracting member_name would escape target_root.
Checks the member’s name against zip-slip / tar-slip (absolute paths, ..
components, anything that resolves outside target_root). When link_target
is provided (tar symlinks / hardlinks), it is checked the same way — a safe member
name pointing at ../../etc/passwd would otherwise bypass the name check.
Archive entries always use forward-slash separators per format spec, so
:class:~pathlib.PurePosixPath is the right parser regardless of host OS.
Examples:
>>> with tempfile.TemporaryDirectory() as d:
... root = Path(d).resolve()
... archive = root / "x.zip"
... _validate_member("sub/file.csv", root, archive_path=archive, fmt_label="zip")
... _validate_member("./ok.csv", root, archive_path=archive, fmt_label="zip")
Absolute paths, .. escapes, and unsafe link targets all raise:
>>> with tempfile.TemporaryDirectory() as d:
... root = Path(d).resolve()
... archive = root / "x.tar"
... _validate_member("/etc/passwd", root, archive_path=archive, fmt_label="tar")
Traceback (most recent call last):
...
ValueError: Refusing to extract unsafe tar member '/etc/passwd' from ...
>>> with tempfile.TemporaryDirectory() as d:
... root = Path(d).resolve()
... archive = root / "x.zip"
... _validate_member("../escaped.csv", root, archive_path=archive, fmt_label="zip")
Traceback (most recent call last):
...
ValueError: Refusing to extract unsafe zip member '../escaped.csv' from ...
>>> with tempfile.TemporaryDirectory() as d:
... root = Path(d).resolve()
... archive = root / "x.tar"
... _validate_member(
... "good.txt", root, archive_path=archive, fmt_label="tar",
... link_target="../../etc/passwd",
... )
Traceback (most recent call last):
...
ValueError: Refusing to extract unsafe tar member 'good.txt' ...: link target '../../etc/passwd' ...
Source code in MEDS_extract/download/unarchive.py
infer_format(path)
Return the archive format implied by a filename, or None if unknown.
Uses lowercase suffix matching — AUMCdb.ZIP and AUMCdb.zip both resolve
to :attr:ArchiveFormat.ZIP. Returns None for any other extension; callers
treat that as “no unpack needed” under :attr:ArchiveFormat.AUTO semantics.
Examples:
>>> infer_format(Path("AUMCdb.zip"))
<ArchiveFormat.ZIP: 'zip'>
>>> infer_format(Path("hirid.tar.gz"))
<ArchiveFormat.TAR_GZ: 'tar.gz'>
>>> infer_format(Path("raw_stage.tgz"))
<ArchiveFormat.TAR_GZ: 'tar.gz'>
>>> infer_format(Path("release.tar"))
<ArchiveFormat.TAR: 'tar'>
Case-insensitive on the extension (not the whole path):
Non-archive extensions return None so AUTO is a no-op on regular
files — e.g. PhysioNet’s LICENSE.txt or patients.csv.gz:
>>> infer_format(Path("patients.csv.gz")) # gz compression, not a tar archive
>>> infer_format(Path("README.md"))
>>> infer_format(Path("plain"))
Source code in MEDS_extract/download/unarchive.py
resolve_format(unarchive, path)
Normalize a user-facing unarchive token to a canonical :class:ArchiveFormat.
:attr:ArchiveFormat.AUTO delegates to :func:infer_format based on path’s
extension; :attr:ArchiveFormat.TGZ folds to :attr:ArchiveFormat.TAR_GZ
(same extractor). Returns None when AUTO is requested but no known
extension matches — the caller treats that as “leave the file alone”.
Examples:
>>> resolve_format("zip", Path("x.zip"))
<ArchiveFormat.ZIP: 'zip'>
>>> resolve_format("tgz", Path("x.tgz")) # folded to canonical TAR_GZ
<ArchiveFormat.TAR_GZ: 'tar.gz'>
>>> resolve_format("auto", Path("AUMCdb.zip"))
<ArchiveFormat.ZIP: 'zip'>
>>> resolve_format("auto", Path("hirid.tar.gz"))
<ArchiveFormat.TAR_GZ: 'tar.gz'>
AUTO on a non-archive returns None (caller leaves the file alone):
Unknown explicit formats raise — catching typos early beats a silent skip:
>>> resolve_format("rar", Path("x.rar"))
Traceback (most recent call last):
...
ValueError: 'rar' is not a valid ArchiveFormat
Source code in MEDS_extract/download/unarchive.py
safe_extract(archive_path, target_dir, fmt)
Extract archive_path into target_dir with path-traversal guards.
All member paths are validated before any bytes are written — a traversal attempt
raises :class:ValueError while the target directory is still clean. fmt
must be one of :attr:ArchiveFormat.ZIP, :attr:~ArchiveFormat.TAR, or
:attr:~ArchiveFormat.TAR_GZ; AUTO / TGZ should be normalized via
:func:resolve_format first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
archive_path
|
Path
|
The archive to unpack. Must exist. |
required |
target_dir
|
Path
|
Destination directory. Created if absent. |
required |
fmt
|
str | ArchiveFormat
|
Canonical archive format (string or :class: |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
On unsupported / alias |
FileNotFoundError
|
If |
Examples:
Zip round-trip — extraction preserves the internal directory layout under the target:
>>> import zipfile as _zipfile
>>> with tempfile.TemporaryDirectory() as d:
... d = Path(d)
... archive = d / "bundle.zip"
... with _zipfile.ZipFile(archive, "w") as zf:
... zf.writestr("a.csv", "col\n1\n2")
... zf.writestr("sub/b.csv", "col\n3\n4")
... target = d / "out"
... safe_extract(archive, target, "zip")
... sorted(p.relative_to(target).as_posix() for p in target.rglob("*") if p.is_file())
['a.csv', 'sub/b.csv']
Tar.gz round-trip:
>>> import io, tarfile as _tarfile
>>> with tempfile.TemporaryDirectory() as d:
... d = Path(d)
... archive = d / "bundle.tar.gz"
... with _tarfile.open(archive, "w:gz") as tf:
... data = b"col,val\n1,x\n2,y\n"
... info = _tarfile.TarInfo(name="records.csv")
... info.size = len(data)
... tf.addfile(info, io.BytesIO(data))
... target = d / "out"
... safe_extract(archive, target, ArchiveFormat.TAR_GZ)
... (target / "records.csv").read_bytes()
b'col,val\n1,x\n2,y\n'
Zip-slip: a member whose relative path walks out of the target is rejected BEFORE any bytes land on disk, so the target directory is still empty after the rejection:
>>> with tempfile.TemporaryDirectory() as d:
... d = Path(d)
... archive = d / "evil.zip"
... with _zipfile.ZipFile(archive, "w") as zf:
... zf.writestr("../escaped.txt", "pwned")
... target = d / "out"
... target.mkdir()
... try:
... safe_extract(archive, target, "zip")
... except ValueError as e:
... print(f"rejected: {e}")
... print(f"target empty: {sorted(p.name for p in target.iterdir()) == []}")
rejected: Refusing to extract unsafe zip member '../escaped.txt' from ...
target empty: True
Absolute-path member is also rejected:
>>> with tempfile.TemporaryDirectory() as d:
... d = Path(d)
... archive = d / "evil.tar"
... with _tarfile.open(archive, "w") as tf:
... info = _tarfile.TarInfo(name="/etc/passwd-hijack")
... info.size = 3
... tf.addfile(info, io.BytesIO(b"bad"))
... safe_extract(archive, d / "out", "tar")
Traceback (most recent call last):
...
ValueError: Refusing to extract unsafe tar member '/etc/passwd-hijack' ...
Tar-slip via .. is rejected the same way:
>>> with tempfile.TemporaryDirectory() as d:
... d = Path(d)
... archive = d / "evil.tar.gz"
... with _tarfile.open(archive, "w:gz") as tf:
... info = _tarfile.TarInfo(name="../escaped.csv")
... info.size = 3
... tf.addfile(info, io.BytesIO(b"bad"))
... safe_extract(archive, d / "out", "tar.gz")
Traceback (most recent call last):
...
ValueError: Refusing to extract unsafe tar member '../escaped.csv' ...
Aliases (AUTO / TGZ) are not directly extractable — :func:resolve_format
must collapse them first:
>>> with tempfile.TemporaryDirectory() as d:
... safe_extract(Path(d) / "any.bin", Path(d) / "out", "auto")
Traceback (most recent call last):
...
ValueError: safe_extract: alias format 'auto' must be resolved via resolve_format(...) first.
Source code in MEDS_extract/download/unarchive.py
322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 | |