finalize_MEDS_metadata

Writes out schema compliant MEDS metadata files for the extracted dataset.

In particular, this script ensures that (1) a metadata/codes.parquet file exists, validated against the MEDS code metadata schema: - code (string) is required - description (string) and parent_codes (list of strings) are typed to the schema when present in the input, but are not added when absent (only the empty-input case emits the full three-column schema) (2) a metadata/dataset.json file exists that has the keys - dataset_name (string) - dataset_version (string) - etl_name (string) - etl_version (string) - meds_version (string) (3) a metadata/subject_splits.parquet file exists that has the mandatory columns - subject_id (Int64) - split (string)

This stage should almost always be the last metadata stage in an extraction pipeline.

The stage is deterministic and cheap, so it is idempotent under resume: any pre-existing output files (e.g., left behind when a prior run was killed after writing its outputs but before the pipeline runner recorded the stage as complete) are removed and rewritten unconditionally. Skipping a completed stage is the runner’s job, not this stage’s.

Details

Property	Value
Type	main
Metadata stage	True

Usage

MEDS_transform-stage <pipeline.yaml> finalize_MEDS_metadata input_dir=<input> output_dir=<output>

Examples

default

Takes the cohort’s aggregated metadata/codes.parquet plus the metadata/.shards.json split map and produces three schema-compliant MEDS metadata files:

• metadata/codes.parquet — validated against the MEDS code metadata schema (required code column; description / parent_codes typed to the schema when present, not added if absent).
• metadata/subject_splits.parquet — derived from the shards map (Int64 subject_id, str split).
• metadata/dataset.json — dataset-level info (name, version, ETL version, MEDS version, and a created_at timestamp; not diffed by the stage test since it changes each run).

This stage should almost always be the last metadata stage in an extraction pipeline.

This example uses the stage’s config.yaml file.

Input files:

metadata/codes.parquet:
  code:
    [EYE_COLOR//BLUE, EYE_COLOR//BROWN, EYE_COLOR//GREEN, HR, MEDS_BIRTH, TEMP]
  description: [null, null, null, Heart Rate, null, Body Temperature]
  code_template: [null, null, null, $test_name, null, $test_name]

metadata/.shards.json:
  train/0: [1, 4]
  tuning/0: [3]
  held_out/0: [2]

Expected output metadata:

metadata/codes.parquet:
  code:
    [EYE_COLOR//BLUE, EYE_COLOR//BROWN, EYE_COLOR//GREEN, HR, MEDS_BIRTH, TEMP]
  description: [null, null, null, Heart Rate, null, Body Temperature]
  code_template: [null, null, null, $test_name, null, $test_name]

metadata/subject_splits.parquet:
  subject_id: [1, 4, 3, 2]
  split: [train, train, tuning, held_out]

Run this stage:

MEDS_transform-stage <pipeline.yaml> finalize_MEDS_metadata input_dir=<input> output_dir=<output>