convert_to_MEDS_events

Convert event-sharded raw data into MEDS events per subject shard.

All arguments are specified through the command line into the cfg object through Hydra.

Details

Property	Value
Type	main
Metadata stage	False

Default Configuration

do_dedup_text_and_numeric: true

Usage

MEDS_transform-stage <pipeline.yaml> convert_to_MEDS_events input_dir=<input> output_dir=<output>

Examples

default

Applies the MESSY event config to each subject-sharded source table, emitting one row per configured (table, event) definition per source row. Each row gets:

• subject_id — resolved via the table’s _defaults.subject_id expression ($MRN, $patient_id)
• code — from the event’s code dftly expression (e.g. f"EYE_COLOR//{$eye_color}", $test_name)
• code_components — per-row struct preserving the raw source columns referenced by code
• time — from the event’s time expression (null for static events like eye_color)
• numeric_value (labs only) — from $result
• source_block — "{table}/{event}" provenance string

Writes one parquet per (split, table) pair at data/<split>/<shard>/<table>.parquet.

Stage configuration:

do_dedup_text_and_numeric: false

This example uses the stage’s config.yaml file.

Input files:

data/train/0/patients.parquet:
  MRN: [1, 4]
  dob: ["2000-01-01T00:00:00", "2003-04-04T00:00:00"]
  eye_color: ["BROWN", "BROWN"]

data/train/0/labs.parquet:
  patient_id: [1, 1, 4, 4]
  timestamp:
    [
      "2020-01-01T10:00:00",
      "2020-01-01T11:00:00",
      "2020-01-04T09:00:00",
      "2020-01-04T10:00:00",
    ]
  test_name: ["HR", "TEMP", "HR", "TEMP"]
  result: [80.0, 36.6, 70.0, 36.8]

data/tuning/0/patients.parquet:
  MRN: [3]
  dob: ["2002-03-03T00:00:00"]
  eye_color: ["GREEN"]

data/tuning/0/labs.parquet:
  patient_id: [3, 3]
  timestamp: ["2020-01-03T14:00:00", "2020-01-03T15:00:00"]
  test_name: ["HR", "TEMP"]
  result: [85.0, 36.5]

data/held_out/0/patients.parquet:
  MRN: [2]
  dob: ["2001-02-02T00:00:00"]
  eye_color: ["BLUE"]

data/held_out/0/labs.parquet:
  patient_id: [2, 2]
  timestamp: ["2020-01-02T12:00:00", "2020-01-02T13:00:00"]
  test_name: ["HR", "TEMP"]
  result: [75.0, 37.0]

metadata/.shards.json:
  train/0: [1, 4]
  tuning/0: [3]
  held_out/0: [2]

messy.yaml: |
  _defaults:
    subject_id: $MRN
  patients:
    eye_color:
      code: 'f"EYE_COLOR//{$eye_color}"'
      time: null
    dob:
      code: MEDS_BIRTH
      time: '$dob::"%Y-%m-%dT%H:%M:%S"'
  labs:
    _defaults:
      subject_id: $patient_id
    lab:
      code: $test_name
      time: '$timestamp::"%Y-%m-%dT%H:%M:%S"'
      numeric_value: $result

Expected output files:

data/train/0/patients.parquet:
  subject_id: [1, 4, 1, 4]
  code: ["EYE_COLOR//BROWN", "EYE_COLOR//BROWN", "MEDS_BIRTH", "MEDS_BIRTH"]
  code_components:
    - { eye_color: "BROWN" }
    - { eye_color: "BROWN" }
    - null
    - null
  time:
    - null
    - null
    - 2000-01-01T00:00:00
    - 2003-04-04T00:00:00
  source_block:
    ["patients/eye_color", "patients/eye_color", "patients/dob", "patients/dob"]

data/train/0/labs.parquet:
  subject_id: [1, 1, 4, 4]
  code: ["HR", "TEMP", "HR", "TEMP"]
  code_components:
    - { test_name: "HR" }
    - { test_name: "TEMP" }
    - { test_name: "HR" }
    - { test_name: "TEMP" }
  time:
    [
      2020-01-01T10:00:00,
      2020-01-01T11:00:00,
      2020-01-04T09:00:00,
      2020-01-04T10:00:00,
    ]
  numeric_value: [80.0, 36.6, 70.0, 36.8]
  source_block: ["labs/lab", "labs/lab", "labs/lab", "labs/lab"]

data/tuning/0/patients.parquet:
  subject_id: [3, 3]
  code: ["EYE_COLOR//GREEN", "MEDS_BIRTH"]
  code_components:
    - { eye_color: "GREEN" }
    - null
  time:
    - null
    - 2002-03-03T00:00:00
  source_block: ["patients/eye_color", "patients/dob"]

data/tuning/0/labs.parquet:
  subject_id: [3, 3]
  code: ["HR", "TEMP"]
  code_components:
    - { test_name: "HR" }
    - { test_name: "TEMP" }
  time: [2020-01-03T14:00:00, 2020-01-03T15:00:00]
  numeric_value: [85.0, 36.5]
  source_block: ["labs/lab", "labs/lab"]

data/held_out/0/patients.parquet:
  subject_id: [2, 2]
  code: ["EYE_COLOR//BLUE", "MEDS_BIRTH"]
  code_components:
    - { eye_color: "BLUE" }
    - null
  time:
    - null
    - 2001-02-02T00:00:00
  source_block: ["patients/eye_color", "patients/dob"]

data/held_out/0/labs.parquet:
  subject_id: [2, 2]
  code: ["HR", "TEMP"]
  code_components:
    - { test_name: "HR" }
    - { test_name: "TEMP" }
  time: [2020-01-02T12:00:00, 2020-01-02T13:00:00]
  numeric_value: [75.0, 37.0]
  source_block: ["labs/lab", "labs/lab"]

Run this stage:

MEDS_transform-stage <pipeline.yaml> convert_to_MEDS_events input_dir=<input> output_dir=<output>