finalize_MEDS_data
Writes out schema compliant MEDS data files for the extracted dataset.
In particular, this script ensures that all shard files are MEDS compliant with the mandatory columns
- subject_id (Int64)
- time (DateTime)
- code (String)
- numeric_value (Float32)
This stage should almost always be the last data stage in an extraction pipeline.
Details
| Property | Value |
|---|---|
| Type | map |
| Metadata stage | False |
Usage
Examples
default
Takes a merged MEDS cohort (the output of merge_to_MEDS_cohort, with the auxiliary
source_block column alongside the MEDS core; the internal code_components struct is
already dropped at merge — see #254) and returns the same rows with the mandatory MEDS
schema applied to the core columns: subject_id: Int64, time: Datetime("us"),
code: String, numeric_value: Float32. Extension columns pass through unchanged.
This stage should almost always be the last data stage in an extraction pipeline.
Input files:
data/train/0.parquet:
subject_id: [1, 1, 1, 1, 4, 4, 4, 4]
code:
[
EYE_COLOR//BROWN,
MEDS_BIRTH,
HR,
TEMP,
EYE_COLOR//BROWN,
MEDS_BIRTH,
HR,
TEMP,
]
time:
- null
- 2000-01-01T00:00:00
- 2020-01-01T10:00:00
- 2020-01-01T11:00:00
- null
- 2003-04-04T00:00:00
- 2020-01-04T09:00:00
- 2020-01-04T10:00:00
source_block:
- patients/eye_color
- patients/dob
- labs/lab
- labs/lab
- patients/eye_color
- patients/dob
- labs/lab
- labs/lab
numeric_value: [null, null, 80.0, 36.6, null, null, 70.0, 36.8]
data/tuning/0.parquet:
subject_id: [3, 3, 3, 3]
code: [EYE_COLOR//GREEN, MEDS_BIRTH, HR, TEMP]
time:
- null
- 2002-03-03T00:00:00
- 2020-01-03T14:00:00
- 2020-01-03T15:00:00
source_block: [patients/eye_color, patients/dob, labs/lab, labs/lab]
numeric_value: [null, null, 85.0, 36.5]
data/held_out/0.parquet:
subject_id: [2, 2, 2, 2]
code: [EYE_COLOR//BLUE, MEDS_BIRTH, HR, TEMP]
time:
- null
- 2001-02-02T00:00:00
- 2020-01-02T12:00:00
- 2020-01-02T13:00:00
source_block: [patients/eye_color, patients/dob, labs/lab, labs/lab]
numeric_value: [null, null, 75.0, 37.0]
Expected output files:
data/train/0.parquet:
subject_id: [1, 1, 1, 1, 4, 4, 4, 4]
code:
[
EYE_COLOR//BROWN,
MEDS_BIRTH,
HR,
TEMP,
EYE_COLOR//BROWN,
MEDS_BIRTH,
HR,
TEMP,
]
time:
- null
- 2000-01-01T00:00:00
- 2020-01-01T10:00:00
- 2020-01-01T11:00:00
- null
- 2003-04-04T00:00:00
- 2020-01-04T09:00:00
- 2020-01-04T10:00:00
source_block:
- patients/eye_color
- patients/dob
- labs/lab
- labs/lab
- patients/eye_color
- patients/dob
- labs/lab
- labs/lab
numeric_value: [null, null, 80.0, 36.6, null, null, 70.0, 36.8]
data/tuning/0.parquet:
subject_id: [3, 3, 3, 3]
code: [EYE_COLOR//GREEN, MEDS_BIRTH, HR, TEMP]
time:
- null
- 2002-03-03T00:00:00
- 2020-01-03T14:00:00
- 2020-01-03T15:00:00
source_block: [patients/eye_color, patients/dob, labs/lab, labs/lab]
numeric_value: [null, null, 85.0, 36.5]
data/held_out/0.parquet:
subject_id: [2, 2, 2, 2]
code: [EYE_COLOR//BLUE, MEDS_BIRTH, HR, TEMP]
time:
- null
- 2001-02-02T00:00:00
- 2020-01-02T12:00:00
- 2020-01-02T13:00:00
source_block: [patients/eye_color, patients/dob, labs/lab, labs/lab]
numeric_value: [null, null, 75.0, 37.0]
Run this stage: