convert_to_parquet
Normalize every needed raw source table into the stage output as parquet.
Reads cfg.input_dir directly rather than deriving it from
stage_cfg.data_input_dir. As the pipeline’s first stage its data_input_dir
is <input_dir>/data, which is where the cohort lives, not where the user’s
raw files are — shard_events compensated with a .parent walk-up, and that
sleight of hand is exactly what made the pipeline unrunnable without it (#186).
Naming input_dir explicitly keeps raw-data resolution in one honest place.
Details
| Property | Value |
|---|---|
| Type | main |
| Metadata stage | False |
Usage
Examples
default
This example uses the stage’s
config.yamlfile.
Input files:
patients.csv: |
MRN,dob,eye_color
1,2000-01-01T00:00:00,BROWN
2,2001-02-02T00:00:00,BLUE
3,2002-03-03T00:00:00,GREEN
4,2003-04-04T00:00:00,BROWN
labs.csv: |
patient_id,timestamp,test_name,result
1,2020-01-01T10:00:00,HR,80
1,2020-01-01T11:00:00,TEMP,36.6
2,2020-01-02T12:00:00,HR,75
2,2020-01-02T13:00:00,TEMP,37.0
3,2020-01-03T14:00:00,HR,85
3,2020-01-03T15:00:00,TEMP,36.5
4,2020-01-04T09:00:00,HR,70
4,2020-01-04T10:00:00,TEMP,36.8
lab_descriptions.csv: |
test_name,description
HR,Heart Rate
TEMP,Body Temperature
messy.yaml: |
_defaults:
subject_id: $MRN
patients:
eye_color:
code: 'f"EYE_COLOR//{$eye_color}"'
time: null
dob:
code: MEDS_BIRTH
time: '$dob::"%Y-%m-%dT%H:%M:%S"'
labs:
_defaults:
subject_id: $patient_id
lab:
code: $test_name
time: '$timestamp::"%Y-%m-%dT%H:%M:%S"'
numeric_value: $result
_metadata:
lab_descriptions:
test_name: $test_name
description: $description
Expected output files:
# One parquet per INPUT file, at the same relative position — no row-range chunking.
# Columns are projected to what the MESSY config references: ``labs`` keeps all four
# of its columns because the event block uses them all, while ``patients`` drops
# nothing here for the same reason. ``lab_descriptions`` is a ``_metadata`` table and
# is deliberately absent: ``extract_code_metadata`` reads those from the raw input.
data/patients.parquet:
MRN: [1, 2, 3, 4]
dob:
[
"2000-01-01T00:00:00",
"2001-02-02T00:00:00",
"2002-03-03T00:00:00",
"2003-04-04T00:00:00",
]
eye_color: ["BROWN", "BLUE", "GREEN", "BROWN"]
data/labs.parquet:
patient_id: [1, 1, 2, 2, 3, 3, 4, 4]
timestamp:
[
"2020-01-01T10:00:00",
"2020-01-01T11:00:00",
"2020-01-02T12:00:00",
"2020-01-02T13:00:00",
"2020-01-03T14:00:00",
"2020-01-03T15:00:00",
"2020-01-04T09:00:00",
"2020-01-04T10:00:00",
]
test_name: ["HR", "TEMP", "HR", "TEMP", "HR", "TEMP", "HR", "TEMP"]
result: [80.0, 36.6, 75.0, 37.0, 85.0, 36.5, 70.0, 36.8]
Run this stage: