extract_code_metadata
Extracts any dataset-specific metadata and adds it to any existing code metadata file.
This script can extract arbitrary, code-linked metadata columns from input mappings and add them to the
metadata/codes.parquet file. The metadata columns are extracted from the raw metadata files using
_metadata blocks in the MESSY_config_fp file; see MEDS_extract.config (MessyConfig and
compile_metadata_block) for the block semantics and the dftly documentation for the expression
DSL.
Metadata is attached to codes through one join path: each _metadata entry’s key
columns (the produced columns whose names match the code expression’s component
columns — see MEDS_extract.config.compile_metadata_block) are joined against the
observed code_components values, scoped to the declaring event block, with null
keys matching null components.
The stage is a shard-scoped map-reduce whose work items are the extracted event
files, so per-task memory is bounded by one shard plus one (vocabulary-sized)
metadata table — never by the dataset. Each map task writes vocabulary-sized
partials (a shard’s distinct observed codes, and per _metadata block the
shard’s metadata-to-code expansion — _self blocks read values off the shard’s
own rows with no join at all), and worker 0’s reduction only ever unions and
aggregates those partials. External metadata tables are re-read once per (shard,
block) task — deliberate: they are vocabulary-sized, so re-reading a small file
beats any whole-dataset pass.
The output enumerates EVERY code observed in the data — recent MEDS spec versions
require the full observed code vocabulary in metadata/codes.parquet for the
dataset to be valid. Codes with no metadata match carry all-null metadata columns.
When no _metadata blocks are configured at all, the stage still writes the full
observed code vocabulary (a codes-only table).
When _metadata blocks ARE configured, the stage’s data input must be the
component-bearing convert_to_MEDS_events output: every block’s join keys must
name component columns some event file carries, and an input with no
code_components anywhere (e.g. merged data — merge_to_MEDS_cohort drops that
column) is an error, never a silent codes-only degrade.
Note that there are two sentinel columns in the output metadata that have certain mandates for MEDS
compliance: The description column and the parent_codes column. The description column must be a
string, and if there are multiple matches in the extracted metadata for a code, in this script they will
be concatenated into a single string with the description_separator string. The parent_codes column
must be a list of strings, each formatted as an OMOP vocabulary name, followed by a “/”, followed by the
OMOP concept code. This column is used to link codes to their parent codes in the OMOP vocabulary.
The reduced output has a canonical, data-independent shape: every extracted metadata column
other than description and code_template is aggregated per code to a List(String) of
distinct values, sorted within each group (a single-source unique code yields a one-element
list). description joins its distinct values with description_separator in canonical
config order. code_template is a plain String — one code has exactly one template, and
distinct templates colliding on a single code is a configuration error. Missing values are
null, never an empty list or empty string.
All arguments are specified through the command line into the cfg object through Hydra.
The cfg.stage_cfg object is a special key that is imputed by OmegaConf to contain the stage-specific
configuration arguments based on the global, pipeline-level configuration file.
Details
| Property | Value |
|---|---|
| Type | main |
| Metadata stage | True |
Default Configuration
Usage
Examples
default
Joins external metadata files (declared via _metadata blocks in the event config) onto the
extracted event codes. Each _metadata entry maps output column names to dftly expressions
over the raw metadata table; produced columns whose names match the code’s component columns
are the join keys. Here, labs/lab._metadata.lab_descriptions produces test_name (the
code’s single component, so it is the join key) and a description output from
lab_descriptions.csv. The reducer writes the joined table to metadata/codes.parquet, with
a code_template column preserving the original dftly expression that produced each code
(useful for downstream provenance tracking). The output enumerates every observed code — a
MEDS validity requirement — so codes with no metadata match (EYE_COLOR//*, MEDS_BIRTH)
appear with null description / code_template.
This example uses the stage’s
config.yamlfile.
Input files:
data/train/0.parquet:
subject_id: [1, 1, 1, 1, 4, 4, 4, 4]
code:
[
EYE_COLOR//BROWN,
MEDS_BIRTH,
HR,
TEMP,
EYE_COLOR//BROWN,
MEDS_BIRTH,
HR,
TEMP,
]
code_components:
- { eye_color: BROWN, test_name: null }
- null
- { eye_color: null, test_name: HR }
- { eye_color: null, test_name: TEMP }
- { eye_color: BROWN, test_name: null }
- null
- { eye_color: null, test_name: HR }
- { eye_color: null, test_name: TEMP }
time:
- null
- 2000-01-01T00:00:00
- 2020-01-01T10:00:00
- 2020-01-01T11:00:00
- null
- 2003-04-04T00:00:00
- 2020-01-04T09:00:00
- 2020-01-04T10:00:00
source_block:
- patients/eye_color
- patients/dob
- labs/lab
- labs/lab
- patients/eye_color
- patients/dob
- labs/lab
- labs/lab
numeric_value: [null, null, 80.0, 36.6, null, null, 70.0, 36.8]
data/tuning/0.parquet:
subject_id: [3, 3, 3, 3]
code: [EYE_COLOR//GREEN, MEDS_BIRTH, HR, TEMP]
code_components:
- { eye_color: GREEN, test_name: null }
- null
- { eye_color: null, test_name: HR }
- { eye_color: null, test_name: TEMP }
time:
- null
- 2002-03-03T00:00:00
- 2020-01-03T14:00:00
- 2020-01-03T15:00:00
source_block: [patients/eye_color, patients/dob, labs/lab, labs/lab]
numeric_value: [null, null, 85.0, 36.5]
data/held_out/0.parquet:
subject_id: [2, 2, 2, 2]
code: [EYE_COLOR//BLUE, MEDS_BIRTH, HR, TEMP]
code_components:
- { eye_color: BLUE, test_name: null }
- null
- { eye_color: null, test_name: HR }
- { eye_color: null, test_name: TEMP }
time:
- null
- 2001-02-02T00:00:00
- 2020-01-02T12:00:00
- 2020-01-02T13:00:00
source_block: [patients/eye_color, patients/dob, labs/lab, labs/lab]
numeric_value: [null, null, 75.0, 37.0]
metadata/codes.parquet:
code: [HR, TEMP]
lab_descriptions.csv: |
test_name,description
HR,Heart Rate
TEMP,Body Temperature
messy.yaml: |
_defaults:
subject_id: $MRN
patients:
_defaults:
subject_id: $patient_id
eye_color:
code: 'f"EYE_COLOR//{$eye_color}"'
time: null
dob:
code: MEDS_BIRTH
time: '$dob::"%Y-%m-%dT%H:%M:%S"'
labs:
_defaults:
subject_id: $patient_id
lab:
code: $test_name
time: '$timestamp::"%Y-%m-%dT%H:%M:%S"'
numeric_value: $result
_metadata:
lab_descriptions:
test_name: $test_name
description: $description
Expected output metadata:
metadata/codes.parquet:
code:
[EYE_COLOR//BLUE, EYE_COLOR//BROWN, EYE_COLOR//GREEN, HR, MEDS_BIRTH, TEMP]
code_template: [null, null, null, $test_name, null, $test_name]
description: [null, null, null, Heart Rate, null, Body Temperature]
Run this stage: