Track 2

Extracting Social Determinants of Health

Tentative Timeline

Registration January 11—April 1, 2022
Training Data Release for Subtasks A & B February 14, 2022
Test Data Release for Subtasks A & B June 6, 2022
System Outputs Due for Subtasks A & B June 7, 2022 (11:59pm Eastern Time)
Training Data Release for Subtask C June 8, 2022
Test Data Release for Subtask C June 9, 2022
System Outputs Due for Subtask C June 10, 2022 (11:59pm Eastern Time)
Aggregate Results Release July 29, 2022
Abstract Submission August 15, 2022

Description

Social determinants of health (SDOH) are the conditions in which people live that affect quality-of-life and health outcomes[1]. SDOH include a wide range of conditions, including but not limited to substance use, living situation, employment, education, racism, geography, and pollution. SDOH may be contributing to decreased life expectancy, especially for specific populations[2–4]. Knowledge of SDOH and the behaviors influenced by these social factors can inform clinical decision-making and improve health outcomes[5].

While some patient SDOH information is contained within the Electronic Health Record (EHR) as structured data, the clinical narrative contains a detailed characterization of several important SDOH, beyond the structured representation. Utilizing text-encoded SDOH information in large-scale retrospective studies, clinical decision-support systems, and other secondary use applications, requires the automatic extraction of a structured representation that captures the key aspects of the social determinants. 

Task Overview

This task explores the extraction of SDOH information from clinical notes. It will utilize the Social History Annotation Corpus (SHAC)[6]. SHAC includes social history sections annotated for substance use (alcohol, drug, tobacco), employment, and living status. SHAC uses an event-based annotation scheme, where each social determinant event includes a trigger that anchors the event and one or more arguments that characterize the event. The arguments capture status, type, extent, and temporal information.

Figure 1 presents some example annotations for employment and substance use. SHAC includes clinical text from MIMIC-III[7] and the University of Washington (UW). A portion of SHAC was actively sampled to increase the diversity and richness of the annotations. The corpus includes many important risk factors, including substance abuse, homelessness, and unemployment. Training and test data for this task will utilize the SHAC annotations, which will be provided using the BRAT standoff format[8]. The prediction task involves identifying trigger and argument spans, normalizing arguments, and predicting links between trigger and argument spans. This SDOH extraction task will include three subtasks related to information extraction, generalizability, and transfer learning. Example SDOH annotations in BRAT
Figure 1. Example SDOH annotations in BRAT

  • Subtask A: Extraction — In this subtask, participants will be provided a MIMIC-III training set \((\mathcal{D}_{train}^{mimic})\), and extraction performance will be evaluated on a MIMIC-III test set \((\mathcal{D}_{test-a}^{mimic})\)
  • Subtask B: Generalizability — This subtask will explore the generalizability of the extraction frameworks from one institution to another. Participants will be provided the same MIMIC-III training set as Subtask A \((\mathcal{D}_{train}^{mimic})\), and extraction performance will be evaluated on a UW test set \((\mathcal{D}_{test-b}^{uw})\)
  • Subtask C: Learning Transfer — The goal of this subtask is to explore learning transfer from one institution to another. Participants will be provided the same MIMIC-III training set \((\mathcal{D}_{train}^{mimic})\), as well as a UW training set \((\mathcal{D}_{train}^{uw})\). Extraction performance will be evaluated on a new UW test set \((\mathcal{D}_{test-c}^{uw})\)

Evaluation Format

Evaluation will be conducted using withheld test sets for each subtask (A, B, and C). Performance will be measured by the precision, recall, and F1 score for the extraction of events (triggers, arguments, and argument roles) relative to the gold standard annotations.

Data will be provided and evaluated in BRAT format (see https://brat.nlplab.org/). The evaluation routine compares two directories with BRAT-style annotations (*.txt and *.ann files). It identifies all the *.ann files in both directories, finds matching filenames in the directories, and then compares the annotations defined in the *.ann files. The evaluation criteria and evaluation scripts are available at https://github.com/Lybarger/brat_scoring.

Participating teams are required to register and sign Data Use Agreements for MIMIC and for the University of Washington to get access to the datasets.

Each team can submit up to 3 runs for each subtask. Prediction submissions shall be in BRAT standoff format[7], similar to the gold annotations.

Dissemination

Participants are asked to submit a 500-word abstract describing their methodologies. Abstracts may also have a graphical summary of the proposed architecture. The document should not exceed 2 pages (1.5 line spacing, 12pt-font size). The authors of top performing systems or particularly novel approaches will be invited to present or demonstrate their systems at the workshop. A journal venue will be organized following the workshop.

Contact

Please join the discussion group below for announcements. Questions about the challenge can be addressed to the organizers by posting to the group (New conversation button) or sending email to the address below.
Discussion Group: N2C2/UW_2022_SDOH
Email: n2c2_uw_2022_sdoh@googlegroups.com

References

[1]Social determinants of health [Internet]. Centers for Disease Control and Prevention. 2021. Available from: https://www.cdc.gov/socialdeterminants/index.htm

[2]Daniel H, Bornstein SS, Kane GC. Addressing social determinants to improve patient care and promote health equity: an American College of Physicians position paper. Annals of Internal Medicine. 2018 Apr 17;168(8):577-8. doi: 10.7326/m17-2441.

[3]Himmelstein DU, Woolhandler S. Determined action needed on social determinants. Annals of Internal Medicine. 2018 Apr 17;168(8):596-7. doi: 10.7326/M18-0335.

[4]Singh GK, Daus GP, Allender M, Ramey CT, Martin EK, Perry C, De Los Reyes AA, Vedamuthu IP. Social determinants of health in the United States: addressing major health inequality trends for the nation, 1935-2016. International Journal of MCH and AIDS. 2017;6(2):139. doi: 10.21106/ijma.236.

[5]Blizinsky KD, Bonham VL. Leveraging the learning health care model to improve equity in the age of genomic medicine. Learning Health Systems. 2018 Jan;2(1):e10046. doi: 10.1002/lrh2.10046.

[6]Lybarger K, Ostendorf M, Yetisgen M. Annotating social determinants of health using active learning, and characterizing determinants using neural event extraction. Journal of Biomedical Informatics. 2021 Jan 1;113:103631. doi: 10.1016/j.jbi.2020.103631.

[7]Johnson AE, Pollard TJ, Shen L, Li-Wei HL, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG. MIMIC-III, a freely accessible critical care database. Scientific Data. 2016;3(1):1-9. doi: 10.1038/sdata.2016.35.

[8]Stenetorp P, Pyysalo S, Topić G, Ohta T, Ananiadou S, Tsujii JI. BRAT: a web-based tool for NLP-assisted text annotation. In European Chapter of the Association for Computational Linguistics 2012 Apr: 102-107. Available from: https://aclanthology.org/E12-2021.