Contextualized Medication Event Extraction CMED

Tentative Timeline

Description

Understanding medication events in clinical notes is essential to achieving a complete picture of a patient's medication history. While prior research has explored identification of medication changes in clinical notes, due to the longitudinal and narrative nature of clinical documentation, extraction of medication change alone without the necessary clinical context is insufficient for use in real-world applications, such as medication timeline generation and medication reconciliation. To bridge this gap, this track aims to capture multi-dimensional context of medication changes documented in clinical notes.

Task Overview

This track will be using the Contextualized Medication Event Dataset (CMED). The overall task is to identify all medication mentions within a clinical note, indicate whether a change has been/is being discussed, and classify change events along 5 contextual dimensions. The task will consist of 3 subtasks:

• [NER] Medication Extraction – Extract all medication mentions in clinical notes. This is a Named-entity recognition task.
• [Event] Event Classification – Classify medication mentions in clinical notes as either: Disposition (medication change discussed), NoDisposition (no change discussed), or Undetermined (need more information).
• [Context] Context Classification – Classify the contextual information for Disposition events along 5 orthogonal dimensions: Action (e.g. start, stop), Negation (e.g. negated), Temporality (e.g. past, present), Certainty (e.g. hypothetical, conditional), and Actor (e.g. patient, physician).

Participants can choose to participate in one or all these tasks. Training data will contain 400 clinical notes from Partners Healthcare with corresponding annotations in brat (.ann) format. Test data will contain 100 clinical notes. The notes will be distributed with a data use agreement.

Evaluation Format

Evaluation will be conducted using the withheld test data. Test data will be released in 3 stages.

• Release 1 will evaluate NER, NE+Event, and End-to-end tasks on unannotated notes.
• Release 2 will evaluate Event and Event+Context tasks on gold-standard medications.
• Release 3 will evaluate Context task on gold-standard Disposition medication events.

Evaluation will be measured using F₁ calculated at micro- and macro- averaged levels.

For medication extraction only, we will employ two kinds of evaluation—strict and lenient matching. For strict matching, the offsets of a span must match exactly. For lenient matching, it is sufficient for spans to overlap.

Each team can submit up to 3 runs for each Release. During submission time, for each submission, teams can specify which subtask the submission is for. All submissions should be in brat format, similar to the gold annotations.

Dissemination

Participants are asked to submit a 500-word long abstract describing their methodologies. Abstracts may also have a graphical summary of the proposed architecture. The document should not exceed 2 pages (1.5 line spacing, 12pt-font size). The authors of either top performing systems or particularly novel approaches will be invited to present and/or demonstrate their systems at the workshop.

Contact

Please join the discussion group below for announcements. Questions about the challenge can be addressed to the organizers by posting to the following discussion group (New Topic button): https://groups.google.com/g/cmed-sharedtask

References

Mahajan, D., Liang, J.J. and Tsou, C.H., 2020. Toward Understanding Clinical Context of Medication Change Events in Clinical Narratives. arXiv preprint arXiv:2011.08835.