CONCERN Early Warning System (EWS):
Communicating Narrative Concerns Entered by RNs
The CONCERN Early Warning System (EWS) has shown that nursing surveillance patterns are a valuable signal to predict and successfully prevent deterioration of hospitalized patients 2 days earlier than other EWS. CONCERN EWS is based on nursing surveillance patterns in acute care units and intensive care units. A multisite, cluster randomized controlled trial showed that hospitalized patients whose care team were exposed to CONCERN EWS had a 35.6% decreased risk of mortality, 11.2% decreased LOS, 7.5% decreased risk of sepsis, and 24.9% increased risk of unanticipated ICU transfers compared with usual-care. Our team developed a CONCERN Toolkit to support implementations at new health systems. Current CONCERN EWS research is focused on expanding to additional sites, implementation
science evaluations, and expansion of the predictive model to new patient populations, including inpatient pediatric setting and emergency department setting.
Nature Medicine Publication: Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial
Contact Email: CONCERN-EWS@cumc.columbia.edu
CONCERN GRANTS and AIMS
CONCERN GRANTS and AIMS
NINR CONCERN 2.0 (4/25/2023 – 1/31/2027)
Title: Communicating Narrative Concerns Entered by RNs (CONCERN)
Funder: National Institute of Nursing Research (NINR)
Grant number: 5R01NR016941 – 10
MPI: Rossetti (contact) (Other MPI: Cato)
Grant overview: In this renewal R01, we will conduct an observational implementation study across 5 health systems to evaluate the algorithmic fairness and generalizability of the CONCERN predictive model and clinical decision support intervention. We will also explore how CONCERN could leverage pediatric caregiver input in real-time to improve the CONCERN predictive model. In this project, we will also research how to improve the fairness of the artificial intelligence (AI) based CONCERN model for patients.
Specific Aims:
Aim 1: Translate the CONCERN-PM to 2 additional specialty patient populations (emergency department and inpatient pediatrics) to evaluate model calibration and performance across different patient populations and health systems.
Aim 2: Evaluate CONCERN-CDS implementation across all adult inpatient units at 5 health systems (CUIMC, MGB, VUMC, WUMC, AMC), in inpatient pediatric units at 3 health systems (CUIMC, AMC, and VUMC) and in the emergency department at 2 health systems (CUIMC and AMC) to understand health system and patient population characteristics associated with a successful implementation using RE-AIM and CFIR Frameworks.
Aim 3: Using a patient-centered design approach, explore the creation of a patient-facing CONCERN-PM and CDS for caregivers of inpatient pediatric patients, to compare predictive model performance with additional caregiver feedback to the clinician-only data-driven model.
ANF (2/1/2022 – 1/31/2025)
Title: CONCERN Implementation Toolkit: Advancing technology-enabled nursing expertise and equitable predictions
Funder: American Nurses Foundation (ANF), The Reimagining Nursing Initiative
MPI: Rossetti (Other MPI: Cato)
Grant overview: This 3-year study will develop and validate the CONCERN implementation toolkit and spread the CONCERN clinical decision support tool to 7 other hospitals with diverse patient and nurse populations. We will also evaluate for biases in EHR data that impact the equitable use of the CONCERN implementation and develop scalable solutions to mitigate those biases. Evaluation metrics are based on the RE-AIM framework and will guide the optimization of the toolkit, cost-benefit analyses, and opportunity for further spread.
Specific Aims:
Aim 1: Develop and validate the CONCERN Early Warning Score Implementation toolkit with all key stakeholders (e.g., nurses, physicians, hospital leadership, information technology analysts) involved in its implementation at current sites. Anticipated toolkit components are:
- Readiness assessment tools
- Nursing practice and documentation tools
- Technical Implementation tools
- Governance tools
Meet The CONCERN Team
CONCERN Team Members
Columbia University
• Sarah Rossetti, PhD, RN
• Haomiao Jia, PhD
• Yu-Ching (Katie) Yang, PhD, RN
• Rachel Lee, PhD, RN
• Brandon Lau
• Temmi Daramola
University of Pennsylvania
• Kenrick Cato, PhD, RN
Emory University
• Patricia Dykes, PhD, MA, RN
Mass General Brigham
• Min Jeoung Kang, PhD
• Sandy Cho, MPH, BSN, RN-BC
Washington University
• Po-Yin Yen, RN, FAMIA, FAAN
• Albert Lai, PhD, FACMI, FAMIA
• Adam Wilcox, PhD, FACMI
• Lisa Kidin, PhD, RN
• Michele Butkiewicz, MSN, RN
Vanderbilt University
• Catherine Ivory, PhD, RN-BC, RNC-OB, FAAN
• Brian Douthit, PhD, RN-BC
• Alvin Jeffery, PhD, RN
University of Colorado
• David Albers, PhD
• Samantha Stonbraker, PhD, RN
• Christina Baker, PhD, RN
Advisory Board
Informatics Experts
• Suzanne Bakken, PhD, RN, FAAN, FACMI – Columbia University
• David W. Bates, MD, MS, FACMI – Brigham and Women’s Hospital
• Bonnie L. Westra, PhD, RN, FAAN, FACMI – University of Minnesota
Clinical Nursing Subject Matter Experts
NYP Site
• Monika Tukacs, BSN, RN, CCRN
• Robert Schroeder, RN
• Amy Moynihan, RN
• Colleen Schneiderman, RN
• Jennifer Osborne, RN
• Leo Rotter, BSN, RN
• Amy Moynihan, RN
MGB Site
• Sarah Beth Thomas, BSN, RN
NWH Site
• Hailey Poole, RN
- Model calibration and bias mitigation tools
- Evaluation measure tools
- Sustainability and Spread tools
Aim 2: Spread the CONCERN predictive model and CDS intervention using the CONCERN toolkit to three other healthcare systems: remaining hospitals within Mass General Brigham; Vanderbilt University Medical Center; and Washington University Medical Campus.
Aim 3: Apply the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework to evaluate and refine the CONCERN toolkit by analyzing its perceived usefulness from key stakeholders, its cost-benefit analysis, and its impact on nursing practice and patient outcomes.
- Sub-aim 3a: Evaluate how variability in nursing practice and documentation influences the equitable use of CONCERN and other early warning scores.
NINR CONCERN 1.0 (2017-2023)
Title: Communicating Narrative Concerns Entered by RNs (CONCERN)
Funder: National Institute of Nursing Research (NINR)
Grant number: 5R01NR016941-05
MPI: Rossetti (contact) (Other MPI: Cato)
Grant overview: The aim of this project is to design and evaluate Electronic Health Record (EHR) clinical decision support (CDS) tools across two large academic medical centers that expose to physicians and nurses our new predictive data source from nursing documentation to increase care team situational awareness of at-risk patient to decrease mortality and 30-day readmissions.
Specific Aims:
Aim 1. Perform analytics of existing nursing data and documentation patterns to confirm predictive factors and notification thresholds for patients at risk of adverse outcomes in the hospital
- Natural Language Processing, Machine Learning, Predictive analytics
Aim 2. User-centered design and testing of CONCERN SMART App; Prototype development and simulation testing
Aim 3. Implementation and evaluation of the impact of the CONCERN SMART App on patient outcomes
- Primary outcomes: in-hospital mortality and length of stay
- Secondary outcomes: cardiac arrest, unanticipated transfers to the intensive care unit, and 30-day hospital readmission rates.
Video: About The CONCERN Initiative
CONCERN Journal Publications/Abstracts
CONCERN Journal Publications/Papers
- Rossetti SC, Dykes PC, Knaplund C, Cho S, Withall J, Lowenthal G, Albers D, Lee RY, Jia H, Bakken S, Kang MJ, Chang FY, Zhou L, Bates DW, Daramola T, Liu F, Schwartz-Dillard J, Tran M, Bokhari SMA, Thate J, Cato KD. Real-time surveillance system for patient deterioration: a pragmatic cluster-randomized controlled trial. Nat Med. 2025 Jun;31(6):1895-1902. doi:10.1038/s41591-025-03609-7.
This pragmatic cluster-randomized controlled trial evaluated the CONCERN Early Warning System, which uses real-time nursing surveillance documentation patterns to identify deterioration risk. Across over 60,000 encounters in 4 hospitals, intervention encounters showed decreased instantaneous risk of death, decreased length of stay, decreased sepsis risk, and increased unanticipated ICU transfer compared with usual care.
- Rossetti S, Cato K. Nursing surveillance from invisible to measurable to indispensable: the CONCERN early warning system trial. Nurs Econ. 2025;43(5):253-258. https://doi.org/10.62116/NEC.2025.43.5.253.
This article positions nursing surveillance as an essential but historically invisible form of clinical expertise, arguing that AI systems should be designed to amplify nurses’ surveillance work, reduce low-value data-entry tasks, and make nursing expertise indispensable to patient safety, workforce sustainability, and health system value.
- Wan YJ, Abdelrahman SE, Facelli JC, Madaras-Kelly K, Kawamoto K, Dishman D, Himes SR, Cato K, Rossetti SC, Del Fiol G. Conceptual framework for prediction models of patient deterioration based on nursing documentation patterns: reproducibility and generalizability with a large number of hospitals across the United States. J Biomed Inform. 2025 Sep;169:104887. doi:10.1016/j.jbi.2025.104887.
This study was an external and independent validation of the predictive modeling approach from in the CONCERN Early Warning System using the eICU-CRD dataset from 208 U.S. hospitals. Results demonstrate reproducibility and generalizability beyond the original health systems to predict in-hospital deterioration from nursing documentation-pattern-derived signals.
- Wan YJ, Abdelrahman SE, Facelli JC, Madaras-Kelly K, Kawamoto K, Dishman D, Cato K, Rossetti SC, Del Fiol G. Enhancing prediction of inpatient deterioration by combining clinical and nurse concern features, with or without temporal clustering. JAMIA Open 2026 (in press).
This study evaluated the use of healthcare process modeling signals plus clinical values for early warning systems identifying potential value in predictions that leverage both types of signals.
- Lee RY, Cato KD, Dykes PC, Lowenthal G, Cho S, Jia H, Daramola T, Tuteja S, Rossetti SC. Earlier ICU transfer after CONCERN early warning system score escalation reduced sepsis-related mortality: results from a multi-site pragmatic cluster randomized controlled trial. AMIA Annu Symp Proc. 2025 May 22;2024:705-714. PMID: 41726485; PMCID: PMC12919569. (Recipient of the Martin Epstein award (1st place))
This post-hoc analysis examined whether the timing of ICU transfer after CONCERN score escalation was associated with in-hospital mortality among patients who later developed sepsis. In 54 patients with CONCERN score changes before unanticipated ICU transfer, shorter time from score escalation to ICU transfer was significantly associated with lower odds of in-hospital death. A 36-hour threshold emerged as a potential inflection point; all patients transferred after this time interval died in the hospital.
- Lee RY, Cato KD, Dykes PC, Lowenthal G, Jia H, Daramola T, Rossetti SC. Evaluating Equity in Usage and Effectiveness of the CONCERN Early Warning System. Appl Clin Inform. 2025 Aug;16(4):838-847. doi: 10.1055/a-2630-4192. Epub 2025 Jun 10. PMID: 40494563; PMCID: PMC12349966.
This paper examines equity in the usage and effectiveness of CONCERN EWS. Usage did not significantly differ by patients’ demographic characteristics, suggesting equitable usage. Patients whose primary language was not English experienced a greater reduction of mortality risk compared to patients whose primary language was English (adjusted HR = 0.419, 95% CI: 0.287-0.610, p = 0.0082). Ongoing evaluations are needed to understand the observed variability and ensure responsible implementation.
- Lee RY, Cato KD, Dykes PC, Lowenthal G, Tuteja S, Rossetti SC. Understanding clinicians’ usage patterns of the CONCERN early warning system: insights from a multi-site pragmatic cluster randomized controlled trial. Stud Health Technol Inform. 2025 Aug 7;329:1256-1260. doi:10.3233/SHTI251040.
This study analyzed clinician engagement with the CONCERN detailed display in a multisite pragmatic cluster randomized trial. The authors used display launches as a proxy for active engagement and examined patterns by clinician role, care setting, and shift.
- Lee Y, Kang MJ, Baris VK, Lowenthal G, Rossetti SC, Cato KD, Lee RY, Kramer J, Huffam R, Dykes PC. Exploring the implementation experience and use of CONCERN early warning system in a rural community hospital: a mixed method convergent approach. AMIA Annu Symp Proc. 2025 May 22;2024:724-733. PMID: 41726452; PMCID: PMC12919451.
This mixed-method convergent study evaluated implementation and use of CONCERN EWS in a rural community hospital. It combined qualitative nurse interviews, quantitative data analysis, and clinical record review to assess user experience, system utilization and accuracy in identifying patient deterioration. Findings suggest that CONCERN EWS is usable in the context of busy acute care nursing workflows and is accurate with overall 97% score agreement with expert chart review across 265 patients.
- Kang MJ, Rossetti SC, Lowenthal G, Knaplund C, Zhou L, Schnock KO, Cato KD, Dykes PC. Designing and testing clinical simulations of an early warning system for implementation in acute care settings. J Am Med Inform Assoc Open. 2024 Oct 16;7(4):ooae092. doi:10.1093/jamiaopen/ooae092. PMID: 39415945; PMCID: PMC11483109.
This JAMIA Open article developed and tested a standardized clinical simulation process for evaluating a CDS system prototype before implementation. The simulation approach was designed to assess interpretability and feasibility from the end-user perspective. Six simulation sessions with 17 nurses were held to investigate how nurses interact with the CONCERN Smart application and findings were used to inform implementation strategies.
- Lee RY, Cato KD, Dykes PC, Lowenthal G, Withall JB, Cho S, Jia H, Rossetti SC. Influence of the CONCERN early warning system on unanticipated ICU transfers, in-hospital mortality, and length of stay: results from a multi-site pragmatic randomized controlled clinical trial. AMIA Annu Symp Proc. 2025 May 22;2024:655-663. PMID: 40417527; PMCID: PMC12099359. (Recipient of Homer R. Warner Award for the AMIA Annual Symposium)
This AMIA Symposium paper evaluated CONCERN EWS in relation to unanticipated ICU transfers, in-hospital mortality, and length of stay in a multisite pragmatic randomized clinical trial. In a retrospective cohort of 1,013 hospital encounters with unanticipated ICU transfers from acute care units with CONCERN EWS had a lower in-hospital mortality rate and a shorter average hospital stay than those transferred from units receiving usual care. These results suggest that CONCERN EWS enhances shared situational awareness for care teams, improves communication, and effectively facilitates timely interventions, thereby streamlining care processes and improving patient outcomes.
- Lee RY, Knaplund C, Withall J, Bokhari SM, Cato KD, Rossetti SC. Variability in Nursing Documentation Patterns across Patients’ Hospital Stays. AMIA Annu Symp Proc. 2024 Jan 11;2023:1037-1046. PMID: 38222368; PMCID: PMC10785899.
This study examined variability in nursing documentation patterns across acute care and ICU hospital stays. It focused on vital signs and note documentation and found that documentation varied by documentation type, stay trajectory, and patient comorbidity.
- Moy AJ, Hobensack M, Marshall K, Vawdrey DK, Kim EY, Cato KD, Rossetti SC. Understanding the perceived role of electronic health records and workflow fragmentation on clinician documentation burden in emergency departments. J Am Med Inform Assoc. 2023 Apr 19;30(5):797-808. doi:10.1093/jamia/ocad038. PMID: 36905604; PMCID: PMC10114050.
This qualitative study investigated how emergency department clinicians perceived EHR factors and workflow fragmentation as contributors to documentation burden. The authors interviewed U.S. ED prescribing providers and nurses. Six themes were identified related to EHR factors perceived to contribute to documentation burden including lack of advanced EHR capabilities, absence of EHR optimization for clinicians, poor user interface design, hindered communication, increased manual work, and added workflow blockages, and five themes associated with cognitive load.
- Hu Y, Cato KD, Chan CW, Dong J, Gavin N, Rossetti SC, Chang BP. Use of real-time information to predict future arrivals in the emergency department. Ann Emerg Med. 2023 Jun;81(6):728-737. doi:10.1016/j.annemergmed.2022.11.005. PMID: 36669911.
This Annals of Emergency Medicine study developed prediction models for shift-level emergency department patient volume to support prediction-driven staffing. It evaluated the predictive value of rich real-time information and compared approaches including linear regression, regression trees, extreme gradient boosting, and time-series models.
- Schwartz JM, George M, Rossetti SC, Dykes PC, Minshall SR, Lucas E, Cato KD. Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study. JMIR Hum Factors. 2022 May 12;9(2):e33960. doi:10.2196/33960. PMID: 35550304; PMCID: PMC9136656.
This qualitative descriptive study explored clinician trust in machine-learning-based predictive CDSSs for in-hospital deterioration. Interviews with 17 clinicians found that perceived understandability, perceived technical competence/accuracy, concordance with clinical impressions, explanations, and training influenced trust.
- Rossetti SC, Dykes PC, Knaplund C, Kang MJ, Schnock K, Garcia JP Jr, Fu LH, Chang F, Thai T, Fred M, Korach TZ, Zhou L, Klann JG, Albers D, Schwartz J, Lowenthal G, Jia H, Liu F, Cato K. The Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) clinical decision support early warning system: protocol for a cluster randomized pragmatic clinical trial. JMIR Res Protoc. 2021 Dec 10;10(12):e30238. doi:10.2196/30238. PMID: 34889766; PMCID: PMC8709914.
This protocol describes a cluster randomized pragmatic trial protocol evaluating CONCERN CDS at two study sites. The trial aimed to assess effectiveness and usability, including outcomes such as in-hospital mortality, length of stay, cardiac arrest, unanticipated ICU transfer, and 30-day readmission.
- Fu LH, Knaplund C, Cato K, Perotte A, Kang MJ, Dykes PC, Albers D, Collins Rossetti S. Utilizing timestamps of longitudinal electronic health record data to classify clinical deterioration events. J Am Med Inform Assoc. 2021 Aug 13;28(9):1955-1963. doi:10.1093/jamia/ocab111. PMID: 34270710; PMCID: PMC8363809.
This retrospective study proposed an algorithm that uses only timestamps from longitudinal EHR data to classify clinical deterioration events. The model used sequences of timestamps from vital signs, flowsheet comments, order entries, and nursing notes, organized into regular time bins.
- Rossetti SC, Knaplund C, Albers D, Dykes PC, Kang MJ, Korach TZ, Zhou L, Schnock K, Garcia J, Schwartz J, Fu LH, Klann JG, Lowenthal G, Cato K. Healthcare process modeling to phenotype clinician behaviors for exploiting the signal gain of clinical expertise (HPM-ExpertSignals): development and evaluation of a conceptual framework. J Am Med Inform Assoc. 2021 Jun 12;28(6):1242-1251. doi:10.1093/jamia/ocab006. PMID: 33624765; PMCID: PMC8200261.
This conceptual framework paper describes HPM-ExpertSignals, which treats clinician behaviors captured in clinical information systems as signals of expertise for prediction. The framework uses a three-step approach: identify behavior patterns, interpret them as proxies for decisions/knowledge/expertise, and use them in predictive models associated with outcomes. This is the underlying framework for the CONCERN Early Warning System.
- Schnock KO, Kang MJ, Rossetti SC, Garcia J, Lowenthal G, Knaplund C, Chang F, Albers D, Korach TZ, Zhou L, Klann JG, Cato K, Bates DW, Dykes PC. Identifying nursing documentation patterns associated with patient deterioration and recovery from deterioration in critical and acute care settings. Int J Med Inform. 2021 Sep;153:104525. doi:10.1016/j.ijmedinf.2021.104525. PMID: 34171662; PMCID: PMC8390439.
This study quantified variations in nursing documentation patterns and identified patterns associated with deterioration and recovery in critical and acute care settings. It treated nursing documentation behavior in the EHR as a possible reflection of nurse concern about patient deterioration.
- Kang MJ, Rossetti SC, Knaplund C, Chang FY, Schnock KO, Whalen K, Gesner EJ, Garcia JP, Cato KD, Dykes PC. Nursing documentation variation across different medical facilities within an integrated healthcare system. Comput Inform Nurs. 2021 May 3;39(12):845-850. doi:10.1097/CIN.0000000000000736. PMID: 33935196; PMCID: PMC8720562.
This study examined nursing documentation variation across facility types within a single integrated health system and EHR. It compared flowsheet measures across academic medical centers, community hospitals, and rehabilitation centers to understand how documentation differed by facility and level of care.
- Cato KD, McGrow K, Rossetti SC. Transforming clinical data into wisdom: artificial intelligence implications for nurse leaders. Nurs Manage. 2020 Nov;51(11):24-30. doi:10.1097/01.NUMA.0000719396.83518.d6. PMID: 33086364; PMCID: PMC8018525.
This nursing management article explains implications of artificial intelligence for nurse leaders. It argues that the growth of AI in healthcare requires nurses and nurse leaders to understand AI alongside clinical models, critical thinking, and evidence-based practice.
- Fu LH, Schwartz J, Moy A, Knaplund C, Kang MJ, Schnock KO, Garcia JP, Jia H, Dykes PC, Cato K, Albers D, Rossetti SC. Development and validation of early warning score system: a systematic literature review. J Biomed Inform. 2020 May;105:103410. doi:10.1016/j.jbi.2020.103410. PMID: 32278089; PMCID: PMC7295317.
This systematic review assessed early warning score system development and validation for adult acute care patients. It evaluated reporting quality, summarized development and validation methods, and identified strengths, limitations, and future directions for EWS research. Great variability was observed between models and validation processes. Only 3 of 29 studies validated their model using an external database separate from the derived database
- Rossetti SC, Knaplund C, Albers D, Tariq A, Tang K, Vawdrey D, Yip NH, Dykes PC, Klann JG, Kang MJ, Garcia J, Fu LH, Schnock K, Cato K. Leveraging clinical expertise as a feature—not an outcome—of predictive models: evaluation of an early warning system use case. AMIA Annu Symp Proc. 2020 Mar 4;2019:323-332. PMID: 32308825; PMCID: PMC7153052.
This AMIA paper evaluated an early warning system use case that treats clinical expertise as a predictive feature rather than merely an outcome. It frames nursing documentation and clinician expertise as actionable signals for earlier detection of hospital deterioration.
- Korach ZT, Cato KD, Collins SA, Kang MJ, Knaplund C, Dykes PC, Wang L, Schnock KO, Garcia JP, Jia H, Chang F, Schwartz JM, Zhou L. Unsupervised machine learning of topics documented by nurses about hospitalized patients prior to a rapid-response event. Appl Clin Inform. 2019 Oct;10(5):952-963. doi:10.1055/s-0039-3401814. PMID: 31853936; PMCID: PMC6920051.
This study used unsupervised machine learning to discover rapid-response event risk or protective factors from unstructured nursing notes. Topic modeling was applied to notes from non-ICU hospitalized patients, and nursing experts mapped topics to SNOMED CT concepts.
- Korach ZT, Yang J, Rossetti SC, Cato KD, Kang MJ, Knaplund C, Schnock KO, Garcia JP, Jia H, Schwartz JM, Zhou L. Mining clinical phrases from nursing notes to discover risk factors of patient deterioration. Int J Med Inform. 2020 Mar;135:104053. doi:10.1016/j.ijmedinf.2019.104053. PMID: 31884312; PMCID: PMC7103062.
This study developed an unsupervised, data-driven method to mine clinical phrases from nursing notes as potential risk factors for rapid response events. It was motivated by the need to use nurses’ narrative clinical judgment for earlier recognition of patient deterioration.
- Kang MJ, Dykes PC, Korach TZ, Zhou L, Schnock KO, Thate J, Whalen K, Jia H, Schwartz J, Garcia JP, Knaplund C, Cato KD, Rossetti SC. Identifying nurses’ concern concepts about patient deterioration using a standard nursing terminology. Int J Med Inform. 2020 Jan;133:104016. doi:10.1016/j.ijmedinf.2019.104016. PMID: 31707264; PMCID: PMC6957124.
This study used standard nursing terminology and subject-matter expert input to identify nurse concern concepts related to patient deterioration. A total of 29 CCC concepts were selected as reflecting nurse concerns. From these, 111 entities and 586 seed terms were generated into a fundamental lexicon. The resulting terminology work is intended to support downstream automated tasks such as natural language processing and risk prediction.
- Garcia JP Jr, Collins SA, Cato KD, Bakken S, Jia H, Kang MJ, Knaplund C, Schnock KO, Dykes PC. CONCERN Factorial Design Survey (FDS) methods test: using REDCap as a survey platform. Stud Health Technol Inform. 2019 Aug 21;264:1462-1463. doi:10.3233/SHTI190485. PMID: 31438182; PMCID: PMC7020103.
This methods paper tested REDCap as a platform for a factorial design survey to inform user-centered design. Because REDCap lacked some randomization and automation functions, the authors created a copy-and-hide workaround that configured 42 surveys in approximately three hours.
- Collins S, Couture B, Kang MJ, Dykes P, Schnock K, Knaplund C, Chang F, Cato K. Quantifying and visualizing nursing flowsheet documentation burden in acute and critical care. AMIA Annu Symp Proc. 2018 Dec 5;2018:348-357. PMID: 30815074; PMCID: PMC6371331.
This AMIA paper quantified EHR flowsheet documentation burden in acute and critical care nursing. It reported hundreds of manual flowsheet entries per 12-hour shift and described documentation rates that averaged roughly one data point every 0.82 to 1.14 minutes for nurses.
- Collins SA, Cato K, Albers D, Scott K, Stetson PD, Bakken S, Vawdrey DK. Relationship between nursing documentation and patients’ mortality. Am J Crit Care. 2013 Jul;22(4):306-13. doi:10.4037/ajcc2013426. PMID: 23817819; PMCID: PMC3771321.
This study used data-mining methods to analyze electronic nursing documentation in relation to patient mortality. It used data from 15,000 acute care patients and 145 cardiac arrest patients to identify statistically significant patterns in optional flowsheet comments and vital-sign documentation to mortality risk. Patients who died had a mean of 0.9 to 1.5 more optional comments and 6.1 to 10 more vital signs documented within 48 hours than did patients who survived. These foundational findings support the idea that documentation behavior can signal patient deterioration.
- Collins SA, Vawdrey DK. “Reading between the lines” of flow sheet data: nurses’ optional documentation associated with cardiac arrest outcomes. Appl Nurs Res. 2011 Nov 12;25(4):251-7. doi:10.1016/j.apnr.2011.06.002. PMID: 22079746; PMCID: PMC3288795.
This study analyzed electronic flowsheet comments and documentation patterns for 201 cardiac arrest patients. It found that optional free-text comments had a statistically significant association to abnormal clinical measurements and that patterns of optional documentation were related to outcomes with 3 or more comments indicating statistically significant greater likelihood of dying by discharge. Documentation of ICU vital signs greater than the minimum hourly requirement was associated with increased survival of a cardiac arrest (p<0.05). These findings serve as foundational evidence for the development of the CONCERN Early Warning System.
CONCERN Conference Abstracts
- Rossetti SC, Dykes PC, Withall J, Lowenthal G, Knaplund C, Lee RY, Hobensack M, Albers D, Daramola T, Ivory C, Yen P, Douthit B, Kang M, Chang FY, Bokhari M, Tran M, Cato K. Readying AI-Based CONCERN Early Warning System for Multi-Site Deployments. System Demo at AMIA Informatics Summit, March 2024, Boston, MA. 2024 AMIA Informatics Summit. Boston, MA.
- Lee, R.Y., Cato, K.D., Dykes, P.C., Lowenthal, G., Withall, J.B., Bokhari, M.A.B., Tran, M., Cho, S., Jia, H., Rossetti, S.C. (2024). Early alerts, timely actions: evaluating the influence of the concern early warning system on unanticipated ICU transfers and sepsis risks. 2024 AMIA Informatics Summit. Boston.[Poster Presentation].
- Rossetti, S.C., Cato, K.C. Development, Optimization, and Evaluation of CONCERN Early Warning Score in Multi-site Clinical Trial. 2023. AMIA AI Showcase podium presentation at AMIA 2023 Annual Symposium; 2023 Nov 11-15; New Orleans, LA.
- Lee, R.Y., Knaplund, C., Withall, J.,Bokhari, S.M., Cato,K.C., Rossetti, S.C. Variability in Nursing Documentation Patterns across Patients’ Hospital Stays. 2023. Podium presentation at AMIA 2023 Annual Symposium; 2023 Nov 11-15; New Orleans, LA.
- Withall, J., Tran, M., Schroeder, B., Lee, R. Y., Moy, A.J., Bokhari, S.M., Cato, K.C., Rossetti, S.C. Identifying Reuse and Redundancies in Respiratory Flowsheet Documentation: Implications for Clinician Documentation Burden. 2023. Presentation at AMIA 2023 Annual Symposium; 2023 Nov 11-15; New Orleans, LA.
- Hobensack, M., Knaplund, C., Jia H., Ruan, E., Soroush, A., Rossetti, S.C., Cato ,K.C. Nursing Surveillance Risk Scores Preceding Psychological Deterioration. 2023. Poster presentation at AMIA 2023 Annual Symposium; 2023 Nov 11-15; New Orleans, LA.
- Moy, A.J., Withall, J., Kim, E., Cato, K.C., Rossetti, S.C. A Computational Framework to Evaluate Emergency Department Clinician Task Switching in the Electronic Health Record Using Event Logs. Presentation at AMIA 2023 Annual Symposium; 2023 Nov 11-15; New Orleans, LA.
- Moy, A.J., Kim, E., Cato, K.C., Tatonetti, N., Withall, J., Rossetti, S.C. Extending the Use of Time Series Clustering to Detect Emergency Department Shifts from Electronic Health Record Audit Logs to Physicians. Presentation at AMIA 2023 Annual Symposium; 2023 Nov 11-15; New Orleans, LA.
- Rossetti, S.C. Adler-Milstein, J., Kannampallil, T., Moy, A.J., Murphy, J., Measuring Long Term Impact on Health Professionals’ Documentation Burden Reduction: 25×5 Task Force Logic Model, Mixed-Methods, and Innovative Approaches. Panel presentation at AMIA 2023 Annual Symposium; 2023 Nov 11-15; New Orleans, LA.
- Hobensack, M., Withall, J., Cato, K., Dykes, P., Lowenthal, G., Cho, S., Ivory, C., Yen, P.Y., Rossetti, S. (July 2023). Understanding the technical implementation of a clinical decision support: A qualitative analysis. [Poster Presentation]. MedInfo2023, Sydney, Australia.
- Von Gerich, H., Dowding, D., Hobensack, M., Peltonen, L-M, Sequiera, L., Withall, J. (July 2023) Learning from experts on the evolution of decisions support systems – a career progression panel. [Panel Presentation]. MedInfo2023, Sydney, Australia.
- Hobensack, M., Withall, J., Dykes, P.C., Lowenthal, G., Rossetti, S., Cato, K. (May, 2023). Key takeaways from technical experts on the implementation of a clinical decision support smart app: A qualitative analysis. [Podium Presentation]. AMIA Clinical Informatics Conference, Chicago, IL.
- Withall, J., Hobensack, M., Rossetti, S., Dykes, P.C., Cato, K. (May, 2023). Implementation of a Nurse-Driven Early Warning System on Inpatient Clinical Nursing Units: Lessons Learned from a Pragmatic Clinical Trial. [Poster Presentation]. AMIA Clinical Informatics Conference, Chicago, IL.
- Yen, P. Fast Healthcare Interoperability Resources (FHIR) for CONCERN – technical discussion; Barnes-Jewish Hospital Clinical Decision Support; March 27, 2023
- Rita Kobb Nursing & Health Informatics Symposium on 2/24/23, https://nursing.ufl.edu/2023-rita-kobb-nursing-and-health-informatics-symposium-speakers/.
- Ivory, C. Vanderbilt University Medical Center Executive Nursing Committee, 2022
- Cato,K. Harnessing Clinical Nursing Data for Predictive Analytics and Decision Support in the Hospital Setting. Veteran’s Administration Nursing Informatics Continuing Education Series. December 2022
- Yen, P. CONCERN Implementation – nurse leaders; Barnes-Jewish Hospital Interprofessional Practice, Education, Innovation and Research, November 11, 2022
- Chae S, Song J, Barrón Y, Bowles KH, McDonald MV, Rossetti SC, Cato KD, Hobensack M, Evans L, Topaz M. Heart Failure Patient Characteristics and Symptoms Documented in Home Health Care Clinical Notes Associated with Emergency Department Visits and Hospitalizations. Podium presentation at the AMIA 2022 Annual Symposium; Nov 5-9, 2022; Washington, DC.
- Moy AJ, Withall J, Hobensack M, Lee RY, Levy D, Rosenbloom ST, Rossetti SC, Johnson KB, Cato KC. Using Topic Modeling to Elicit Insights from the 25×5 Symposium to Reduce Documentation Burden Chat Logs. Podium presentation at the AMIA 2022 Annual Symposium; Nov 5-9, 2022; Washington, DC.
- Rossetti SC, Dykes P, Sengstack P, Murphy J. Technology Will Resolve the Nursing Workforce Shortage Within 5 Years. Informatics Debate at the AMIA 2022 Annual Symposium; Nov 5-9, 2022; Washington, DC.
- Taylor BN, Rossetti SC, Cato KD. Deterioration Events in Patients With Depression. Poster presentation at the AMIA 2022 Annual Symposium; Nov 5-9, 2022; Washington, DC.
- Lee RY, Rossetti SC, Cato KD. Differences in Frequencies of Nursing Flowsheet Documentation by Patients’ Primary Language. Poster presentation at the AMIA 2022 Annual Symposium; Nov 5-9, 2022; Washington, DC.
- Lee, R.Y., Rossetti, S.C., Cato, K.D. Differences in frequencies of nursing flowsheet documentation by patients’ primary language. Poster session to be presented at: AMIA 2022 Annual Symposium; 2022 Nov 5-9; Washington D.C.
- Taylor, B.N, Collins Rossetti, S., & Cato, K.D. (2022). Deterioration Events in Patients With Depression. Poster Presentation at the AMIA 2022 Annual Symposium; October 2022; Washington, DC.
- Yen, P. CONCERN Implementation – obtaining approval; Washington University/Barnes-Jewish Hospital Research Steering Committee; September 23, 2022
- Yen, P. Fast Healthcare Interoperability Resources (FHIR) for CONCERN – technical discussion; Barnes-Jewish Hospital Clinical Decision Support; August 16, 2022
- Cato, K. Modeling Clinician Behavior to Support Clinical Decision Making and Improve Patient Outcomes: The CONCERN Study. Dynamics and Data Assimilation, Physiology and Bioinformatics: Mathematics at the Interface of Theory and Clinical Application. 2022. Banff International Research Station for Mathematical Innovation and Discovery. Banff, Canada
- Rossetti S, Albers D, Dykes P, Sengstack P, Del Fiol G. 2022. Informatics Research and Implementation During COVID: Challenges, Opportunities and Recommendations for Building a Sustainable Infrastructure. AMIA Annual Symposium, Washington, DC.
- Cato, K. 2021. Articulating Judgment Into AI: The CONCERN Study. NINR AI Boot Camp – Clinical Applications. Virtual Meeting
- Cato, K. What the CONCERN Study Has Taught Me about Racial Bias in Nursing Workflow. 2021. New England HIMSS 2021 Annual Spring Conference. Podium Presentation. Boston, MA
- Schwartz, J., Knapland, C., Withall, J., Rossetti, S.C., Kang, M-J, Dykes, P., Schnock, K.O., Zhou, L., Lucas, E., Minshall, S., Cato, K. Clinicians’ perspectives on factors influencing trust in machine-learning-based predictive clinical decision support. AMIA CIC 2021
- Fu, L-H., Knaplund, C., Cato, K. Albers, D., Rossetti, SC. Utilizing Timestamps of Longitudinal Data from Electronic Health Record to Predict Clinical Deterioration Events. AMIA 2020 Annual Symposium, Virtual Conference, Nov 14-18 2020
- Schwartz J, Knaplund C, Rossetti SC, Kang MJ, Dykes P, Korach TZ, Schnock K, Zhou L, Cato KD. Nurse Use of Vital Sign Flowsheet Comments: Shared or Individual Practice? Poster session presented at: 2020 American Medical Informatics Association (AMIA) Annual Symposium; 2020 Nov 14-18; Virtual Conference
- Dykes, P., Rossetti, SC., Cato, K., Dinov, I. Nursing Science: Stronger Together. Columbia University and University of Michigan Schools of Nursing and the Center for Nursing Excellence at Brigham and Women’s Hospital. Virtual Panel. Jul 14 2020
- Cato K., Albers, J., Freeman, R., Hoeksema, L., Womack, D. Jeffery, A. Bench to Bedside: The Power of Nursing Data for Prediction. 2020, Annual Meeting American Medical Informatics Association, Pre-Conference All-Day Workshop. Virtual Meeting.
- Rossetti, S., Moy, A., Kang, M. Schwartz, J., Cato K., Mixed-Methods Approaches to Understanding, Measuring, and Reducing Clinical Documentation Burden. 2020, Annual Meeting American Medical Informatics Association, Podium Presentation. Virtual Meeting.
- J. Schwartz and K. Cato, Machine Learning Based Clinical Decision Support and Clinician Trust, 2020 IEEE International Conference on Healthcare Informatics (ICHI)
- Garcia J, Collins S, Cato K, et al. CONCERN factorial design survey (FDS) methods test: Using REDCap as a survey platform. Poster Presentation at MedInfo World Congress 2019 Aug 25-30, Lyon, France
- Kang MJ, Zhou L Chang FY, Knaplund C, Garcia JP, Cato KD, Rossetti SC, Dykes P. 2019. Methodology of Sepsis Prognosis Prediction Model Tailored Clinical Practice. Podium abstract presentation at MedInfo 2019 World Congress of Medical Health Informatics; 2019 Aug 25-30; Lyon, France.
- Cato K, Knaplund C, Klann J, Dykes P, Albers D, Kang MJ, Schnock K, Zhou L, Korach T, Collins S. 2019. Nursing medication administration data quality: assessment for reuse in analytics. Presentation at AMIA 2019 Clinical Informatics Conference; 2019 Apr 30-May 2; Atlanta, GA.
- Kang MJ, Zhou L, Chang F, Knaplund C, Garcia JP, Cato K, Collins S, Dykes P. 2019. Methodology of Sepsis Prognosis Prediction Model Tailored Clinical Practice. Presentation at AMIA 2019 Clinical Informatics Conference; 2019 Apr 30-May 2; Atlanta, GA
- Klann J, Kang MJ, Schnock K, Knaplund C, Dykes P, Collins S, Cato K. 2019. A Data Warehousing Design for Multi-Domain Inpatient Nursing Data in i2b2. Abstract at AMIA 2019 Clinical Informatics Conference; 2019 Apr 30-May 2; Atlanta, GA.
- Korach ZT, Collins SA, Cato KD, Kang MJ, Schnock KO, Couture B, Knaplund C, Whalen K., Thate J, Dykes P, Zhou L. 2019. Active Learning for the Identification of Nurses’ Concerns from Nursing Notes. Presentation at AMIA 2019 Clinical Informatics Conference; 2019 Apr 30-May 2; Atlanta, GA.
- Korach T, Cato K, Collins S, Kang MJ, Knaplund C, Dykes P, Wang L, Schnock K, Garcia JP, Jia H, Chang F, Schwartz J, Zhou L. 2019. Unsupervised-learning of concern topics documented by nurses about hospitalized patients prior to a rapid-response event. Presentation at AMIA 2019 Clinical Informatics Conference; 2019 Apr 30-May 2; Atlanta, GA.
- Cato K, Knaplund C, Klann J, Dykes P, Albers D, Kang MJ, Schnock K, Zhou L, Korach T, Collins S. 2019. Signals of Nurse Workarounds in Electronic Medication Administration Record Data. Abstract at AMIA 2019 Informatics Summits; 2019 Mar 25-28; San Francisco, CA.
- Collins S, Dykes P, Albers D, Klann J, Kang MJ, Schnock K, Zhou L, Korach T, Knaplund C, Cato K. 2019. EHR Surveillance Patterns to Compute when A “Patient does not look right”. Ignite Talk Abstract at AMIA 2019 Informatics Summits; 2019 Mar 25-28; San Francisco, CA.
- Kang MJ, Dykes P, Schnock KO, Korach ZT, Zhou L, Thate J, Whalen K, Garcia JP, Knaplund C, Cato KD, Collins SA. 2019. Identifying and Grading Nurses’ Concern Concepts about Patient Deterioration Using a Standard Nursing Terminology. Podium abstract at AMIA 2019 Informatics Summits; 2019 Mar 25-28; San Francisco, CA.
- Klann JG, Kang MJ, Schnock KO, Knaplund C, Dykes P, Collins SA, Cato KD. 2019. A Data Warehousing Design for Multi-Domain Inpatient Nursing Data in i2b2. Ignite Talk Abstract at AMIA 2019 Informatics Summits; 2019 Mar 25-28; San Francisco, CA.
- Schnock KO, Kang MJ, Chang FY, Collins SA, Knaplund C, Garcia JP, Cato KD, Dykes P. 2019. Nursing Flowsheet Elements Across Integrated Health Care System. Podium abstract at AMIA 2019 Informatics Summits; 2019 Mar 25-28; San Francisco, CA.
- Cato, K. CONCERN: A paradigm shift in nursing. 2019. Brocher Foundation Workshop: Artificial Intelligence for Nursing: Ethical, Legal, and Social Implications. Podium Presentation. Hermance, Switzerland.
- Cato, K. The CONCERN Study: Analyzing Inpatient Nursing Data to Develop Healthcare Process Models of Clinical Concern. 2019. 55th Annual Isabel Maitland Stewart Conference on Research in Nursing. End-Note Speaker. New York, NY.
- Collins S, Cato K. The CONCERN Study: Clinical Decision Support Communication for Risky Patient States. 2019. New York Academy of Medicine Urban Health Informatics Innovation Conference. Podium Presentation. New York, NY.
- Couture B, Klann J, Cato K, Knaplund C, Kang MJ, Dykes P, Collins S. 2018. Harmonizing Flowsheet Datasets Across EHRs for a Multi-Site Study. Poster at AMIA 2018 Annual Fall Symposium; 2018 Nov 3-7; San Francisco, CA.
- Kang MJ, Dykes P, Korach T, Zhou L, Thate J, Whalen K, Schnock K, Knaplund C, Couture B, Cato K, Collins S. 2018. Identifying Concepts of Nurses’ Concerns Using a Standard Nursing Terminology. Poster at AMIA 2018 Annual Fall Symposium; 2018 Nov 3-7; San Francisco, CA.
- Korach, Z. T., Collins, S. A., Cato, K. D., Kang, J., Kumiko, O., Couture, B., Knaplund, C., Whalen, K., Thate, J., Dykes, P., Zhou, L. Active Learning for the Identification of Nurses’ Concerns from Nursing Notes. AMIA 2018 Annual Fall Symposium; 2018 Nov 3-7; San Francisco, CA.
- Klann, J.G., Collins S, Cato K, Waitman LR, Westra B. 2018. Nursing Documentation and the Clinical Research Informatics Pipeline. Abstract and Panel Presentation at AMIA 2018 Informatics Summits; 2018 March 12-15; San Francisco, CA.
Contact Us
Contact Our Co-Investigators
Sarah Collins Rossetti RN, PhD: sac2125@cumc.columbia.edu
Kenrick Cato RN, PhD: kcato@nursing.upenn.edu