Patient portal messages have become a major part of a clinician’s day. One widely cited estimate puts it at about 85 minutes daily, and much of that work spills into evenings and weekends, a phenomenon known as “pajama time.” Generative AI has been promoted as a fix: tools that draft replies, so clinicians don’t have to start from a blank draft. But does it actually help? 

A new study led by researchers from the Columbia University Department of Biomedical Informatics (DBMI) suggests the answer is nuanced. When clinicians used an AI-drafted reply, patients got their answer about 29% sooner than on the clinician’s other messages. But clinicians used the drafts only about 5% of the time, and overall, the tool was not linked to a reduction in after-hours work.

The study, AI-Generated Draft Response Tool, Message Turnaround Time, and After-Hours Replies Among Health Care Professionals, published Oct. 30 in JAMA Network Open, suggests that generative AI is not an automatic fix, and that how these tools are implemented will be critical to their success.

A JAMA Network Open commentary on this study is available here.

A Fairer Way to Measure the Tool

The team studied Epic’s In-Basket Augmented Response Technology (ART), a generative AI tool built into the electronic health record that drafts replies to patient messages. They analyzed nearly one million patient portal messages answered by more than 1,500 clinicians at Columbia University Irving Medical Center.

Earlier studies often compared messages where clinicians chose to use a draft with messages where they didn’t. That can be misleading, because clinicians are likely to use drafts on simple, routine messages that would have been answered quickly anyway. They compared each clinician’s own messages before and after they started using the tool to account for differences between clinicians’ usual response behavior and adjust for message characteristics to ensure the comparison is fair and not simply because of which messages were more amenable to AI use. As far as the authors know, it is the first analysis of this kind, and the first to look at when replies are sent, not just how fast.

Slower When Adopted But Faster When Used

The results pointed in two directions. After clinicians started using the tool, turnaround time rose by about 8.6%. However, when they did use an AI draft, turnaround was nearly 30% shorter.

The authors say a likely explanation for the slowdown is the added step of reading a draft first, which is wasted effort if the draft doesn’t fit. This idea draws on earlier research and wasn’t directly tested in this study.

“Before the tool was introduced, clinicians directly wrote their replies. With ART, clinicians must read the draft, decide whether to use it, and, if they judge it inadequate, compose their own reply from scratch,” said Vincent Jeanselme, PhD, postdoctoral research scientist in DBMI and lead author of the study. “When clinicians actually use the draft, turnaround is faster because reviewing a draft takes less time than writing it. When inadequate, however, reading the draft slows the process.”

An Early Rollout, and Cautious Drafts

Of the more than 185,000 messages sent after clinicians began using the tool, the AI draft was used in just 9,757, or roughly 5%. Researchers note that the findings reflect an early stage of the technology.

“Our study cannot identify all the reasons for limited adoption because of limited direct feedback from users, but it is important to recognize that these findings reflect an early rollout of the tool,” said senior author Shalmali Joshi, PhD, assistant professor of biomedical informatics at Columbia University. “At that stage, the system had restricted integration with the deeper patient record and, by design, produced conservative drafts to protect patient safety. As the technology matures, both the draft quality and the clinical context available to the model are likely to improve, although whether that translates into greater day-to-day adoption remains an open question.”

The study also found no clear sign that clinicians got faster the longer they used the tool. The authors caution that the study period may have been too short to capture longer-term effects.

What About Pajama Time?

Health systems had hoped AI drafts would reduce clinical burden. The research team proposed to quantify burden through the likelihood of responding after hours. Overall, the study found no reduction in responding after hours.

The conclusions were mixed. For one type of message, test-result questions that arrived during working hours, using the tool was linked to fewer after-hours replies. But for messages that arrived outside working hours, using the tool made an after-hours reply more likely, by about 4.5 percentage points. The authors suggest a draft may make it easier to start a reply, which could make it easier to respond after work hours, though both findings were less clear when working hours were defined more broadly.

“Technologies often bring unintended effects that depend heavily on how they are implemented,” said Joshi. “While after-hours messaging is a clear concern, ‘pajama time’ is only one surface-level proxy for burden. Moving forward, the outcomes that matter even more are changes to clinicians’ actual cognitive workload and the long-term impact on patient experience and care quality. Those are the vital questions informatics research must answer next.”

What Comes Next

Unlocking real gains, the authors suggest, will likely require moving beyond generic text generation toward tools that integrate more intelligently with the healthcare ecosystem.

“Efficiency will likely depend on both the quality of the AI-generated drafts and the amount of relevant context available to the tool—including specific details about the patient, the clinician, and the broader care team,” said Joshi. “Beyond adapting to a clinician’s individual writing style, future systems may also need to account for variability in workflows and message types across specialties and care team roles. Better personalization along these dimensions may lead to greater efficiency gains, but establishing that requires additional research.”

More Information

The study, AI-Generated Draft Response Tool, Message Turnaround Time, and After-Hours Replies Among Health Care Professionals, was published Oct. 30 by JAMA Network Open.

In addition to Jeanselme and Joshi, the study’s authors include DBMI’s Jungmi Han and Karthik Natarajan, along with collaborators from Children’s Hospital of Philadelphia, the University of Pennsylvania (formerly at Columbia), Columbia University Irving Medical Center, and NewYork-Presbyterian Hospital.

The co-authors did the work while they were at Columbia