Innovation and Design Experiments in Academic Surgery (I.D.E.A.S.)

Tejas S. Sathe

Recently, Asim A, MD had a thoughtful post on X about a question he thinks that applicants should ask interviewers in the 2024 Match. This is a very insightful question because of how much the answer reveals about the program. First, the answer will tell you about the practical support systems that exist.

Do you have any questions for me?

Meghal Shah

I embarked on my year of academic development with the goal of making operating less painful for surgeons. The field of surgical ergonomics is nascent, and while there is an appetite for technological solutions, behavioral interventions are one of the only effective (albeit limited) tools available to us right now.

Fumbling at the Five Yard Line

This tool represents my own ideas and is not currently in use or endorsed by any institution. Medical students applying to residency often have to grapple with an incredible amount of information when evaluating prospective training programs. The increased number of applications and interviews inherent to the Zoom era further exacerbates this issue.

The Residency Visual Abstract

Imagine that you want to read a paper. You probably either search the subject area of interest on PubMed or Google Scholar. Perhaps you were directed to the paper from an interesting Twitter thread or other social media site. Or maybe you were assigned the paper for a journal club or weekly educational activity. If you're trying to read the paper from within your institution's network, the process is usually seamless.

Why Surgery Needs I.D.E.A.S.

We are amidst a revolution in artificial intelligence (AI). In this paper, we aim to share our early experience with creating custom AI-powered chatbots in surgical education. While our versions are prototypes, we hypothesize that similar chatbots will one day play a significant role in the field. In brief, AI is the ability of a computer to perform human-like tasks.

How I GPT It: Development of Custom AI Chatbots for Surgical Education

Analysis of Reported Voting Behaviors of US Physicians, 2000-2020

Barriers to Voting in 2020 Among Resident Physicians

As residents within the healthcare profession, our first duty is to care for our patients. While working upwards of 80 hours per week pursuing that mission, it is hard to think of ourselves as anything beyond our job title. But we are also citizens. And while patient care remains a worthy North Star, decisions made by faraway policymakers can impact our patients as much as the assessments and plans we ourselves craft.

Voter Turnout Among General Surgery Residents in the 2022 U.S. Midterm Election

"Do you have an IRB?" is often the first question anyone asks when a new research idea is proposed. It shouldn't be the last. Despite the personal frustrations of surgeons who deal with them, institutional review boards (IRBs) maintain a revered status within our profession. For instance, the perceived necessity of IRB review is so ubiquitous that most readers will know to ask the above question even if they do not know why.

Silencing with Red Tape

One of the most counter-intuitive principles when applying design thinking to healthcare and surgery specifically is the idea that quantity of ideas is initially more important than quality of ideas.

Design Thinking: A Method for Building Creative Immunity

Joseph L'Huillier

This is a preprint of a manuscript currently under peer review.
<strong>
 <strong>
  Abstract
 </strong>
</strong>
Importance: Gossip, defined by social scientists as “evaluative talk about an absent third party,” is anecdotally pervasive, yet poorly understood in surgical residency programs. Objective: This study sought to deconstruct the role of gossip in surgical residency and evaluate its impact through the lens of surgical residents.

The Anatomy of Gossip: Dissecting Dynamics and Impacts in Surgical Residency

Use of Open Science Practices in Surgical Journals

Preprints in Health Professions Education: Raising Awareness and Shifting Culture

The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape

Releasing a preprint is associated with more attention and citations for the peer-reviewed article

The effect of bioRxiv preprints on citations and altmetrics

Hate the peer-review process? Einstein did too

Journal policies and practices

This is a preprint of a perspective paper that has been submitted for peer review. Think about the last time you submitted a paper. After meticulous editing, you uploaded your manuscript and hit the "submit" button. Then, you waited. And waited. The reviewers requested revisions, which you obliged. More waiting. Finally, the paper was accepted and sent for proofing. You were one of the lucky ones.

Preprints: Opening Surgical Science to Greater Innovation and Transparency

Validity: on the meaningful interpretation of assessment data

How Important Are American Board of Surgery In-Training Examination Scores When Applying for Fellowships?

The pediatric surgery match by the numbers: Defining the successful application

Predicting Performance on the American Board of Surgery Qualifying and Certifying Examinations

Using the American Board of Surgery In-Training Examination to Predict Board Certification: A Cautionary Study

Association Between American Board of Surgery In-Training Examination Scores and Resident Performance

Success in pediatric surgery: An updated survey of Program Directors 2020

Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application

Relationships between study habits, burnout, and general surgery resident performance on the American Board of Surgery In-Training Examination

Let's Not Throw the Baby Out with the Bath Water – Keep the ABSITE a Numerically Scored Exam

The Effect of Prior Night Call Status on the American Board of Surgery In-Training Examination Scores: Eight Years of Data From a Single Institution

Effect of resident call status on ABSITE performance

Effect of January Vacations and Prior Night Call Status on Resident ABSITE Performance

Preliminary Development of a Program ABSITE Dashboard (PAD) to Guide Curriculum Innovation

Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents

Holistic Review

Holistic Review — Shaping the Medical Profession One Applicant at a Time

Diversity, Fairness, and Excellence: Three Pillars of Holistic Admissions

This is a preprint of a manuscript currently under peer review. The American Board of Surgery In-Training Examination (ABSITE
<sup>
 ®
</sup>
) is a multiple-choice exam administered yearly to surgical residents. The ABS provides the following information on its website:  "The ABSITE is furnished to program directors as a formative evaluation instrument to assess residents’ progress.

Proposed Reforms to the American Board of Surgery In-Training Examination (ABSITE)

<strong>
 Abstract
</strong>
The institutional knowledge required for residency is vast, complex, and frequently changing. Most institutions do not have a resource that serves as a singular source of truth for this information. Furthermore, since information is mostly presented on static documents such as emails or PDFs, there is no easy way to update and share the information in real time.

An Operating System for Residency

Recently, Drs. Robert Wachter and Erik Brynjolfsson published a viewpoint in the Journal of the American Medical Association JAMA on Generative AI in healthcare. We agree with their views and wished to add the following points in a response letter. While our letter was not accepted, we have pre-printed it here. Please find the original paper and our response below.

Response Letter

Students and residents aspiring for a career in academic surgery are looking for training programs that will help jumpstart their careers by exposing them to mentorship and opportunities to conduct research. While the gold standard of academic productivity (for the time being) remains peer-reviewed publications, conference posters and presentations are an important secondary metric.

How to Build an Academic Powerhouse: Let's Study Who's Doing it

A digital reference guide to sutures and needles (adapted from Twitter Thread)  As a student and junior resident, I didn’t understand how surgeons decided when to use each suture/needle. I tried to pick up on patterns but the info was never in one place.

What's the Stitch?

We asked GPT-4 to pretend that it was an academic surgeon and write an essay on the biggest challenges in academic surgery and how to solve them. Here is an unedited output of what GPT-4 wrote as well as the series of prompts we used to get this output.
<strong>
 Disclaimer:
</strong>
AI large language models (LLM) can confidently state inaccurate information and make up facts.

GPT-4 on the Biggest Challenges in Academic Surgery

Greenhouse Gases Equivalencies Calculator - Calculations and References | US EPA

Abstract
<strong>
 Background
</strong>
Reducing costs and carbon footprint are important, parallel priorities for the U.S. healthcare system. Within surgery, reducing the number of instruments that are sterilized and disposable supplies that are used for each operation can help achieve both goals.

Instrument and Supply Variability

Multiple Views of Participatory Design

Dilemmas in a general theory of planning

5 Examples of Design Thinking in Business | HBS Online

<b>
 <strong>
  Abstract
 </strong>
</b>
The field of surgery faces complex, systemic challenges that will require new academic frameworks. In this paper, we propose design thinking as a useful problem solving technique to apply to such challenges. We define design thinking and provide a brief history of this practice.

Teach Design to Surgical Trainees

I.D.E.A.S.

::: {.kg-card .kg-callout-card .kg-callout-card-grey}
::: {.kg-callout-text}
**Abstract**  
The institutional knowledge required for residency is vast, complex, and
frequently changing. Most institutions do not have a resource that
serves as a singular source of truth for this information. Furthermore,
since information is mostly presented on static documents such as emails
or PDFs, there is no easy way to update and share the information in
real time. In this article, we present our use of the Notion platform to
solve these problems. Using this technology, we created a centralized
knowledge hub for the residency. We use this platform to house an intern
guide, publish service specific workflows and protocols, and track
quality improvement initiatives. While we aim to continually refine the
platform, our experience can serve as a template for other institutions
attempting to create a one-stop shop for resident productivity and
well-being.
:::
:::

## Introduction

Surgical residents are responsible for learning and remembering a vast
amount of information. While much of this information relates to the
science and practice of surgery, a large portion is institutional and
related to process. Most institutions do not maintain this information
in a centralized and accessible format. Though exploratory conversations
and surveys, we found that information was often organized piece-meal
across a combination of paper, email, and cloud platforms such as Google
Drive, WhatsApp, or Dropbox. While some programs did have central
knowledge platforms, they were difficult to update in real-time. Here,
we present our vision for a digital source-of-truth for such
institutional knowledge - an operating system for residency.

## Methodology

First, we analyzed the pain points that residents experienced accessing
and organizing institutional knowledge. This information was previously
siloed on multiple different platforms. Service and surgeon-specific
protocols were uploaded to Google Drive, buried in text threads, or
printed on paper depending on the service. Finding administrative forms
often required searching through emails. Resident and faculty contacts
were available on a Google sheet while hospital numbers were printed on
a paper card. Furthermore, when such information could be found, it was
often out-of-date, as no mechanism existed to update and disseminate
knowledge in real-time.

After evaluating these pain points, we developed the following user
requirements to guide the design of a new platform. First, the solution
had to be mobile-friendly - as residents always have their phones on
hand. Second, the solution had to be easy to update - as the annual
review process previously in place was inconsistent and left too many
outdated protocols in circulation. Third, the solution had to be
intuitive - easy to use with minimal training. Fourth, the solution had
to be low cost - as there was no funding for this project.

There are multiple software options that offer the functionalities we
were looking for. Ultimately, we decided to use Notion to build our
platform. Notion allows users to build custom, collaborative workspaces
consisting of documents and databases. It is used by individuals and
organizations across industries to create wikis, manage projects, write
documents, and build internal tools. It is available as a web, desktop,
and mobile application. Users can update any of the content in
real-time. [Notion](https://notion.so/?ref=ideasurg.pub) is offered for
free to educational institutions for up to 100 guest users at present.
Additional features are available at a charge. Perhaps its most useful
feature is the ability to update information in one place and have that
update reflected elsewhere. For example, if you update a surgeon\'s
email address on the \'Contacts\' page, the updated address will
automatically appear on the pages of any services or committees with
which that surgeon is affiliated. As such, there is always a single
source of truth for any piece of information.

## Implementation of the Platform

First, we created a digital version of our \'Intern Survival Guide.\'
Each topic from the physical paper guide -- representing a clinical
scenario that an intern may encounter -- was transformed into a digital
card containing the relevant steps in triage and work up. Issues
requiring escalation to a senior resident or attending surgeon were
highlighted.

<figure class="kg-card kg-embed-card kg-card-hascaption">
<div class="iframe">
<div id="player">

</div>
<div class="player-unavailable">
<h1 id="ein-fehler-ist-aufgetreten." class="message">Ein Fehler ist
aufgetreten.</h1>
<div class="submessage">
<a href="https://www.youtube.com/watch?v=Z-mlK5LN7w8"
target="_blank">Sieh dir dieses Video auf www.youtube.com an</a> oder
aktiviere JavaScript, falls es in deinem Browser deaktiviert sein
sollte.
</div>
</div>
</div>
<figcaption>Video demo of the Intern Survival Guide</figcaption>
</figure>

Next, the disjointed resources containing service-specific protocols
were adapted to the Notion platform. In a multi-week effort, content was
systematically transferred, and post-operative protocols were
standardized in a tabular format. Tutorials detailing how to enact these
protocols in the electronic health record (EHR) were constructed in
parallel - including step-by-step screenshots of common tasks including
pre-rounding, placing orders, and admitting or discharging patients.

<figure class="kg-card kg-embed-card kg-card-hascaption">
<div class="iframe">
<div id="player">

</div>
<div class="player-unavailable">
<h1 id="ein-fehler-ist-aufgetreten." class="message">Ein Fehler ist
aufgetreten.</h1>
<div class="submessage">
<a href="https://www.youtube.com/watch?v=nKQXLAlv5Dg"
target="_blank">Sieh dir dieses Video auf www.youtube.com an</a> oder
aktiviere JavaScript, falls es in deinem Browser deaktiviert sein
sollte.
</div>
</div>
</div>
<figcaption>Video Demo of a Typical Service Page and Ability to Quickly
Access Post-op Care Pathways</figcaption>
</figure>

Next, the contact directory was migrated to Notion. This effort
consolidated information that was previously housed on Google sheets and
index cards. Moreover, users could now make calls directly from the
Notion app (rather than retyping or copying and pasting the number),
reducing wasted time and potential errors.

Over the following year, we launched Logistics and Wellness committees
to lead projects to continually improve the program. The committees meet
a few times per year to determine priorities and share updates. We are
now utilizing Notion's project management features to organize our work.
These features include the ability to create Kanban boards, build
progress bars, and assign tasks to projects and projects to committees.
We have already seen this help us achieve better focus during meetings
and leave participants with specific actionable tasks.

Through built-in analytics, we are able to track view counts for the
entire workspace and each individual page. Over the last 90 days, the
platform was viewed 1,241 times. The platform also maintains full
version history which allows us to track who is making edits and what
changes are made. In the last month, 90 edits were made including the
addition of new pages and database items, modification of existing
content, and changing user permissions. The high level of active and
passive engagement is encouraging to the viability of the platform.

<figure class="kg-card kg-image-card kg-card-hascaption">
<img
src="https://www.ideasurg.pub/content/images/2023/03/Xnapper-2023-03-04-15.18.55-1.png"
class="kg-image" loading="lazy"
srcset="https://www.ideasurg.pub/content/images/size/w600/2023/03/Xnapper-2023-03-04-15.18.55-1.png 600w, https://www.ideasurg.pub/content/images/size/w1000/2023/03/Xnapper-2023-03-04-15.18.55-1.png 1000w, https://www.ideasurg.pub/content/images/size/w1600/2023/03/Xnapper-2023-03-04-15.18.55-1.png 1600w, https://www.ideasurg.pub/content/images/2023/03/Xnapper-2023-03-04-15.18.55-1.png 1990w"
sizes="(min-width: 720px) 720px" width="1990" height="1110" />
<figcaption>Screenshot of built-in analytics function. Blue line
represents total views while yellow line represents unique
views.</figcaption>
</figure>

Minimal training is needed to use the platform effectively. Incoming
interns receive a one-hour onboarding session which is practical in
nature (interns are given questions simulating common resident tasks and
asked to use Notion to figure out how to achieve them). Following this,
questions about application use are answered on an as-needed basis
through our residency group chat or email.

## Limitations and Future Directions

Our Notion instance is a work in progress and several improvements will
be necessary to achieve our vision of a \'one-stop shop\' for resident
education and well-being.

While residents frequently update content on various service pages, it
is difficult to maintain adherence to a standardized format. As everyone
is responsible for maintaining the integrity of the platform, the burden
of oversight falls to no singular individual. As such, we have not yet
systematized a process to review existing information. Page bloat and
content disorganization occur as content is added, but efforts to
address these issues are currently being discussed.

We have not completely solved the issue of multiple platforms, though
the total number of information sources is reduced. The resident
schedule continues to be maintained on a Google sheet and capabilities
to incorporate it into Notion are limited. Our program previously used a
proprietary platform to host instructional videos for operations, and
migration of these videos to Notion is in progress. Our goal is to make
Notion a hub for resident education, where didactic content as well as
operative videos can be easily accessed.

Fundamental questions remain surrounding data governance and access. We
currently give all residents and faculty members complete edit access to
democratize the information ecosystem. As a result, information is
updated with much greater frequency, and these updates are immediately
reflected to all users. Yet, this amount of access carries risks. While
inadvertent edits are made (which can easily be seen and undone through
version history), to this day, there have been no malicious edits that
negatively impact patient care. While the decentralized model optimizes
ease of editability, it does forgo a traditional hierarchical structure
that previously existed.

## Conclusions

In the past few years, collaboration applications such as Notion have
flourished to facilitate asynchronous information sharing. Such
platforms are addressing challenges that organizations have faced for
years but have grown in complexity in the COVID era. We are using Notion
to create a digital operating system for residency. We champion an
approach that embraces technology and empowers each resident to take
ownership of the collective knowledge of the program. Our vision is to
have all the information of residency in the palm of your hand, in three
clicks or less. While any technology has its limitations and raises new
issues, we believe this platform can revolutionize the way our trainees
access, interact with, and learn the information of residency.

::: {.kg-card .kg-toggle-card kg-toggle-state="close"}
::: {.kg-toggle-heading}
#### Responses to Reviewers {#responses-to-reviewers .kg-toggle-heading-text}

![](data:image/svg+xml;base64,PHN2ZyBpZD0iUmVndWxhciIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Ym94PSIwIDAgMjQgMjQiPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTIzLjI1LDcuMzExLDEyLjUzLDE4LjAzYS43NDkuNzQ5LDAsMCwxLTEuMDYsMEwuNzUsNy4zMTEiPjwvcGF0aD48L3N2Zz4=){#Regular}
:::

::: {.kg-toggle-content}
Our paper was submitted for publication and rejected during the peer
review process. We asked the journal if we could publish the reviewer
comments and our responses but were denied permission. Instead, we will
attempt to paraphrase the comments and our rebuttals here.

One reviewer felt that many programs had similar platforms in place. We
did not find this to be true in our exploratory analysis. We conducted
informal surveys and spoke to residents at national conferences. A
majority of survey responses indicated that residents did not have
centralized knowledge sharing platforms in place. Further, while some
residents were using such platforms, they differed from ours because
they were not directly editable in real-time. The reviewer also felt our
paper constituted an advertisement of one software option. We want to
emphasize that none of the authors received financial compensation from
Notion or any other company in this space. Finally, we accept that there
are many platforms that could be used to organize the institutional
knowledge of residency. While we present one, we would be excited to
hear about the experiences that other institutions have with other
platforms and collaboratively work towards designing the best optimal
technology stack to teach and train residents. For that reason, it is
even more important that technologies like this are discussed in the
academic literature.

Another reviewer brought up the reasonable criticism that we are
attempting to create a technological solution to a cultural problem and
questioned whether residents would actually use the platform. We added
data on both platform viewership (1241 views over last 90 days) and
number of edits made (90 in the last month) to the Implementation
section.

Many people - including one of the reviewers and people we speak to at
conferences - want to know who is in charge of updating the information
and making sure it is correct. We have deliberately chosen a
decentralized approach that gives residents autonomy to maintain
information. Before we implemented Notion, there was no routine faculty
review of paper-based guides. Ultimately, there is a trade-off between
conducting an overview of every piece of information posted on the
platform and keeping the platform up-to-date. We chose to focus on the
latter. We are also introducing faculty onto the platform and giving
them edit access so that they can make edits and provide whatever
oversight they feel is necessary.

Our central hypothesis in creating this platform was that providing
residents a technical solution to access and organize information would
allow them to quickly develop working knowledge of rote tasks and
protocols and spend more time learning how to take care of patients and
operate. Certainly, future work will need to be done in order to
understand whether we are successful. Today, we can confidently say we
have built a user-friendly, fast system that is frequently used and
updated.
:::
:::

::: {.kg-card .kg-toggle-card kg-toggle-state="close"}
::: {.kg-toggle-heading}
#### Disclosures {#disclosures .kg-toggle-heading-text}

![](data:image/svg+xml;base64,PHN2ZyBpZD0iUmVndWxhciIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB2aWV3Ym94PSIwIDAgMjQgMjQiPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTIzLjI1LDcuMzExLDEyLjUzLDE4LjAzYS43NDkuNzQ5LDAsMCwxLTEuMDYsMEwuNzUsNy4zMTEiPjwvcGF0aD48L3N2Zz4=){#Regular}
:::

::: {.kg-toggle-content}
None of the authors received financial compensation from Notion or have
any disclosures relevant to this paper.
:::
:::
