Original Research

Implementation and evaluation of a “works‐in‐progress” session to promote scholarship in an academic hospitalist group

Abstract

BACKGROUND

Hospitalists frequently work on diverse projects, but often do not have the training and experience necessary to translate projects into peer‐reviewed publications and grants.

OBJECTIVE

Describe implementation and effect of a works‐in‐progress (WIP) series on progress and training in scholarly work.

DESIGN

Cross‐sectional survey.

SETTING

Urban academic medical center.

INTERVENTION

A weekly WIP session, named Incubator, serving as a forum where researchers, clinicians, and educators meet to review and provide feedback on projects underway across the Division of Hospital Medicine.

MEASUREMENTS

We surveyed presenters at Incubator to evaluate the impact of Incubator on scholarly activities. Responses were based on Kirkpatrick's 4‐level training hierarchy: (1) Reaction: participants' satisfaction; (2) Learning: knowledge acquisition; (3) Behavior: application of skills; and (4) Results of projects. We compared responses between researchers and nonresearchers using χ2 tests.

RESULTS

Of 51 surveys completed (response rate 70%), 35 (69%) projects were nonresearcher led. Reaction, behavior change, and results were all positive, with >90% respondents reporting a positive outcome in each category, a high rate of publication/funding, and 35% reporting learning as a result of Incubator. Comparison of researchers and nonresearchers revealed no significant differences, except nonresearchers reported significantly more favorable results in behavior and mentoring (P < 0.05).

DISCUSSION

A regularly scheduled, researcher‐led WIP session within a largely clinically oriented hospital medicine division can provide a venue for feedback that may promote progress and practical training in scholarly projects. In addition to robust career mentorship programs and protected time, a WIP can be an adjunct to improve scholarly output among academic hospitalists. Journal of Hospital Medicine 2016;11:719–723. © 2016 Society of Hospital Medicine

© 2016 Society of Hospital Medicine

Academic hospital medicine is a fast‐growing specialty and has a strong emphasis on high‐value care, efficiency, and quality improvement (QI).[1] Developing scholarly work in these areas and describing findings in peer‐reviewed publications can help disseminate ideas and innovations more widely. In addition, success in academic medicine, at least in part, continues to be measured by traditional academic benchmarks, including the production of scholarly publications, conference presentations, and abstracts.[2]

Hospital medicine, however, faces challenges in providing an academic environment conducive to fostering scholarly work. As a relatively young specialty, there may be a dearth of senior mentors and experienced researchers; lack of structured mentorship can be associated with failure to produce publications or lead national teaching sessions.[3] Relatively few hospitalists undergo fellowships or other specialized training that provides a clinical research background, and internal medicine residency programs rarely provide the comprehensive research skill set required to design, implement, or disseminate academic work.[4, 5, 6] Finally, heavy clinical responsibilities may hinder efforts to conduct and sustain research.

A works‐in‐progress (WIP) session, commonly employed in clinical research groups, can provide a forum to discuss and receive feedback on evolving projects and can foster mentorship, motivation, and training.[7] Although a WIP session may stimulate discussion and advance project ideas, academic hospitalist groups do not commonly employ this model, and it is not known if a regularly scheduled WIP session can provide the mentorship, training, and motivation necessary to assist junior faculty in advancing scholarly project to completion.[8] In this article, we describe how we developed a regular WIP series to promote scholarship activities within our rapidly growing, primarily clinically focused Division of Hospital Medicine (DHM) at the University of California, San Francisco (UCSF), and the results of a survey of WIP participants. We hope that our experience can help illustrate key features of such a model, as well as describe inherent challenges and lessons learned to help promote successful academic efforts at other institutions.

METHODS

Program Setting

During years 2010 to 2013, the time period captured by our survey, the DHM at UCSF grew from 37 to 46 full‐time hospitalists, with 76% primarily clinical faculty (nonresearchers) and 24% primarily clinician‐investigators (researchers), defined as individuals having completed a 2‐year clinical research fellowship and/or dedicating 70% time in their faculty position to clinical research. In addition, there were between 1 and 3 hospitalist fellows per year. In 2012, a PhD researcher joined the division to support research and academic activities within the division as well as to pursue an independent research career.

Program Description

The DHM WIP, named the Incubator, was initially developed in 2007 when researchers recognized the need and desire for a forum where scholarly projects could be reviewed and evaluated. In the first year, the Incubator was primarily utilized by junior research‐trained mentees applying for National Institutes of Health career development awards. However, it soon became clear that nonresearch trained junior fellow and faculty members were pursuing scholarly projects needing additional guidance and input. In particular, the Incubator became frequently utilized by academic hospital medicine fellows and resident trainees pursuing QI and education projects. Over time, more DHM faculty, and junior faculty in particular, began to present their projects and receive structured feedback from researchers as well as other senior members of the group.

Incubator is structured as a 50‐minute session held from 1:10 to 2:00 pm on Thursdays in a DHM conference room. The time was selected because it did not conflict with other divisional conferences and to reserve mornings for clinical responsibilities. Incubator is held on most weeks of the year except for holidays or when there is no scheduled presenter. Presenting at Incubator is voluntary, and presenters sign up for open spots in advance with the upcoming presenter schedule sent out to the division in advance of the conference. Incubator is also used as a forum to provide feedback on anticipated abstract submissions for professional society meetings. For the purposes of the survey described in this article, we did not include Incubator sessions on reviewing abstracts/posters. Trainees and hospitalists present a broad range of projects at any stage of preparation. These include project ideas, grant applications, manuscripts, abstracts, and oral presentations at any stage of completion for feedback. Our mission was to create a forum where researchers, clinicians, and educators meet to provide the tools and guidance necessary to promote scholarly projects across the range of the division's activities by connecting individuals with complementary skills and interests and providing necessary mentorship and peer support. We have defined scholarship broadly, including evaluation of QI, global health, or other health system innovations, as well as advancements in medical education and traditional clinical research.

All faculty are invited to Incubator, and attendees include senior and junior faculty, researchers in the division, fellows, and occasionally residents and medical students. One week prior to the session, an administrative assistant solicits project information, including any related materials and questions the presenter may have for the group using a prespecified template, and emails this information to division members for review. In addition, the same materials are also printed prior to Incubator for any attendees who may not have reviewed the material in advance. Also, prior to the session, a physician is specified to serve as moderator of the discussion, and another physician is assigned the role of primary reviewer to provide the initial specific feedback and recommendations. The role of the moderator is to manage the discussion and keep the focus on time, and is assigned to a researcher or senior clinical faculty member. The role of primary reviewer is assigned to provide more junior faculty (both researchers and nonresearchers) the opportunity to practice their editing and critiquing skills by providing the initial feedback. Presenters and moderators receive worksheets outlining the structure of Incubator and their respective roles (see Supporting Information, Appendix 1, in the online version of this article).

Incubator begins with the presenter providing a brief synopsis of their project and their specific goals and objectives for the session. The moderator then leads the discussion and guides the format, often starting with any questions the group may have for the presenter followed by the specific feedback from the primary reviewer. The primary reviewer, having reviewed the materials in advance of the session, answers the prespecified questions as listed by the presenter, occasionally providing additional targeted feedback. The session is then opened to the rest of the group for feedback and suggestions. Meanwhile, the presenter is encouraged to wait until the end of the hour to summarize their take on the feedback and what their initial thoughts on the next to do items would be (Table 1).

Summary of Incubator Roles
PresenterAdministrative assistant
2‐ to 3‐sentence summary of career focusSchedule session and conference room
Distribute short set of materials in advanceCollect presenters' materials in advance
Summarize feedback at end of sessionPrepare materials for Incubator
Brainstorm on next steps at end of sessionMonitor attendance and topics of presentation
Primary reviewerModerator
Junior faculty (24 years)Senior or research faculty
Provide brief overview of projectKeep session on time
Reiterate key questionsGive additional input
Provide 2 major, 3 minor suggestionsSummarize comments from group at the end
Constructive, outside the box feedbackAllow last 10 minutes for presenter to discuss plans

Program Evaluation

Survey Respondents and Process

We retrospectively surveyed the lead presenter for each Incubator session held between May 2010 through November 2013. Surveys were administered through the Research Electronic Data Capture application (REDCap).[9] Participants who were lead presenters at Incubator for more than 1 Incubator session completed a survey for each individual presentation. Therefore, some presenters completed more than 1 survey. The presenters included resident physicians, hospital medicine fellows, junior faculty, and researchers. We defined researchers as hospitalists who had completed a 2‐year research fellowship and/or devoted at least 70% time in their faculty position to research.

Survey Development and Domains

We developed a survey questionnaire using the Kirkpatrick 4‐level model to evaluate the educational experience of the primary presenters and to determine how the session impacted their progress on the project, with each model component graded according to a Likert scale.[10] The 4 major components of the model are: (1) Reaction: participants' estimates of satisfaction with Incubator; (2) Learning: extent of knowledge acquisition achieved at Incubator; (3) Behavior: extent to which learning has been applied or transfer of skills through participation in Incubator; and (4) Results: results of the project, wider changes in organizational scholarship as impacted by Incubator.

We also collected information on the presenter's status at time of presentation including career paths (researcher or nonresearcher), their job description (faculty, fellow, resident), and the total number of years on faculty (if applicable). Hospitalists in their first 2 years on the faculty were considered junior physicians. We also collected information on the number of times they had presented at the Incubator sessions and stage of progress of the project, whether in the early, mid, or late phase at the time of presentation. Early phase was defined as presenting an initial project idea or brainstorming possible project options and/or directions. Mid phase was defined as presenting initial results, data, and initial drafts prior to completion of analysis. Late phase was defined as presenting a project nearing completion such as a written abstract, oral presentation, paper, or grant application. Respondents were also asked to identify the main focus of their projects, selecting the categories based on the interests of the division, including medical education, clinical research, QI, high‐value care, and global health.

Survey Data Analysis

We converted Likert scale data into dichotomous variables, with paring of positive responses versus the negative options. We summarized survey responses using descriptive statistics and determined if there were any differences in responses between career researchers and nonresearchers using 2 tests. All analysis was performed using StataSE version 13.1 (StataCorp, College Station, TX).

RESULTS

Survey Respondent Characteristics

We received 51 completed surveys from presenters at an Incubator session, for a total survey response rate of 70%. Of the 51 presentations, 26 (51%) of the projects were led by physicians in training or junior faculty, and 35 (69%) of the presenters were nonresearchers.

Project Characteristics

The most frequently presented topic areas were QI (N = 20), clinical research (N = 14), medical education (N = 6), and global health (N = 6). Whereas researchers were more likely to present clinical research topics and grant applications, nonresearchers more often presented on QI or medical education projects (Table 2). Projects were presented at all stages of development, with the middle stage, where presenters presented initial results, being the most common phase.

Characteristics of Work‐in‐Progress Session Presentations Among 51 Nonresearchers and Researchers
AllNonresearcher, No. (%)Researcher, No. (%)P Value
  • NOTE: *Early stage was defined as presenting an initial project idea or brainstorming possible project options and/or directions. Mid stage was defined as presenting initial results, data, and initial drafts prior to completion of analysis. Late stage was defined as presenting a project nearing completion such as a written abstract, oral presentation, paper, or grant application.

Total513516
Trainee or junior faculty 19 (54%)7 (44%)0.49
Topic of project 0.02
Quality improvement20 (39%)15 (43%)5 (31%)
Clinical research14 (27%)8 (23%)6 (38%)
Medical education6 (12%)5 (14%)1 (6%)
Health technology4 (8%)0 (0%)4 (25%)
High‐value care1 (2%)1 (3%)0 (0%)
Global health6 (12%)6 (12%)0 (0%)
Stage of project 0.31
Early*12 (23%)7 (20%)5 (31%)
Middle24 (47%)19 (54%)5 (31%)
Late15 (29%)9 (26%)6 (38%)

Impact of Incubator

The reaction to the session was very positive, with 100% of respondents recommending Incubator to others (Table 3), and 35% reported learning as a result of the session. Twenty‐three (45%) of respondents reported that the session helped reframe the project idea and changed the study design, and 20 (39%) reported improved written or oral presentation style. A majority (45, 88%) reported that Incubator was valuable in advancing the project to completion.

Survey Responses of 51 WIP Presenters According to the Kirkpatrick Evaluation Hierarchy
AllNonresearcher, No. (%)Researcher, No. (%)P Value
  • NOTE: Abbreviations: WIP, works‐in‐progress.

Trainee or junior faculty5135 (69%)16 (31%)0.49
Reaction
Satisfied with their WIP session50 (98%)35 (100%)15 (94%)0.25
Would recommend WIP to others51 (100%)35 (100%)16 (100%)1.00
Any of the above 35 (100%)16 (100%)1.00
Learning
Advanced research methodology18 (35%)12 (34%)6 (38%)0.82
Advanced knowledge in the area9 (18%)5 (14%)4 (25%)0.35
Any of the above 14 (40%)9 (56%)0.28
Behavior
Current project
Reframed project idea23 (45%)15 (43%)8 (50%)0.63
Changed study design or methodology23 (45%)16 (46%)7 (44%)0.9
Improved written or oral presentation style20 (39%)15 (43%)5 (31%)0.43
Future projects
Changed approach to future projects19 (37%)17 (49%)2 (13%)0.01
Any of the above 34 (97%)14 (88%)0.17
Results
Valuable in advancing project to completion45 (88%)31 (89%)14 (88%)0.18
Provided mentoring and peer support29 (57%)24 (69%)5 (31%)0.01
Connected individuals with similar results13 (13%)9 (26%)4 (25%)0.96
Any of the above 34 (97%)14 (88%)0.17

Survey results of researchers compared to nonresearchers were similar overall, although nonresearchers were more likely to report changes in behavior and in improved mentoring as a result of presenting at Incubator. Notably, 17 (49%) of nonresearchers reported that Incubator changed their approach to future projects as opposed to only 2 (13%) researchers (P = 0.01). In addition, 24 (69%) nonresearchers reported value in mentorship and peer support compared to 5 (31%) researchers (P = 0.01). A reasonably large proportion of projects originally presented during the Incubator sessions became published articles at the time of survey completion (N = 19, 37%) or were publications in progress (N = 14, 27%). For all remaining items, there were no statistically significant differences in the survey responses among junior faculty/trainees (N = 26) compared to nonjunior faculty (N = 25) presenters (P > 0.05).

Attendance at Incubator During the Study Period

Attendance at Incubator was open and voluntary for all DHM faculty, fellows, and collaborating UCSF trainees. From July 2012, when we began tracking attendance, through the end of the survey period in November 2013, the average number of attendees for each session was 10.7 (standard deviation [SD] 3.8). On average, 50% (SD 16%) of attendees at Incubator were career researchers.

DISCUSSION

The results of this program evaluation suggest that a WIP session employed by an academic division of hospital medicine, consisting of a weekly moderated session, can help advance scholarly work. Our evaluation found that presenters, both researchers and nonresearchers, favorably viewed the regular WIP sessions and reported that feedback in the Incubator helped them advance their project to completion. Importantly, nonresearch‐focused faculty and fellows reported the biggest gains in learning from presenting at Incubator. Whereas half the Incubator attendees were career researchers, consistent with the observation that researchers within the division were most committed to attending Incubator regularly, 69% of the presenters were nonresearchers, demonstrating strong participation among both researchers and nonresearchers within the division.

WIP sessions, though informal, are interactive, inspire critical self‐reflection, and encourage physicians to act on generated ideas, as evidenced by the change in behavior of the participants after the session. These sessions allow for transformative learning by encouraging physicians to be open to alternative viewpoints and engage in discourse, boosting learning beyond just content knowledge. Prior assessments of WIP seminars similarly found high satisfaction with these formats.[11]

Although we cannot identify specifically which features made Incubator effective, we believe that our WIP had some characteristics that may have contributed to its success and may aid in implementation at other institutions: holding the session regularly, voluntary participation, distributing the materials and questions for the group in advance, and designating a moderator for the session in advance to facilitate discussion.

A potential strength of the Incubator is that both researchers and nonresearchers attend. We hypothesize that combining these groups provides improved mentorship and learning for nonresearchers, in particular. In addition, it creates a mutually beneficial environment where each group is able to witness the diversity of projects within the division and learn to provide focused, constructive feedback on the presented work. Not only did this create a transparent environment with better understanding of divisional activities, but also fostered collaboration among hospitalists with similar interests and complementary skills.

Challenges, Setbacks, Updated Approaches

The creation of a successful Incubator session, however, was not without its challenges. At initial inception, the WIP was attended primarily by researchers and had low overall attendance. Members of the division who were primarily clinicians initially perceived the conferences as largely inapplicable to their career objectives and had competing demands from patient care, educational, or administrative responsibilities. However, over time and with encouragement from divisional leaders and service line directors, increasing numbers of hospitalists began to participate in Incubator. The timing of Incubator during afternoons after the Department of Medicine Grand Rounds was chosen specifically to allow clinicians to complete their responsibilities, including morning rounds and teaching, to allow better attendance.

In addition, the results of our survey informed changes to the structure of Incubator. The efficacy of assigning a primary reviewer for each session was not clear, so this component was eventually dropped. The finding that nonresearchers in particular reported a benefit from mentoring and peer‐support at Incubator led to the implementation of querying the presenter for a wish list of faculty attendees at their Incubator session. We then sent a special invitation to those faculty members thought to have special insights on the project. This gave junior faculty the opportunity to present their projects to more senior faculty members within their areas of research, as well as to receive focused expert feedback.

Finally, we have initiated special Incubator sessions focused more on didactics to teach the process of writing manuscripts and brainstorming workshop ideas for national meetings.

Limitations

Our study has limitations. It is a single‐center study based on a small overall sample size, and it is not certain whether a similar innovation would have comparable effects at another institution. In addition, generalizability of our results may be limited for hospital medicine groups without a robust research program. We did not have a control group nor do we know whether participants would have been equally successful without Incubator. We also were unable to assess how Incubator affected long‐term outcomes such as promotion and overall publication record, as we do not have detailed data on productivity prior to the survey period. Finally, we are unable to quantify the effect of Incubator on scholarly success in the division. Although the numbers of published articles and grant funding has increased since the Incubator began (data not shown), the division also grew both in number of research‐focused and nonresearch‐focused faculty, and this study does not account for other temporal changes that may have contributed to improvements in the scholarly output of the division.

CONCLUSIONS

In summary, the Incubator has been a successful program that fostered progress on scholarly projects within a largely clinically focused DHM. Given the importance of scholarship in academic hospital medicine, a WIP session such as the one we describe is a valuable way to support and mentor junior hospitalists and nonresearchers.

Acknowledgements

The authors extend special thanks to Oralia Schatzman, divisional administrative assistant, who organized and arranged the Incubator sessions and recorded attendance, and to Katherine Li, who collected data on numbers of faculty within the division over the duration of the study.

Disclosures: Dr. Hemali Patel, Dr. Margaret Fang, and Mr. James Harrison report no conflicts of interest. At the time the research was conducted, Dr. Kangelaris was supported by the National Heart, Lung, and Blood Institute (1K23HL116800‐01). Dr. Auerbach was supported by the National Heart, Lung, and Blood Institute (K24 K24HL098372).

References

Online-Only Materials

AttachmentSize
PDF icon Appendix1.pdf792.87 KB
File Survey_for_Incubator.docx20.7 KB
   Comments ()