Original Research

Quantifying internal medicine resident clinical experience using resident‐selected primary diagnosis codes




While experiential learning is a desirable goal of residency education, little is known regarding the actual clinical experience of internal medicine residents during their training.


We modified an electronic patient handoff tool to include a system for resident entry of a primary diagnosis for each of their patients. Using the International Classification of Diseases, Ninth Revision (ICD‐9) system, we created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes; and 2) a search option for the complete ICD‐9 database. The codes were then grouped using ICD‐9 categorization.


A total of 7562 resident–patient diagnostic encounters were studied. A wide spectrum of clinical conditions was observed, with symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue accounting for about 86% of resident clinical experience. Symptoms and ill‐defined conditions were noted to represent a sizable portion of resident clinical experience. Within this category, the most common conditions were fever; abdominal pain; and chest pain, unspecified.


Analysis of resident‐selected ICD‐9 codes might serve as a method to attempt to define resident clinical experience, and may be useful in the development of innovative experiential learning‐based residency curricula. This might also be used to assess gaps in experiential learning at the program or resident level, and may serve to identify topics that require additional teaching supplementation. Journal of Hospital Medicine 2011. © 2011 Society of Hospital Medicine

Copyright © 2011 Society of Hospital Medicine

Internal medicine residency training continues to evolve as competency‐based and with education organized around patient care.13 Making the patient the center of resident education provides an opportunity for experiential learning in which learning can be organized around the clinical conditions that residents encounter. Despite the renewed emphasis on using patient experience as the basis for residency education, little is known regarding what specific diagnostic conditions are seen by internal medicine residents throughout their training. Attempts have been made to quantify resident clinical experience in various fields, using approaches such as review of medical records, case logs, and prescription profiles, but to date, we lack systematic methods to obtain clinical experience data for internal medicine residents.47

While residency curricula in internal medicine typically outlines specific rotations in various clinical areas such as general medical wards, cardiology services, and intensive care units, time spent on such rotations does not necessarily provide quantitative data on the actual clinical conditions that residents encounter, nor does it ensure consistent clinical experience between residents. It is plausible that there may be substantial variability in clinical experience between residents within the same program, and that the overall spectrum of clinical disorders seen by residents in a program may or may not be consistent with a desired optimum, though this is yet to be defined.

If residency education in internal medicine is to progressively incorporate more experiential learning, detailed knowledge of the clinical conditions seen by residents should be useful, not only for overall curriculum design, but this might also allow for various educational interventions to be made when there are variations in clinical experience between residents. Our program has been interested in the application of electronic resources for the improvement of patient care, such as through the handoff process and the use of personal digital assistants.8 We previously did a small analysis of clinical conditions seen by residents through non‐International Classification of Diseases, Ninth Revision (ICD‐9)‐based data they entered onto personal digital assistants. This suggested to us that electronic resources used by residents might serve as a venue by which they could enter diagnostic information which we could use to generate a more detailed analysis of the clinical conditions that they see. Here we describe a method by which we have attempted to quantify resident clinical experience in internal medicine using a modification of an electronic handoff system.


The study was conducted within the Internal Medicine Residency Program at the Long Island Jewish Medical Center in New Hyde Park, New York, part of the North ShoreLong Island Jewish Health System, and was approved by the Institutional Review Board. This work was carried out as part of our participation in the Educational Innovation Project of the Residency Review Committee for Internal Medicine. A central objective of our proposal was to develop a method to assess residents' clinical experience on an individual and an aggregate basis. A group of faculty and residents in our residency program developed an electronic handoff tool which residents use for rapid access to key clinical data for their patients and for the handoff of clinical information for on call coverage. This handoff tool was developed with the technical assistance of MedTech Notes LLC which owns Patient Data Transfer System (PDTS) HandOff Note. We modified the handoff tool to include a section in which residents were required to enter a primary diagnosis for each of their patients (a hard stop design). We chose to use the ICD‐9 system for standardization and created two methods to select the code: 1) an organ system‐based dropdown list containing frequently used codes and 2) a search box allowing for searching of the complete ICD‐9 database. For the organ‐based dropdown list, selection of that organ system would reveal a brief list of frequently used codes to make it easier for residents to find them. Prior to using the handoff tool with the ICD‐9based primary diagnosis coding system, training sessions with the residents were conducted by 3 of the investigators along with 3 chief medical residents. These sessions included training not only in technical aspects of how to find diagnosis codes, but also how to make decisions regarding what the primary diagnosis should be. We also instructed our postgraduate year (PGY)‐1s to update their diagnostic selections during the course of the hospital stay.

Each data point represents a resident caring for a patient with a specific diagnostic entity, and is counted once for that resident's period of taking care of that patient. Thirty‐three PGY‐1s were studied and, on the internal medicine service, they were supervised by either hospitalist faculty or voluntary faculty in comparable proportions. If the patient's care is taken over by another resident, that second resident was also recorded as having had a diagnostic encounter with that patient, hence 1 patient could provide experience with the same diagnostic entity for 1 or more residents. Using this method, the denominator is not patients seen, but residentpatient diagnostic encounters that have taken place. The ICD‐9 diagnostic conditions entered by the residents were grouped using the ICD‐9 system. Individual diagnostic profiles for each resident, as well as an aggregate profile for all residents to reflect the residency program as a whole, were generated. We also carried out an analysis of the ICD‐9 codes entered by 6 consecutive PGY‐1s to assess how the diagnostic spectrum might vary among a small sampling of PGY‐1s. In order to evaluate the accuracy of the residents' diagnostic selections, we carried out a validation assessment using a tool used by the residents' supervising hospitalists (who were the attendings of record for those patients). This was carried out on a subset of patients and could be done at any time during the hospital stay. The hospitalists were asked to review their residents' ICD‐9 codes and indicate whether they agreed or disagreed.


A total of 7562 residentpatient diagnostic encounters were studied from July 1, 2007 through June 1, 2008. Mean patient age was 66 19.4 years. The age distribution is given in Table 1 and reveals that 65% of diagnostic encounters were with patients age 60 years or greater. Twelve housestaff teams were studied, each consisting of 2 PGY‐1s and a supervising PGY‐2 or PGY‐3 resident. All ICD‐9 codes were selected by categorical and preliminary internal medicine PGY‐1s on medical ward and intensive care unit rotations. Residents from other departments doing rotations on the medical service were excluded. A validation assessment of 341 patients indicated 83.3% agreement by the supervising hospitalist with the primary ICD‐9 code selected. ICD‐9 codes were then grouped and categorized using ICD‐9 nomenclature with the distribution provided in Table 2. A wide spectrum of clinical conditions is apparent including symptoms and ill‐defined conditions, circulatory disorders, respiratory disorders, neoplasms, genitourinary disorders, digestive disorders, diseases of the blood/blood forming organs, endocrinologic/nutritional/metabolic/emmmune disorders, and disorders of the skin and subcutaneous tissue, overall accounting for about 86% of resident clinical experience.

Patient Age Categories (n = 7,562)
Age CategoryNo.Percent of Total
Frequency of the Most Commonly Encountered Diagnoses by ICD‐9 Category Among Patients Evaluated by Internal Medicine Residents
ICD‐9 Category DescriptionFrequencyPercent
  • Abbreviations: ICD‐9, International Classification of Diseases, Ninth Revision.

Symptoms/Ill‐Defined Conditions1,47519.51
Circulatory System1,38118.26
Respiratory System93912.42
Genitourinary System5026.64
Digestive System4646.14
Blood/Blood‐Forming Organs4445.87
Skin and Subcutaneous Tissue3805.03
Injury and Poisoning2222.94
Musculoskeletal/Connective Tissue1992.63
Mental Disorders1662.20
Nervous System/Sense Organs1251.65
Health Status/Contact with Health Services811.07

We also examined the most common diagnostic conditions within each of these categories. The 3 most common ICD‐9 codes entered by residents within each category are provided in Table 3. Symptoms and ill‐defined conditions represent a sizable portion of resident clinical experience (19.51%). Within this category, the most common conditions were fever; abdominal pain (unspecified site); and chest pain, unspecified. Disorders of the circulatory and respiratory systems were the next most common categories of conditions seen by residents, comprising 18.26% and 12.42%, respectively, of resident clinical experience. Within the category of circulatory disorders, congestive heart failure and acute myocardial infarction were the most common conditions seen; for respiratory disorders, pneumonia, chronic airway obstruction, and asthma were most commonly encountered. In aggregate, symptoms and ill‐defined conditions, and disorders of the circulatory and respiratory systems accounted for 50% of resident clinical experience.

Top 3 ICD‐9 Diagnosis Codes Within Each ICD‐9 Category
ICD‐9 Category DescriptionICD‐9 CodeCode DescriptionFrequencyPercent
  • Abbreviations: Hb‐C, hemoglobin C; ICD‐9, International Classification of Diseases, Ninth Revision.

Symptoms/Ill‐Defined Conditions780.6Fever1902.51
789Abdominal pain; unspecified site1491.97
786.5Chest pain, unspecified1401.85
Circulatory System428Congestive heart failure, unspecified3464.58
410.9Acute myocardial infarction; unspecified site; unspecified episode of care1351.79
410.1Acute myocardial infarction; other anterior wall; unspecified episode of care1061.40
Respiratory System486Pneumonia, organism unspecified3634.80
496Chronic airway obstruction, not elsewhere classified1622.14
493.9Asthma, unspecified; unspecified961.27
Neoplasms199.1Malignant neoplasm without specification of site; other861.14
162.9Malignant neoplasm; bronchus lung; unspecified730.97
202.8Other lymphomas; unspecified site, extranodal and solid organ sites710.94
Genitourinary System599Urinary tract infection, site not specified2473.27
584.9Acute renal failure, unspecified911.20
585.6End stage renal disease400.53
Digestive System578.9Hemorrhage of gastrointestinal tract, unspecified1191.57
558.9Other and unspecified noninfectious gastroenteritis and colitis690.91
577Acute pancreatitis360.48
Blood/Blood‐Forming Organs285.9Anemia, unspecified1271.68
282.64Sickle‐cell/Hb‐C disease with crisis801.06
282.6Sickle‐cell disease, unspecified730.97
Endo/Nutritional/Metabolic/Immunity276.1Hypoosmolality and/or hyponatremia570.75
251.2Hypoglycemia, unspecified560.74
250.1Diabetes with ketoacidosis; type II, not stated as uncontrolled500.66
Skin and Subcutaneous Tissue682.9Other cellulitis and abscess; unspecified site2563.39
682.5Other cellulitis and abscess; buttock370.49
686.9Unspecified local infection of skin and subcutaneous tissue230.30
Injury and Poisoning848.9Unspecified site of sprain and strain320.42
977.9Poisoning by unspecified drug or medicinal substance320.42
829Fracture; unspecified bone, closed220.29
Musculoskeletal/Connective Tissue730.2Unspecified osteomyelitis; site unspecified330.44
710Systemic lupus erythematosus250.33
728.87Muscle weakness (generalized)190.25
Infectious/Parasitic38.9Unspecified septicemia580.77
8.45Intestinal infection/clostridium difficile540.71
9.1Colitis, enteritis, and gastroenteritis of presumed infectious organ150.20
Mental Disorders291.81Alcohol withdrawal430.57
307.9Other and unspecified special symptoms or syndromes, not elsewhere classified350.46
294.8Other persistent mental disorders due to conditions classified elsewhere200.26
Nervous System/Sense Organs322.9Meningitis, unspecified300.40
331Alzheimer's disease140.19
340Multiple sclerosis60.08
Health Status/Contact with Health Services885.9Accidental fall from other slipping tripping or stumbling180.24
884.4Accidental fall from bed70.09
V13.02Personal history of urinary (tract) infection40.05
Pregnancy/Childbirth/Puerperium673.8Other pulmonary embolism; unspecified episode of care90.12
665Rupture of uterus before onset of labor; unspecified episode of care10.01
665.7Pelvic hematoma, unspecified episode of care10.01

Individual resident clinical experience varied as well. As shown in Table 4, for a group of 6 PGY‐1s, there was substantial variability in the ICD‐9 diagnostic categories. For example, the percentages of codes falling into the cardiovascular disease category ranged from 15.27% to 27.91%, and for respiratory disease ranged from 8.22% to 18.55%. These data suggest that there may be sizable differences in the proportions of various clinical conditions seen by residents over a year of training.

ICD‐9 Category Variability Among PGY‐1s
ICD‐9 Category DescriptionMeanSDMinMax
  • NOTE: To evaluate the extent of variability in diagnostic conditions seen by PGY‐1s based on their entry of ICD‐9 codes, we examined ICD‐9 data for 6 PGY‐1s over the time period of the study, calculated percentages in each ICD‐9 category, and evaluated the mean, standard deviation (SD), minimum (Min), and maximum (Max) values in each category.

  • Abbreviations: ICD‐9, International Classification of Diseases, Ninth Revision; PGY‐1s, postgraduate year‐1s.

Symptoms/Ill‐Defined Conditions21.435.0715.5029.90
Circulatory System21.844.3815.2727.91
Respiratory System12.433.838.2218.55
Genitourinary System5.
Digestive System4.530.963.095.65
Blood/Blood‐Forming Organs4.642.733.0510.05
Skin and Subcutaneous Tissue4.281.632.426.19
Injury and Poisoning3.901.013.095.43
Musculoskeletal/Connective Tissue2.861.361.554.58
Mental Disorders1.470.620.812.28
Nervous System/Sense Organs1.490.870.623.09


Years ago, residency training transitioned from a predominantly bedside experience to a curriculum with a large didactic, non‐bedside component, following parameters defined by organizations such as the Accreditation Council for Graduate Medical Education. Residency training is undergoing substantial change to become competency‐based and to organize learning around patient care experiences.2, 3, 9 The Educational Innovation Project of the Residency Review Committee for Internal Medicine is one such endeavor to help develop new methods by which to accomplish this.1 Effective incorporation of innovative experiential learning methods, based on the core competencies, will require a detailed knowledge of resident clinical experience during the course of their training, yet such data have been sparse in internal medicine. Sequist et al. analyzed data from an electronic medical record to assess resident clinical experience in the outpatient setting.4 Bachur and Nagler have used an electronic patient tracking system to assess the clinical experience of pediatric emergency medicine fellows.5, 6 Most attempts to describe resident clinical experience have relied upon extracting diagnostic information from medical records, case logs, etc, though in another approach, Rohrbaugh et al. reviewed psychiatric resident prescription profiles,7 which might provide some indirect data on clinical experience if applied to internal medicine.

In this study, we attempted to quantify resident clinical experience using resident‐selected ICD‐9 codes, in contrast to other methods that have relied upon medical record review and other resident‐independent approaches. There are various strengths and limitations to this approach. Using the ICD‐9 system provides a number of strengths, a major one being standardization, allowing comparisons between different programs and perhaps even facilitating the development of guidelines for resident clinical experience. In addition, this approach using the ICD‐9 system could be readily implemented at any institution and does not require any specific technology. While we chose to do this through our handoff system, an institution could use any of a variety of other systems to accomplish this. For example, resident‐entered ICD‐9 coding systems could be incorporated into electronic discharge summaries, history and physicals, or progress notes. There may also be some practical benefits to having residents learn how to use the ICD‐9 system at this stage of their careers.

There are limitations to this approach as well. The ICD‐9 system was not intended to be used for medical education purposes. There are features of it that can make finding the best diagnosis difficult, and routes to it may at times seem counterintuitive. While we did not carry out resident surveys, a number of residents anecdotally mentioned that it took time to become comfortable using the system, and it could be challenging at times to find a diagnosis description that best fit what they were looking for. To make diagnosis selection easier, we created an organ system‐based dropdown list in the handoff tool so that when residents select an organ system, another list opens up containing commonly used ICD‐9 codes. This grouping is based on organ system alone and does not necessarily follow the ICD‐9 grouping (in contrast, our reported data in this article are all based on ICD‐9 grouping). A search tool to allow searching the entire ICD‐9 database was also made available on the handoff tool. Other factors that could limit diagnosis code accuracy could be lack of clinical knowledge, and error as a result of pressure to come up with a diagnosis because of the hard stop design of our system, in which residents were required to enter a primary diagnosis, potentially causing alert fatigue. A validation assessment that we carried out revealed fairly good agreement with the specific ICD‐9 codes chosen by the resident, but greater accuracy would be desirable. Further education on diagnosis selection and refinements to the handoff tool should help facilitate this. We are currently addressing this by ongoing education on diagnosis selection and by having the hospitalists share the handoff tool with the residents, allowing them to provide direct feedback on diagnostic selections.

More than 19% of the diagnoses selected by residents fell into the category of symptoms and ill‐defined conditions. This raises a number of potential educational issues. One of those is that if residents do, in fact, encounter such entities at such a high frequency, then the internal medicine curriculum must be structured in such a way as to complement this clinical experience with a comprehensive learning program. However, we must also consider the possibility that, in many such instances, a more definitive diagnosis became evident by the time of discharge and this may not have been reflected in the ICD‐9 code that the resident chose. Hence, the category of symptoms and ill‐defined conditions may actually be somewhat smaller than our findings would suggest.

Many issues will need to be addressed as programs obtain more data on their residents' clinical experience. While there may be many reasons to use the ICD‐9 system for selecting diagnoses including those listed above, the system by which ICD‐9 groups diagnoses might not provide ideal educational information, again as the ICD‐9 system was not designed for this purpose. While in this article we have reported the residents' diagnostic encounters grouped according to the ICD‐9 grouping system to provide an initial standardized description, grouping according to another diagnostic system that is felt to be more educationally meaningful may be preferred.

While one might assume that a higher frequency of exposure to certain clinical conditions should enhance competency, that relationship may not be straightforward in internal medicine. For surgical procedures, there are, in fact, data to show improved outcomes for surgeons with higher operative volumes for those procedures,10 but in internal medicine, we do not have data to demonstrate that competence of a resident caring for a particular condition is enhanced by experience alone. Therefore, as programs obtain more data on clinical experience, it will be important that the focus be kept on quality as opposed to quantity.

Obtaining data on resident clinical experience might greatly facilitate experiential learning approaches. For example, as residents go through training and encounter specific diagnostic conditions, those experiences could be supplemented by various learning innovations to make those experiences more meaningful and, hopefully, more likely to result in the development of competence, though that will require measurement. In our program, for example, we have incorporated an approach using illness scenarios, in that when residents have had a certain level of clinical experience with a given clinical condition, they are assembled in small groups and competency‐based case discussions are carried out with a preceptor. In addition, for those instances in which an individual resident may lack direct clinical experience in a certain area, this might be addressed by interventions to increase their contact with those conditions and/or targeted learning interventions to help develop competence. A resident found to be lacking in clinical experience in a certain area could be assigned to the care of more patients with that condition, or to spending more time in a venue in which that condition is more likely to be encountered. Various learning activities including didactics, case discussions, simulation, self‐directed learning, and others could also be used to compensate for such variability. Furthermore, if a residency program's aggregate clinical experience is divergent from some desirable standard yet to be determined, a detailed knowledge of this could help guide that program's curriculum revision. For example, for residents in a program in which there is relatively low exposure to patients with oncological issues, this could be compensated for by external rotations to achieve more clinical experience in oncology, as well as supplementation of the curriculum with additional learning activities in oncology, which could include small group discussions, self‐directed learning activities, case discussions, and others. While at present there are no defined standards for clinical experience and it remains to be seen if there would be a correlation with development of competence, no such standard would serve a purpose if programs did not have reliable and practical means of clinical experience assessment.

In summary, resident‐selected ICD‐9 codes may be a useful means to obtain data regarding resident clinical experience in internal medicine. Such data may be useful to residency training programs in developing new curricula based on experiential learning.


Online-Only Materials

PDF icon Supporting Information (1)116.46 KB
   Comments ()