Original Research

Methodological quality of national guidelines for pediatric inpatient conditions



Guidelines help inform standardization of care for quality improvement (QI). The Pediatric Research in Inpatient Settings network published a prioritization list of inpatient conditions with high prevalence, cost, and variation in resource utilization across children's hospitals. The methodological quality of guidelines for priority conditions is unknown.


To rate the methodological quality of national guidelines for 20 priority pediatric inpatient conditions.


We searched sources including PubMed for national guidelines published from 2002 to 2012. Guidelines specific to 1 organism, test or treatment, or institution were excluded. Guidelines were rated by 2 raters using a validated tool (Appraisal of Guidelines for Research and Evaluation) with an overall rating on a 7‐point scale (7 = the highest). Inter‐rater reliability was measured with a weighted kappa coefficient.


Seventeen guidelines met inclusion criteria for 13 conditions; 7 conditions yielded no relevant national guidelines. The highest methodological‐quality guidelines were for asthma, tonsillectomy, and bronchiolitis (mean overall rating 7, 6.5, and 6.5, respectively); the lowest were for sickle cell disease (2 guidelines) and dental caries (mean overall rating 4, 3.5, and 3, respectively). The overall weighted kappa was 0.83 (95% confidence interval 0.78–0.87).


We identified a group of moderate to high methodological‐quality national guidelines for priority pediatric inpatient conditions. Hospitals should consider these guidelines to inform QI initiatives. Journal of Hospital Medicine 2014;9:384–390. © 2014 Society of Hospital Medicine

© 2014 Society of Hospital Medicine

Researchers from the Pediatric Research in Inpatient Settings (PRIS) network, an open pediatric hospitalist research network,[1] have identified inpatient pediatric medical and surgical conditions considered high priority for quality improvement (QI) initiatives and/or comparative effectiveness research based on prevalence, cost, and interhospital variation in resource utilization.[2] One approach for improving the quality of care within hospitals is to operationalize evidence‐based guidelines into practice.[3] Although guidelines may be used by individual clinicians, systematic adoption by hospitals into clinical workflow has the potential to influence providers to adhere to evidence‐based care, reduce unwarranted variation, and ultimately improve patient outcomes.[3, 4, 5, 6]

There are critical appraisal tools to measure the methodological quality, as defined by the Institute of Medicine (IOM) and others in their guidelines.[7, 8, 9, 10, 11, 12] One such validated tool is the AGREE II instrument, created by the AGREE (Appraisal of Guidelines for REsearch and Evaluation) collaboration.[13, 14] It defines methodological quality as the confidence that the biases linked to the rigor of development, presentation, and applicability of a clinical practice guideline have been minimized and that each step of the development process is clearly reported.[13]

The objective of our study was to rate the methodological quality of national guidelines for 20 of the PRIS priority pediatric inpatient conditions.[2] Our intent in pursuing this project was 2‐fold: first, to inform pediatric inpatient QI initiatives, and second, to call out priority pediatric inpatient conditions for which high methodological‐quality guidelines are currently lacking.


The study methods involved (1) prioritizing pediatric inpatient conditions, (2) identifying national guidelines for the priority conditions, and (3) rating the methodological quality of available guidelines. This study was considered nonhuman‐subject research (A. Johnson, personal e‐mail communication, November 14, 2012), and the original prioritization study was deemed exempt from review by the institutional review board of the Children's Hospital of Philadelphia under 45 CFR 46.102(f).[2]

Prioritizing Pediatric Inpatient Conditions

Methods for developing the prioritization list are published elsewhere in detail and briefly described here.[1] An International Classification of Diseases, 9th Revision, Clinical Modification‐based clinical condition grouper was created for primary discharge diagnosis codes for inpatient, ambulatory surgery, and observation unit encounters accounting for either 80% of all encounters or 80% of all charges for over 3.4 million discharges from 2004 to 2009 for 38 children's hospitals in the Pediatric Health Information Systems (PHIS) database, which includes administrative and billing data.[15] A standardized cost master index was created to assign the same unit cost for each billable item (calculated as the median of median hospital unit costs) to allow for comparisons of resource utilization across hospitals (eg, the cost of a chest x‐ray was set to be the same across all hospitals in 2009 dollars). Total hospital costs were then recalculated for every admission by multiplying the standardized cost master index by the number of units for each item in the hospital bill, and then summing the standardized costs of each line item in every bill. Conditions were ranked based on prevalence and total cost across all hospitals in the study period. The variation in standardized costs across hospitals for each condition was determined.

For the current study, conditions were considered if they had a top 20 prevalence rank, a top 20 cost rank, high variation (intraclass correlation coefficient >0.1) in standardized costs across hospitals, a minimum number of PHIS hospitals with annualized overexpenditures (using the standardized cost master) of at least $50,000 when compared to the mean, or a minimum median of 200 cases per hospital over the 6‐year study period to assure sufficient hospital volume for future interventions. This resulted in 29 conditions; the selected 20 conditions matched the top 20 prevalence rank (see Supporting Information, Table 1, in the online version of this article).[2]

Overall Methodological Quality Ratings of Guidelines for the PRIS Network 20 Priority Conditions With High Prevalence, Cost, and Variability in Resource Utilization
Condition by PRIS Priority RankGuidelines Meeting Inclusion CriteriaaGuidelines CitationMean Overall Reviewer Methodological Quality Rating (Rater 1, Rater 2)bRecommended for Use in the Pediatric Inpatient Setting, Mean (Rater 1, Rater 2)cWeighted Kappa(95% Confidence Interval)
  • NOTE: Abbreviations: m, medical; PRIS, Pediatric Research in Inpatient Settings; s, surgical.

  • Inclusion criteria include national guideline published 20022012, describing pediatric inpatient medical or surgical management for given condition. Guidelines specific to an organism, test, or treatment or condition prevention alone were excluded.

  • Overall methodological quality rating on the AGREE II instrument, using a 7point scale: 1=lowest, 7=highest.

  • Recommended for use scoring on a 3point scale: 1=not recommended, 2=recommended with modifications, 3=recommended.

Otitis media, unspecified, s1American Academy of Family Physicians; American Academy of Otolaryngology‐Head and Neck Surgery; American Academy of Pediatrics Subcommittee on Otitis Media With Effusion. Clinical Practice Guidelines: Otitis media with effusion. Pediatrics. 2004 May;113(5):1412‐29.6 (6, 6)3 (3, 3)0.76 (0.490.93)
Hypertrophy of tonsils and adenoids, s1Baugh RF et al. Clinical practice guideline: tonsillectomy in children. Otolaryngol Head Neck Surg. 2011;144(1 suppl):S1S30.6.5 (7, 6)3 (3, 3)0.49 (0.050.81)
Asthma, m1National Heart, Lung, and Blood Institute; National Asthma Education and Prevention Program. Expert panel report 3 (EPR‐3): guidelines for the diagnosis and management of asthma‐full report 2007. Pages 1‐440. Available at: http://www.nhlbi.nih.gov/guidelines/asthma/asthgdln.pdf Accessed: 8/24/20127 (7, 7)3 (3, 3)0.62 (0.210.87)
Bronchiolitis, m1American Academy of Pediatrics Subcommittee on Diagnosis and Management of Bronchiolitis. Diagnosis and management of bronchiolitis. Pediatrics. 2006;118:17741793.6.5 (6, 7)3 (3, 3)0.95 (0.871.00)
Pneumonia, m1Bradley JS et al.The management of community‐acquired pneumonia in infants and children older than 3 months of age: clinical practice guidelines by the Pediatric Infectious Diseases Society and the Infectious Diseases Society of America. Clin Infect Dis. 2011;53(7):e25e76.6 (6, 6)3 (3, 3)0.82 (0.640.96)
Dental caries, s1American Academy on Pediatric Dentistry Clinical Affairs CommitteePulp Therapy Subcommittee; American Academy on Pediatric Dentistry Council on Clinical Affairs. Guideline on pulp therapy for primary and young permanent teeth. Pediatr Dent. 2008;30:170174.3 (3, 3)1.5 (1, 2)0.51 (0.140.83)
Chemotherapy, m0
Cellulitis, m1Stevens DL et al. Practice guidelines for the diagnosis and management of skin and softtissue infections. Clin Infect Dis. 2005;41:13731406.4.5 (4, 5)2.5 (2, 3)0.52 (0.150.79)
Inguinal hernia, s0
Gastroesophageal reflux and esophagitis, m, s2Vandenplas Y et al. Pediatric gastroesophageal reflux clinical practice guidelines: joint recommendations of NASPGHAN and ESPGHAN. J Pediatr Gastroenterol Nutr. 2009;49(4):498547.5 (5, 5)3 (3, 3)0.69 (0.450.87)
Furuta GT et al. Eosinophilic esophagitis in children and adults: a systematic review and consensus recommendations for diagnosis and treatment. Gastroenterology. 2007;133:13421363.5 (5, 5)2.5 (2, 3)0.93 (0.850.98)
Dehydration, m0
Redundant prepuce and phimosis, s1American Academy of Pediatrics Task Force on Circumcision. Male circumcision. Pediatrics. 2012;130(3):e756e785.6 (6, 6)3 (3, 3)0.66 (0.250.89)
Abdominal pain, m0
Other convulsions, m0
Urinary tract infection, m1Roberts KB et al. Urinary tract infection: clinical practice guideline for the diagnosis and management of the initial UTI in febrile infants and children 2 to 24 months. Pediatrics. 2011;128:595610.5.5 (5, 6)2.5 (2, 3)0.62 (0.230.84)
Acute appendicitis without peritonitis, s1Solomkin JS et al. Diagnosis and management of complicated intra‐abdominal infection in adults and children: guidelines by the Surgical Infection Society and the Infectious Diseases Society of America. Clin Infect Dis. 2010;50:133164.4.5 (5, 4)2.5 (3, 2)0.37 (0.110.81)
Eso‐ exo‐ hetero‐, and hypertropia, s0
Fever, m0
Seizures with and without intractable epilepsy, m3Brophy GM et al; Neurocritical Care Society Status Epilepticus Guideline Writing Committee. Guidelines for the evaluation and management of status epilepticus. Neurocrit Care. 2012;17:323.5 (5, 5)3 (3, 3)0.95 (0.870.99)
Hirtz D et al. Practice parameter: treatment of the child with a first unprovoked seizure: report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2003;60:166175.5 (5, 5)2.5 (2, 3)0.73 (0.410.94)
Riviello JJ Jr et al. Practice parameter: diagnostic assessment of the child with status epilepticus (an evidence‐based review): report of the Quality Standards Subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology. 2006;67:15421550.5 (4, 6)2.5 (2, 3)0.80 (0.630.94)
Sickle cell disease with crisis, m2Section on Hematology/Oncology Committee on Genetics; American Academy of Pediatrics. Health supervision for children with sickle cell disease. Pediatrics. 2002;109:526535.3.5 (3, 4)1.5 (1, 2)0.92 (0.800.98)
National Heart, Lung, and Blood Institute, National Institutes of Health. The management of sickle cell disease. National Heart, Lung, and Blood Institute, National Institutes of Health, Bethesda, MD. Available at: http://www.nhlbi.nih.gov/health/prof/blood/sickle/sub_mngt.pdf. Revised June 2002.4 (4, 4)2.5 (2, 3)0.91 (0.800.97)

Identifying National Guidelines

We developed a search protocol (see Supporting Information, Table 2, in the online version of this article) using condition‐specific keywords and the following criteria: guideline, pediatric, 2002 to 2012. A medical librarian (E.E.) used the protocol to search PubMed, National Guidelines Clearing House, and the American Academy of Pediatrics website for guidelines for the 20 selected conditions.

We limited our study to US national guidelines published or updated from 2002 to 2012 to be most relevant to the 38 US children's hospitals in the original study. Guidelines had to address either medical or surgical or both types of inpatient management for the condition, depending on how the condition was categorized on the PRIS prioritization list. For example, to target inpatient issues, otitis media was treated as a surgical condition when the prioritization list was created, therefore guidelines included in our study needed to address surgical management (ie, myringotomy or tympanostomy tubes).[2] Guidelines specific to 1 organism, test, or treatment were a priori excluded, as they would not map well to the prioritization list, and would be difficult to interpret. Guidelines focusing exclusively on condition prevention were also excluded. Guidelines with a broad subject matter (eg, abdominal infection) or unclear age were included if they contained a significant focus on the condition of interest (eg, appendicitis without peritonitis), such that the course of pediatric inpatient care was described for that condition. Retracted or outdated (superseded by a more current version) guidelines were excluded.

An investigator (G.H.) reviewed potentially relevant results from the librarian's search. For example, the search for tonsillectomy guidelines retrieved a guideline on the use of polysomnography prior to tonsillectomy in children but did not cover the inpatient management or tonsillectomy procedure.[16] This guideline was excluded from our study, as it focused on a specific test and did not discuss surgical management of the condition.

Rating Methodological Quality of Guidelines

Methodological quality of guidelines was rated with the AGREE II tool by 2 investigators (G.H. and K.N.).[13, 17] This tool has 2 overall guideline assessments and 23 subcomponents within 6 domains, reflecting many of the IOM's recommendations for methodological quality in guidelines: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence.[8, 17]

The AGREE II tool rates each of the 23 subcomponent questions using a 7‐point scale (1=strongly disagree7=strongly agree). We followed the AGREE II user's manual suggestion in rating subcomponents as 1, indicating an absence of information for that question if the question was not addressed within the guideline.[13] The AGREE II user's manual describes the option of creating standardized domain scores; however, as the objective of our study was to assess the overall methodological quality of the guideline and not to highlight particular areas of strengths/weaknesses in the domains, we elected to present raw scores only.[13]

For the overall guideline rating item 1 (Rate the overall quality of this guideline.) the AGREE II tool instructs that a score of 1 indicates lowest possible quality and 7 indicates highest possible quality.[13] As these score anchors are far apart with no guide for interpretation of intermediate results, we modified the descriptive terms on the tool to define scores <3 as low quality, scores 3 to 5 as moderate quality, and scores >5 as high quality to allow for easier interpretation of our results. We also modified the final overall recommendation score (on a 3‐point scale) from I would recommend this guideline for use to I would recommend this guideline for use in the pediatric inpatient setting.[13, 17] A score of 1 indicated to not recommend, 2 indicated to recommend with modifications, and 3 indicated to recommend without modification.

Significant discrepancies (>2‐point difference on overall rating) between the 2 raters were to be settled by consensus scoring by 3 senior investigators blinded to previous reviews, using a modified Delphi technique.[18]

Inter‐rater reliability was measured using a weighted kappa coefficient and reported using a bootstrapped method with 95% confidence intervals. Interpretation of kappa is such that 0 is the amount of agreement that would be expected by chance, and 1 is perfect agreement, with previous researchers stating scores >0.81 indicate almost perfect agreement.[19]


The librarian's search retrieved 2869 potential results (Figure 1). Seventeen guidelines met inclusion criteria for 13 of the 20 priority conditions. Seven conditions did not have national guidelines meeting inclusion criteria. Table 1 displays the 20 medical and surgical conditions on the modified PRIS prioritization list, including overall guideline scoring, recommendation scores, and kappa results for each guideline. The highest methodological‐quality guidelines were for asthma,[20] tonsillectomy,[21] and bronchiolitis[22] (mean overall rating 7, 6.5, and 6.5, respectively). The lowest methodological‐quality guidelines were for 2 sickle cell disease guidelines[23, 24] and 1 dental caries guideline[25] (mean overall rating 4, 3.5, and 3, respectively). Seven guidelines were rated as high overall quality, and 10 guidelines were rated as moderate overall quality. Eight of the 17 guidelines[20, 21, 22, 26, 27, 28, 29, 30] were recommended for use in the pediatric inpatient setting without modification by both reviewers. Two guidelines (for dental caries[25] and sickle cell[23]) were not recommended for use by 1 reviewer.

Figure 1

Condition‐specific guideline search results. *Conditions may have been excluded for more that 1 reason.

As an example of scoring, a national guideline for asthma had high overall scores (7 from each reviewer) and high scores across most AGREE II subcomponents. The guideline was found by both reviewers to be systematic in describing guideline development with clearly stated recommendations linked to the available evidence (including strengths and limitations) and implementation considerations.[20] Conversely, a national guideline for sickle cell disease had moderate overall scores (scores of 3 and 4) and low‐moderate scores across the majority of the subcomponent items.[23] The reviewers believe that this guideline would have been strengthened by increased transparency in guideline development, discussion of the evidence surrounding recommendations, and discussion of implementation factors. A table with detailed scoring of each guideline is available (see Supporting Information, Table 3, in the online version of this article).

Agreement between the 2 raters was almost perfect,[19] with an overall boot‐strapped weighted kappa of 0.83 (95% confidence interval 0.780.87) across 850 scores. There were no discrepancies between reviewers in overall scoring requiring consensus scoring.


Using a modified version of a published prioritization list for inpatient pediatric conditions, we found national guidelines for 13 of 20 conditions with high prevalence, cost, and interhospital variation in resource utilization. Seven conditions had no national guidelines published within the past 10 years applicable for use in the pediatric inpatient setting. Of 17 guidelines for 13 conditions, 10 had moderate and 7 had high methodological quality.

Our findings add to the literature describing methodological quality of guidelines. Many publications focus on the methodological quality of guidelines as a group and use a standardized instrument (eg, the AGREE II tool) to rate within domains (eg, domain 1: scope and purpose) across guidelines in an effort to encourage improvement in developing and reporting in guidelines.[31, 32] Our study differs in that we chose to focus on the overall quality rating of individual guidelines for specific prioritized conditions to allow hospitals to guide QI initiatives. One study that had a similar aim to ours surveyed Dutch pediatricians to select priority conditions and used the AGREE II tool to rate 17 guidelines, recommending 14 for use in the Netherlands.[33]

Identifying high methodological‐quality guidelines is only 1 in a series of steps prior to successful guideline implementation in hospitals. Other aspects of guidelines, including the strength of the evidence (eg, from randomized controlled trials) and subsequent force and clarity (eg, use of must instead of consider) of recommendations, may affect clinician or patient adherence, work processes, and ultimately patient outcomes. Strong evidence should translate into forceful and clear recommendations. Authors with the Yale Guideline Recommendation Corpus describe significant variation in reporting of guideline recommendations, and further studies have shown that the force and clarity of a recommendation is associated with adherence rates.[34, 35, 36, 37] Unfortunately, current guideline appraisal tools lack the means to score the strength of evidence, and force and clarity of recommendations.[10]

Implementation science demonstrates that there are many important factors in translating best practice into improvements in clinical care. In addition to implementation considerations such as adherence, patient preferences, and work processes, variability in methodological quality, strength of evidence, and force and clarity of recommendations may be additional reasons why evidence for the impact of guidelines on patient outcomes remains mixed in the literature.[38] One recent study found that adherence to antibiotics recommended within a national pediatric community‐acquired pneumonia guideline, which had a high methodological‐quality score in our study, did not change hospital length of stay or readmissions.[29, 39] There are several possible interpretations for this. Recommendations may not have been based upon strong evidence, research methodology assessing how adherence to recommendations impacts patient outcomes may have been limited, or the outcomes measured in current studies (such as readmission) are not the outcomes that may be improved by adherence to these recommendations (such as decreasing antimicrobial resistance). These are important considerations when hospitals are incorporating recommendations from guidelines into practice. Hospitals should assess the multiple aspects of guidelines, including methodological quality, which our study helps to identify, strength of evidence, and force and clarity of recommendations, as well as adherence, patient preferences, work processes, and key outcome measures when implementing guidelines into clinical practice. A study utilizing a robust QI methodology demonstrated that clinician adherence to several elements in an asthma guideline, which also had a high methodological‐quality score in our study, led to a significant decrease in 6‐month hospital and emergency department readmission for asthma.[6, 20]

Our study also highlights that several pediatric conditions with high prevalence, cost, and interhospital resource utilization variation lack recent national pediatric guidelines applicable to the inpatient setting. If strong evidence exists for these priority conditions, professional societies should create high methodological‐quality guidelines with strong and clear recommendations. If evidence is lacking for these priority conditions, then investigators should focus on generating research in these areas.

There are several limitations to this study. The AGREE II tool does not have a mechanism to measure the strength of evidence used in a guideline. Methodological quality of a guideline alone may not translate into improved outcomes. Conditions may have national guidelines published before 2002, institution‐specific or international guidelines, or adult guidelines that might be amenable to use in the pediatric inpatient setting but were not included in this study. Several conditions on the prioritization list are broad in nature (eg, dehydration) and may not be amenable to the creation of guidelines. Other conditions on the prioritization list (eg, chemotherapy or cellulitis) may have useful guidelines within the context of specific conditions (eg, acute lymphoblastic leukemia) or for specific organisms (eg, methicillin‐resistant Staphylococcus aureus). We elected to exclude these narrower guidelines to focus on broad and comprehensive guidelines applicable to a wider range of clinical situations. Additionally, although use of a validated tool attempts to objectively guide ratings, the rating of quality is to some degree subjective. Finally, our study used a previously published prioritization list using data from children's hospitals, and the list likely under‐represents conditions commonly managed in community hospitals (eg, hyperbilirubinemia).[2] Exclusion of these conditions was not reflective of importance or quality of available national guidelines.


Our study adds to recent publications on the need to prioritize conditions for QI in children's hospitals. We identified a group of moderate to high methodological‐quality national guidelines for pediatric inpatient conditions with high prevalence, cost, and variation in interhospital resource utilization. Not all prioritized conditions have national high methodological‐quality guidelines available. Hospitals should prioritize conditions with high methodological‐quality guidelines to allocate resources for QI initiatives. Professional societies should focus their efforts on producing methodologically sound guidelines for prioritized conditions currently lacking high‐quality guidelines if sufficient evidence exists.


The authors thank Christopher G. Maloney, MD, PhD, for critical review of the manuscript, and Gregory J. Stoddard, MS, for statistical support. Mr. Stoddard's work is supported by the University of Utah Study Design and Biostatistics Center, with funding in part from the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant 8UL1TR000105 (formerly UL1RR025764).


Sanjay Mahant, Ron Keren, and Raj Srivastava are all Executive Council members of the Pediatric Research in Inpatient Settings (PRIS) Network. PRIS, Sanjay Mahant, Ron Keren, and Raj Srivastava are all supported by grants from the Children's Hospital Association. Sanjay Mahant is also supported by research grants from the Canadian Institute of Health Research and Physician Services Incorporated. Ron Keren and Raj Srivastava also serve as medical legal consultants. The remaining authors have no financial relationships relevant to this article to disclose.


Online-Only Materials

   Comments ()