Background: The Assessment of Multiple Systematic Reviews (AMSTAR) tool has become the most widely used tool for investigating the methodological quality of systematic reviews (SR). Originally, AMSTAR was developed for SRs of RCTs for evaluating treatment interventions. Its applicability to SRs of other study designs remains unclear. Our objectives were to: 1) analyze how AMSTAR is applied by authors and (2) analyze whether the authors pay attention to the original purpose of AMSTAR and for what is has been validated.

Methods: We searched Medline (via PubMed) from inception through October 2016 to identify studies that applied AMSTAR to at least one included review. Full-text were sought for all retrieved hits and screened by one reviewer. A second verified the excluded studies (liberal acceleration). Data were extracted into structured tables by one reviewer and were checked by a second reviewer. Discrepancies at any stage were resolved by consensus or consulting a third person. We analyzed the data descriptively as frequencies or medians and interquartile ranges (IQRs). Associations were quantified using the risk ratio (RR), with 95% confidence intervals.

Results: We identified 247 studies, which included a median of 17 reviews (interquartile range (IQR): 8 to 47). AMSTAR was modified in 23% (57/247) of studies. In most studies an AMSTAR score was calculated (200/247; 81%). Methods for calculating an AMSTAR score varied, with summing up all yes answers (yes=1) being the most frequent option (102/200; 51%). More than one third of authors failed to report how the AMSTAR score was obtained (71/200; 36%). In a subgroup analysis we compared overviews of reviews (n=154) with methodological publications (n=93) of our sample. The strongest associations were observed for mentioning limitations. Overviews of reviews were much less likely to mention both, limitations with respect to study designs (if other studies than RCTs were included in reviews) (RR 0.27, 95% CI 0.09 to 0.75) and overall score (RR 0.08, 95% CI 0.02 to 0.35).

Conclusions: Authors, peer reviewers and editors should pay more attention to the correct use and reporting of assessment tools in evidence synthesis. Authors of overviews of reviews should ascertain to have a methodological expert in their review team.

The complete article "How is AMSTAR applied by authors - a call for better reporting" can be downloaded at the website of "German Medical Science" (charges may apply).