CME Office: CME Column

• Grand Rounds: What's in it for YOU?
• Major Players in Continuing Medical Education
• The Consortium for Academic CME Earns Six Year Accreditation from the ACCME

As printed in the September 2001 issue of the Jefferson Medical College Office of Health Policy Newsletter

The national accreditation body for continuing medical education, The Accreditation Council for Continuing Medical Education (ACCME), has increased its emphasis on the 'outcomes' of CME. In order to attain the highest level of accreditation, a provider must design activities that have documented outcomes either in terms of changes in physician behavior or improvement in the healthcare outcome for the patients of those physicians. For CME providers who certify grand rounds activities, this is a challenging task.

Grand Rounds are defined by the ACCME as a "series typically offered in a one hour regularly recurring sessions and designated for credit as one activity." (1) They should be evaluated as a single activity that follows a planned curriculum over the course of time. Since Grand Rounds topics are often so diverse, it is difficult to apply the evaluation requirement set forth by the ACCME.

The literature on the evaluation of grand rounds is relatively sparse. Over the past 10-15 years, there have been just a few articles published that directly address grand rounds as educational activities for physicians beyond their training years.

In 1985 Richmond published his findings on the "Educational Value of Grand Rounds"(2). He reported an abundance of literatiure on the effectiveness of small group learning and lectures but a dearth of data on the effectiveness of grand rounds as a learning experience. Richmond, in Auckland, New Zealand, had conducted two surveys of grand rounds attendees at his hospital. He found that attendee recall of content was inconsistent and that incorrect conclusions from the discussions within the presentations could be made by attendees. He did not attempt to evaluate the participant's ability to use information presented or gained during the sessions.

Hull et al (3), in 1989, presented results of a descriptive study of various grand rounds held within the medical school and affiliated hospitals of the Case Western Reserve University (CWRU) in Ohio. The article culled information from the CWRU CME Office records and also surveyed key physician leaders of these grand rounds to ascertain the "nature" of the "typical grand rounds." They found Grand Rounds to be a significant source of CME for the physician attendees; they "presumed (the practicing physicians who attended) are there to update their knowledge and skills"; they discussed the use of required syllabi/hand out materials for each session as an indicator of forethought that "probably" leads to increased educational value.

In 1990 Parrino and White (4) undertook a survey of all US Medical School Depts of Medicine and their Departmental grand rounds. They queried Chairs of Medicine (or their designees) about the format, objectives, and popularity of grand rounds, and inquired as to who attended and what changes the chairs had noted over time. One interesting statement in this brief survey report is the "the educational impact of grand rounds is infrequently assessed" in spite of the high regard with which chairs and other respondents claim to hold the activity, and in spite of the amount of resources designated to the support of Medicine Grand Rounds at these institutions.

That same year, McLeod and Gold (5) published a similar study of Medicine Grand Rounds at Canadian teaching hospitals. They queried Chairs of Medicine about current practices (objectives, content, clinical vs basic science emphasis, audience participation, format), their perceptions of changes and issues, and provided an opportunity for comment on any "areas of grand rounds not addressed in the questionnaire." Results were similar to Parrino and White in terms of objectives and content though with slight differences in formats used (ie, Canadians reported a dominant use of case presentation format, though patients themselves were rarely present). Evaluation of the effectiveness of these GR in meeting their objectives was not addressed in this survey.

In 1995 Lewkonia and Murray in Calgary (Alberta/Canada) (6) conducted a survey to assess the perceptions of the importance and educational purpose of grand rounds among physician planners and administrators in that city, and to examine organizational aspects of GR as educational events. To this end, they surveyed all teaching hospitals in Calgary. Similar to Parrino's findings, Lewkonia et al noted contradictions in the data they analyzed. They note the respondents placed a high value on the perceived importance of grand rounds but few reported using a curricular structure in place for planning grand rounds, basing topic selection on demonstrated needs of participants. Further, they note there is "little interest is shown in the educational structure or the evaluation of learning". In their literature review they note that "solid evidence for educational efficacy of traditional grand rounds is lacking in the literature" and call for a "philosophical and methodological evaluation of traditional grand rounds models."

Most recently (1999) Boucek et al (7) presented the results of a national survey on Anesthesia Dept Grand Rounds. It was intended to assess the timing, frequency, format, audience makeup and accreditation (CME) status of the key Anesthesia Department education activity in institutions across the US. The survey reports the "usual methods" of lecture evaluation - anonymous completion of a set form, with a poor return rate cited as an issue of concern. The report goes on to identify factors that affect outcomes (such as frequency of education, intensity, timing; and the value of repetitive sessions with links to educational tools like reminders, feedback). It concludes that "Many programs need to more carefully document mechanisms for dealing with conflict of interest and program evaluation."

Why is the evaluation of Grand Rounds important? The latest available ACCME Report (Summer 2001, with data for 2000) (8) shows that medical schools, only one type of provider of CME approved by the ACCME, provided 36% of all reported activities in 2000. However, medical schools provided 54% of all Grand Rounds-type activities. In addition, in 2000 Grand Rounds-type activities accounted for over half (64%) of all Medical School CME credit hours, with episodes of participation totaling 1,494,727 physicians and 312,595 other health care-related learners.

While there are many solid rationales for grand rounds (training of students, residents, fellows; socialization to the culture of medicine, modeling of clinical problem solving, etc) it is important to remember that these are certified educational venues. It is possible for a practicing physician to obtain all required annual CME hours by attending local grand rounds. Yet there is no clear understanding of how participation in grand rounds affects the practicing physician's continuing professional development, practice behavior, or patient outcomes.

This is the challenge facing medical school providers of CME, and as a result, Jefferson Medical College is re-examining how it looks at the effectiveness of grand rounds. We welcome your thoughts on this topic as we formulate new approaches to evaluation of grand rounds. If you wish to comment, please contact Jeanne Cole at jeanne.cole@mail.tju.edu.

Literature Cited

1. ACCME. ACCME's Accreditation Handbook. August 1999
2. Richmond DE. The educational value of grand rounds. N Zeal Med J 1985; 98; 280-82
3. Hull AL, Cullen RJ, Hekelman FP. A retrospective analysis of grand rounds n continuing medical education. Journal of Continuing Education in the Health professions 9;
4. Parrino TA, White AT. Grand rounds revisited: results of a survey of US Departments of Medicine. Am J Med 1990; 89 p 491-495.
5. McLeod PJ; Gold P. Medical grand rounds: alive and well and living in Canada. Can Med Assoc J 1990 142 (10) 1053-1056.
6. Lewkonia RM, Murray FR. Grand rounds: a paradox in medical education. Can Med Assoc J 1995. 152(3), 371-376
7. Boucek CD, Wilks DH, Barnes B, Freeman JA. Anesthesiology grand rounds: a nationwide survey. Am J Anesthesiology 1999 26 (4), 167-170.
8. ACCME Report accessed on September 24, 2001 at http://www.accme.org/incoming/2000_Annual_Data_Tables.pdf

Major Players in Continuing Medical Education
Jeanne G. Cole, MS, Managing Director, Office of CME

As printed in the March 2000 issue of the Jefferson Medical College Office of Health Policy Newsletter

This month we focus on some of the major players in the world of CME, specifically the two largest organizations involved in physician's continuing medical education. Certification of activities for Category 1 credit involves two different national organizations with collaborative rules: The Accreditation Council for Continuing Medical Education and the American Medical Association.

The Accreditation Council for Continuing Medical Education (ACCME) manages the "accreditation" system for the United States. It approves organizations to sponsor continuing medical education for physicians. All organizations wishing to be approved must meet a set of national standards, known and as the Essential Areas and Standards and go through periodic review and reapproval to retain their status as organizations accredited to sponsor CME for physicians. A goal of accreditation is to maintain certain quality standards, with the accrediting organization (The ACCME in this case) encouraging self-improvement in achieving and/or maintaining these quality standards (1).

The American Medical Association (AMA) sets the criteria that organizations accredited by the ACCME must follow in order to certify that an activity meets the Essential Areas and Standards. It determines the types of activities an accredited organization can certify for Category 1 or Category 2 credit, and what physicians may self-report and claim for Category 1 or 2 credit from informal educational activities. While the AMA issues a "Physician Recognition Award" (PRA) which encourages all physicians to participant in a balance personal education program, many states and health care organizations require physicians to participate in certified Category 1 activities to retain hospital/insurance credentialling and state and/or specialty society membership. Some state require ongoing, documented participation in Category 1 credit activities in order to maintain licensure (2)

The AMA PRA Information Booklet (2) identifies six specific types of activities that may be designated by an ACCME-accredited sponsor for Category 1 credit:

• Formal learning activities sponsored by accredited sponsors
• CME activities in journals
• CME enduring materials (monographs, CD ROM, audio tapes, etc)
• International conferences approved by the AMA
• Passing a recertification exam
• Participating in an ACGME accredited program (as residents)
• Study leading to a medically related degree (an MPH for example)

• Preparation of articles published in peer--reviewed journals (defined as journals included in Index Medicus) for one article per year, up to 10 category 1 credits

• Poster preparation for an exhibit at a medial meeting designated for AMA PRA category 1 credit, with a published abstract, for one poster presentation/year up to five category 1 credits

• Teaching in activities designated for AMA category 1 or 2, for up to 10 credits per year with 2 category 1 credits for each of hour of lecture

Who does what?
There is frequently confusion as to how an individual activity ultimately receives the desired Category 1 credit status. The actual designation of credit is done by an organization (such as a medical school or hospital) formally recognized by the ACCME as a sponsor of CME for physicians. This organization "certifies" that the activity meets BOTH all the ACCME Essentials AND all the AMA requirements.

This ability to 'certified' activities is effectively 'delegated' to accredited sponsors by the ACCME and AMA. Both the ACCME and the AMA monitor the work of accredited organizations via different vehicles. The ACCME holds primary responsibility for this monitoring, accomplished through

• annual reports on aggregate data by all accredited sponsors,
• reaccreditation reviews [reporting on all certified activities given by an accredited sponsor] including site surveys,
• Inquiry from the ACCME's "External Monitoring Committee" which investigates complaints and inquiries referred to it.

References:

1. Scriven, M. Evaluation Thesaurus (Sage Publications, Newbury Park CA), 1991, page 46-47.
2. AMA. The Physician's Recognition Award Information Booklet (Chicago IL), February 1999.
3. Wentz D, Scotti Jr, M, Osteen A: CME/CPPD: Headus Up from the AMA. Presented at 25th Annual Conference of the Alliance for CME, January 22, 2000 (New Orleans LA)

The Consortium for Academic CME Earns Six Year Accreditation from the ACCME
Jeanne G. Cole, MS, Managing Director, Office of CME

As printed in the October 2000 issue of the Jefferson Medical College Office of Health Policy Newsletter

The Consortium for Academic Continuing Medical Education (CACME), of which Jefferson is a founding member, is a collaborating group of four medical schools in Pennsylvania dedicated to designing, delivering, and evaluating quality educational experiences/opportunities that stimulate, educate and empower physicians to provide the highest standard of care throughout the lifetime of professional practice. The four member medical schools are Thomas Jefferson University, Penn State College of Medicine, Temple University, and University of Pittsburgh.

The ACCME, at its July Meeting, approved CACME for accreditation, awarding it a six year accreditation, an achievement reached by only a few accredited CME sponsors (only 2% of the July decisions included a six-year accreditation period) (1). CACME's approach to accreditation and certification also earned exemplary ratings in several of the Essential Areas and Standards, and were recognized in the Summer 2000 ACCME Report as a source of information on "Best Practices" in CME (2).

Exemplary ratings in Educational Planning and Evaluation singled out "innovative and creative planning processes" as well as "significant changes in processes for Grand Rounds." For the area of Administration, the report cited both "demonstrated remarkable levels of organization, flexibility, accommodation, governance and management," as well as "a range of innovative and creative practices: in the areas of disclosure and commercial support."

How does this affect CME at Jefferson and the other CACME institutions? Now that CACME's approach to CME certification for medical schools has been accepted by the ACCME, we will integrate CACME philosophy, policies and procedures more fully into our institutional accreditation processes. The primary goal is to emphasize the educational aspects of CME: intent and evaluation of achievement of intent. A corollary goal is to streamline the certification process and reduce paperwork.

The Concepts
There are three key concepts that will impact how CME activities are administered and approved for credit at JMC:

• Preplanning
• Risk Stratification
• Performance Monitoring

The pre-planning philosophy affects needs assessment documentation as well. CACME identifies three functions for needs assessment and places them in a temporal relationship to planning as follows:

Function	Temporal relationship to planning	Source	Value Added	Limitations
Defining the need	Prospective	Stakeholder Sponsoring Organization Learner CME Office	Identifies unaddressed gaps and opportunities	Ability to actualize the activity depends on the interest of the sources
Refining the need	Concurrent	CME Office in collaboration with others	Makes the need more targeted, uses a variety of data sources, defines the intent and curriculum, sets the evaluation standards	Requires investment on the part of the CME Office and collaboration with a variety of constituencies
Justifying the need	Retrospective or non-concurrent	CME Office and others	None	Questionable use of resources

This philosophy places a higher value on 'defining' and 'refining' needs (prospective and concurrent activities) than on "justifying" needs (a retrospective activity).

Risk Stratification
The CACME risk stratification tool attempts a prospective assessment of each proposed activity in terms of compliance with the Standards for Commercial Support. Various criteria are evaluated to develop a 'risk score' that is categorized to a range from "low" to "very high" on a scale of 1 to 4. The level of risk assessed then factors into the level of CME Office and/or CME Committee involvement in the implementation of an approved activity. The CACME Risk Stratification tool outlines primary monitoring responsibilities for various risk levels and includes appropriate monitoring and implementation activities for high and very high risk activities.

Performance Monitoring System
The centerpiece of the CACME system is the Performance Monitoring System, an ACCESS-based database that will ultimately allow JMC, CACME and the ACCME to review activity and program data in a variety of 'slices' at various points during its implementation, not just retrospectively. Performance Monitoring collects data from three distinct portions of certified activities: pre-planning, compliance, and evaluation/outcomes. CACME has been developing a series of standard criteria and accompanying quantifiable data elements, as well as taking beginning steps in developing standard reporting protocols.

CACME Policies
CACME Policies reduce the level of paper documentation by recognizing the unique position of medical schools as educational leaders/sources. CACME Policies are based on RISK category, with minimum requirements for low risk activities and increasing requirements for high and very high risk category activities. Time and cost savings for both sponsoring departments and the Office of CME are anticipated. In a nutshell, the higher the risk, the higher the need to monitor and document an activity.

For example, using the Risk Stratification tool, most on-campus Grand Rounds would likely be identified as a low or medium risk activity. CACME Policy accepts that a CACME member school should accept its Faculty Program Director's judgement as to the qualifications of the speakers he/she selects and requires only that a written verification of this be obtained from the Faculty Program Director. It assumes the program director is in a position to evaluate the credentials of speakers for its specialty area, and recognizes the lack of value-added by having a retrospective review of Program Director choices by Office of CME staff. The paperwork and time savings will be substantial for both sponsoring departments and the OCME.

CACME's Series Policy/Procedure, singled out for commendation by the ACCME, further emphasizes a curricular approach to series - an overall needs assessment (in the 'refining' category ) vs. an individual session needs assessment (usually in the 'justifying' mode). It is, in essence, a prospective approach vs a retrospective one. How grand rounds function at Jefferson and other CACME institutions will be a topic for a future column.

References:

1. ACCME Report 9:2, 3, 2000.
2. ACCME Report 9:2, 4, 2000.

(return to top)