Education Research: The challenge of incorporating formal research methodology training in a neurology residency
E. C. Leira, MD, MS,
M. A. Granner, MD,
J. C. Torner, PhD,
R. C. Callison, MD and
H. P. Adams, Jr, MD
From the Department of Neurology (E.C.L., M.A.G., R.C.C., H.P.A.), Carver College of Medicine, and the Department of Epidemiology (J.C.T.), College of Public Health, University of Iowa, Iowa City.
Address correspondence and reprint requests to Dr. Enrique C. Leira, Department of Neurology, 2147-RCP, University of Iowa Hospital, 200 Hawkins Drive, Iowa City, IA 52242 enrique-leira{at}uiowa.edu
Background: Physicians often do not have good understandingof research methodology. Unfortunately, the mechanism to achievethis important competency in a busy neurology residency programremains unclear. We tested the value and degree of acceptanceby neurology residents of a multimodal educational interventionthat consisted of biweekly teaching sessions in place of anexisting journal club, as a way to provide formal training inresearch and statistical techniques.
Methods: We used a pre- and post-test design with an educationalintervention in between using neurology residents at the Universityof Iowa as subjects. Each test had 40 questions of researchmethodology. The educational intervention consisted of a biweekly,structured, topic-centered, research methodology-oriented electiveseminar following a year-long predefined curriculum. An exitsurvey was offered to gather resident’s perceptions aboutthe course.
Results: While a majority of residents agreed that the interventionenhanced their knowledge of research methodology, only 23% attendedmore than 40% of the sessions. There was no difference betweenpretest and post-test scores (p = 0.40).
Conclusions: Our experience suggests that, in order to accomplishthe Accreditation Council for Graduate Medical Education goalsregarding increasing competency of residents in knowledge aboutresearch methodology, a major restructuring in the neurologyresidency curriculum with more intense formal training wouldbe necessary.
An understanding of research methodology is deemed crucial forboth a successful research career1 and for critically judgingpublications relevant to practice.2 In fact, the AccreditationCouncil for Graduate Medical Education has established as partof the practice-based learning and improvement competency thatresidents "must demonstrate an ability to a) locate, appraise,and assimilate evidence from scientific studies related to theirpatients’ health problems, b) obtain and use informationabout their own population of patients and the larger populationfrom which their patients are drawn, and c) apply knowledgeof study designs and statistical methods to the appraisal ofclinical studies and other information on diagnostic and therapeuticeffectiveness."3 Unfortunately, the mechanism to achieve theseimportant skills in a neurology residency program remains unclear.The already busy and strict neurology residency curriculum isa challenging environment for formally teaching research methodology.We proposed a targeted multimodal educational intervention usingthe time allocation of a biweekly journal club to provide trainingwhile avoiding a major intrusive modification in the other componentsof the curriculum. We evaluated the utility of the educationalprogram by assessing its acceptance by neurology residents,and testing for an increase in knowledge of research methodology.
We used a pre- and post-test design to test knowledge on researchmethodology in conjunction with an educational intervention.Neurology residents and fellows at the University of Iowa wereinvited to participate in this voluntary program that took placeduring one academic year (July 2006–June 2007). In July2006, the subjects were asked to complete a 40-question preteston a selected list of topics of research methodology. The actualquestionnaire is shown in the appendix. Faculty members of theDepartment of Neurology were invited to take the pretest forvalidation purposes.
The educational intervention consisted of an elective biweekly,structured, topic-centered, research methodology-oriented seminar.Each session was centered around a specific topic in researchmethodology. This curriculum is outlined in table 1. This programencouraged active resident participation with faculty supervision.Each research methodology topic was assigned in advance to oneresident. Her or his role included to prepare a short Powerpointpresentation about the topic, to select an article that wouldillustrate that particular topic, and to generate questionsin advance to illustrate important learning points from thatarticle. The residents discussed the Powerpoint presentation,article, and questions with the supervising faculty member (E.C.L.)in order to obtain feedback prior to the presentation. The articleand questions were given to all residents before the presentation.The sessions meant to encourage active participation and discussionthrough case-based learning.
Table 1 Topics covered in 1-year biweekly structured journal club
At the end of the series of seminars the participants were askedto complete a 40-question post-test consisting of the same questionsas the pretest but administered in a shuffled order. Followingthe post-test, residents were asked to complete an anonymoussurvey to gather their perceptions regarding the newly implementededucational intervention. We were particularly interested inperceived benefits for their career, subjective learning, andsuggestions for future improvement. Both tests and exit surveywere administered through WebSurveyor (WebSurveyor Corporation),a password-protected Web-based computer system, and scores werekept confidential. Comparisons between the number of correctresponses in different tests were done using a Wilcoxon ranksum test. Significance was established at the 5% level. Descriptivestatistics were used to analyze the responses to the survey.All statistical analyses were performed using SAS version 9.1(SAS Institute Inc., Cary, NC). This study was previously approvedby the University of Iowa Institutional Review Board.
Eighteen of 23 neurology residents completed the pretest, and10/23 residents completed the post-test. The mean number ofcorrect answers in the pretest was 24.17 (SD = 5.77). The meannumber of correct answers in the post-test was 25.58 (SD = 4.56)(p = 0.40). Using a paired t test for mean difference approximation,we retrospectively calculated that we had a power of 10% todetect a significant difference. The majority of the residentsattended fewer than 40% of the educational sessions. A totalof 10 faculty members of the Department of Neurology anonymouslycompleted the pretest. The mean number of correct answers inthe faculty pretest was 27 (SD = 4.59). Comparison between theresident’s and faculty pretest scores showed no differences(p = 0.16).
Nine neurology residents completed the exit survey. The resultsare shown in table 2. Two thirds of respondents agreed thatthe educational intervention enhanced their knowledge aboutresearch methodology. The majority did not perceive a benefitin regards to patient care. Residents were also asked to volunteeranonymous comments about the program. The residents were notenthusiastic about the seminar’s emphasis on statisticaltechniques.
We tried to address the widespread problem of deficient (ornonexistent) formal research methodology training in neurologyresidencies by proposing a simple intervention that would notfurther stretch the current busy schedule of a neurology trainingprogram.
The tested multimodal educational intervention did not improveknowledge about research methodology, although we recognizethe limitation of our small sample size, which limits the powerto detect a significant difference.
Equally disappointing was the moderate to low enthusiasm forthis new modality of training, judging by the resident’sresponses and attendance. Unfortunately, knowledge of researchmethodology is not a focus of the resident in-training examinationadministered by the American Academy of Neurology, a standardizedmeasure of resident’s performance, or a major focus ofthe American Board Psychiatry and Neurology certification examination,which might be a disincentive to study this field.
These negative objective and subjective results make us skepticalof the future acceptance of more complex and sustained interventionsto enhance understanding of research methodology. Protectedtime for formal teaching of biostatistics and epidemiology isnot easy to achieve in a currently highly regulated residencycurriculum.4 While the NIH provides K grants with required protectedtime to potential clinician-scientists so they obtain the necessaryformal training in research methodology, such assistance isvery limited and usually not available during residency.1 Residencyprograms that successfully train academic physicians need toinclude a period of mentored research, or facilitate participationin courses to enhance methodologic expertise.
A Canadian residency program has incorporated an evidence-basedmedicine (EBM) teaching program in their curriculum.5 Thesesessions are topic-centered, so the trainee is taught EBM inthe process of critically appraising and reviewing the availableliterature regarding the particular clinical question. Whilethese topic-centered EBM seminars are likely to be more acceptedby residents, and the skills taught crucial for clinical-decisionmaking, one can argue that they should be based on a solid foundationin research methodology. There are other important limitations.First, summaries of EBM only include a handful of conditions,2and most of the perceived "landmark" articles in neurology therapeuticswill be randomized clinical trials (RCT). Therefore, that approachto learning will likely result in a biased curriculum in whichRCT are overemphasized to the detriment of other type of studies,such as prospective cohorts, case-control, or community interventiontrials. In fact, a comprehensive assessment of biostatisticalknowledge among internal medicine residents has shown disappointingresults.2 Beyond the implications of potentially learning abiased curriculum, such an approach could also aggravate thecurrent national crisis in electing a clinical research career.1It is unlikely for a young investigator to start his or hercareer organizing a RCT, which are among the most costly andcomplex research experiments. On the other hand, an earlierexposure to other more "affordable" and feasible modalitiesfor research, including retrospective cohorts and case-controlstudies, might inspire neurologists in training to pursue similarstudies early in their careers. Potential solutions includeWeb-based courses, perhaps developed by the AAN, and a gradualexposure starting with more appealing topics such as EBM andclinical research design. We also suggest that restructuringthe neurology residency curriculum6 to allow for formal teachingelectives in research methodology7 would be necessary in orderto fully achieve the Accreditation Council for Graduate MedicalEducation competency goals. In particular, an improved understandingof clinical trial methodology and research is important forthe core competency of lifelong learning. In addition, knowledgeof research methodology is an important component of self-improvement,which also is a core component of maintenance of certification.
This is the table of results for a new test to diagnose CJD:
1) Which statement is correct?
a) Sensitivity of the testis 80%, specificity is 90%
b) Sensitivityof the test is 90%,specificity is 80%
c) Sensitivity of thetest is 20%, specificityis 10%
d) Sensitivity of the testis 10%, specificity is 20%
e) This test has equal sensitivityand specificity
2)All of these are components of the Hill criteria to makea casefor causal inference except:
a) Time sequence
b) Strengthof the association
c) Resultsdifferent than previous studies
d) Plausibility
e) Doseresponse or biological gradient
3) A case-control study is designed to see if there isan associationbetween trauma in the previous months and carotiddissection.There is a concern that subjects with carotid dissectionmighttend to report previous trauma better than the controls,andtherefore bias the results. This would be an example of:
a) Measurement error
b) Recall bias
c) Interviewer bias
d) Selection bias
e) Confounding bias
4) This is astudy assessing the risk of brain tumors in patientsexposedto previous radiation. What is the odds ratio of developinga brain tumor for those exposed to radiation?
a) OR = 1
b)OR = 2
c) OR = 3
d) OR = 4
e) OR = 5
5) Regardingthe chi-square test, all is true except
a) Is based in comparingthe expected and observed frequencies
b) Is a general testto find an association between exposureand outcome
c) Thehigher the chi-square statistic value,the stronger theassociation
d) chi-square can be only usedwith 4 x 4 cells
e) Resultsare expressed with a chi-squarevalue and p value
6) TheCochran-Mantel Test is:
a) A test to determine if a variableis homogeneous
b) A testto reduce the chance for selectionbiases
c) A test to determineif there a dose-response effectacrossdifferent levels
d)A test to adjust for multiple comparisons
e) A test to comparethe agreement between two observers
7) Everything is true about the Kaplan-Meier estimator except
a) Is a non-parametric test
b) Estimates risk over time
c)Typically used with survival analysis
e) Cannot be usedifsubjects drop out of the study for otherreasons
f) Yieldsan estimate of risk at any point in time
8) Everythingis true about logistic regression, except:
a) The responsevariable is usually continuous
b) The predictorvariablescan be of various types
c) The purpose is to determinehowone or more independent variablesare related to the rateofoccurrence of the binary outcome
d) Can calculate the probabilityof an outcome for a particularset of values of the predictorvariables
e) Can calculate odd ratios of the outcome for twodifferentvalues of a predictor
9) For which of these applicationswould a logistic regressionanalysis be most useful?
a) Clinicaltrial comparing the effect of treatment/placeboon stroke volumeon MRI
b) A prospective cohort assessingthe usefulness andweightof different predictive variablesin predicting the developmentof Alzheimer disease
c) A case-controlstudy testing an associationof AED exposurewith birth controldefects
d) A survey amongneurology residents about theirknowledgen research methodology
e) A comparison study ofidentification of signs of earlyischemiaon CT between radiologyresidents and neurology residents
10) When a meaningfully different interpretation of therelationshipof interest occurs if an extraneous variable isignored or includedin the analysis, we call that concept
a)Interaction
b) Confounding
c) Interference
d) Effectmodifier
e) Bias
11) All about the kappa statistic is true except:
a) Is a measure of reliability
b) Measures agreement betweentwo observers
c) Compares two categorical measures
d) Comparesthe observed agreement with the expected agreementby mere chance
e) A Kappa of 0.35 is considered excellent
12) Regardingrates, all is true except:
a) Crude rates estimate the actualdisease frequency for a population
b) Crude rates can be usedto provide public health planning
c) Crude rates can be misleadingif compared over time oracrosspopulations
d) Adjusted ratesrepresent actual diseasefrequency in thepopulation studies
e) Adjusted rates accountfor differences in population characteristics
13) Everything is true about simple linear regression,except:
a) The response variable is continuous
b) Triesto find thestraight line that best fit the data
c) The purposeis todetermine the relationship between theindependent variablesand the predictor variable
d) Can predict the value of theresponse variable for any valueof the predictor
e) Can calculateodds ratios of the outcome of interest
14) For which ofthese applications would a linear regressionanalysis be mostuseful?
a) Clinical trial comparing the effect of treatment/placeboon stroke volume on MRI
b) A prospective cohort assessingthe usefulness and weightof different predictive variablesin predicting the developmentof Alzheimer disease
c) A case-controlstudy testing an association of AED exposurewith birth controldefects
d) A study establishing relationship between bloodpressureon admission and volume of the intracranial hemorrhage
e) A comparison study of identification of signs of earlyischemiaon CT between radiology residents and neurology residents
15) Which of these is an example of a cross-sectional study?
a) Clinical trial
b) Case-control
c) Survey
d) Prospectivecohort
e) Retrospective cohort
16) What do all observationalstudies have in common?
a) Prospective in nature
b) Retrospectivereviews
c) Absenceof intervention
d) Time of observation
e) Require informedconsent
17) All of these are functionsof an Institutional Review Boardexcept
a) Protect patientsafety
b) Assure that the research methodologyis adequate
c) Assure that informed consent is obtained
d)Assure thatsubjects are compensated for research
e) Protectvulnerablepopulations
18) A study wants to determine whether commercialads on TVabout stroke increase the number of patients presentingto hospitalswith stroke symptoms. The best method to test thiswould be:
a) Community survey
b) Community interventiontrial
c) Retrospectivecohort
d) Quasi-randomized trial
e) Randomized clinicaltrial
19) Patients admitted toa VA hospital with an even social securitynumber are admittedto Medicine and those with an odd socialsecurity number areadmitted to Neurology. One investigatoris interested to knowif those admitted to a neurology serviceis associated withbetter outcome than those in medicine. Thiswould be an exampleof a:
a) Survey
b) Community intervention trial
c) Quasi-randomizedtrial
d) Randomized clinical trial
e) Prospective cohort
20) Investigators want to find if there is a relationshipbetweensmoking and developing pseudoseizures. They review allthe recordsto the video-EEG unit from 2006–1996 and reviewfor eachchart the diagnosis of the spells (epileptic vs pseudo)andwhether there is a history of smoking. This study wouldbe bestcharacterized as a:
a) Survey
b) Case-control
c)Retrospective review
d) Prospectivereview
e) Ecological
(NOTE: this question is linked to previous)
21) All ofthese would be potential limitations to the studyexcept:
a)Uncertain temporal sequence events
b) Misrandomizations
c)Misclassification bias
d) Selection bias
e) Incompleteinformation
Match the following concepts and definitions
22) Numberof existing cases in a population (a)
23) Proportion of unaffectedindividuals who on average willcontract the disease of interestover a specified period oftime (c)
24) Rapidity with whichnewly diagnosed disease develops (b)
a) Prevalence
b) Incidencerate
c) Risk
In a clinical trial, match the followingconcepts:
25) The two interventions are truly different, butthe trialgives negative results (b)
26) The two interventionsare truly not different but the trialshows positive results(a)
27) The power of the trial (e)
28) The likelihood ofType I error (c)
29) The likelihood of Type II error (d)
a)Type I error
b) Type II error
c) Alpha level
d) Betalevel
e) 1-Beta level
30) All these are true regarding randomizationprocedures ina clinical trial except:
a) Achieves equalityin the baseline characteristics of treatmentgroups
b) Allowsfor a fair comparison of a treatment effect
c) Avoids selectionbias
d) Prevents imbalanced treatmentassignment
e) Increasespower study
31) All of these are advantages of prospectivecohorts overretrospective cohorts, except:
a) Shorter completiontime
b) More complete and accurate
c)Clear temporal relationshipexposure-disease
d) Better forrare exposures
e) Minimizesbiases
32) All of these are advantages of case-controlstudies, except
a) Good for rare diseases
b) Less expensive
c) Rapid completion
d) Good for chronic diseases
e) Minimizesselection bias
33) A review of the red wine drinking habitsand rate strokein French and US men reveals that French mendrink more redwine and have lower rate of stroke than US men.If based onthat study we conclude that red wine drinking reducesthe riskof stroke, this could be an example of:
a) Selectionbias
b) Ecological fallacy
c) Generalization
d) Recallbias
e) Chauvinism
Match these concepts
34) Selectionbias (b)
35) Information bias (c)
36) Confounding (a)
a)An extraneous variable that accounts for the observed resultrather than the risk factor of interest
b) Sample distortedby the selection process
c) Misclassification of the variables
37) All of these are advantages of randomized clinicaltrialsexcept
a) Balanced groups through randomization
b)Detailed baselinedata
c) Blinding permits objective outcomes
d) A small numberof participants is usually required
e)Treatment doses arepre-determined by investigator
38)All of these are important ethical issues that clinicaltrialsshould meet except:
a) There should be equipoity between treatmentstested
b)The research question should be meaningful
c)The study shouldinclude underrepresented groups
d) Interimanalysis of efficacyand safety should be conducted
e) Thestudy should assurea benefit to the participant
Matchthe following:
39) Clinical trials of clinical efficacy (c)
40) Dose-drug level trials in healthy individuals (a)
Hauser SL, McArthur JC. Saving the clinician-scientist: report of the ANA long range planning committee. Ann Neurol 2006;60:278–285.[Medline]
Windish DM, Huot SJ, Green ML. Medicine residents’ understanding of the biostatistics and results in the medical literature. JAMA 2007;298:1010–1022.[Abstract/Free Full Text]
Accredited Council Graduate Medical Education. Outcome Project, General competencies. Available at: www.acgme.org. Accessed September 20, 2007.
Ringel SP, Steiner JF, Vickrey BG, Spencer SS. Training clinical researchers in neurology: we must do better. Neurology 2001;57:388–392.[Abstract/Free Full Text]
Burneo JG, Jenkins ME. Teaching evidence-based clinical practice to neurology and neurosurgery residents. Clin Neurol Neurosurg 2007;109:418–421.[Medline]
Hatala R, Keitz SA, Wilson MC, Guyatt G. Beyond journal clubs: moving toward an integrated evidence-based medicine curriculum. J Gen Intern Med 2006;21:538–541.[Medline]
Akl EA, Izuchukwu IS, El-Dika S, Fritsche L, Kunz R, Schunemann HJ. Integrating an evidence-based medicine rotation into an internal medicine residency program. Acad Med 2004;79:897–904.[Medline]
Top.
Abstract.
METHODS
