Script concordance test in medical schools in Brazil
possibilities and limitations
Keywords:
Educational measurement, Students, medical, Education, medical, Curriculum, CognitionAbstract
CONTEXT AND OBJECTIVE: Routine use of the script concordance test (SCT) is not common in Brazilian universities. This study aimed to analyze application of the SCT in the medical school of a Brazilian university. DESIGN AND SETTING: Quantitative, analytical and descriptive study in the medical school of a Brazilian university. METHODS: A total of 159/550 students participated. The test comprised ten clinical cases within internal medicine, with five items per case, rated on a five-point Likert scale. The test was scored in accordance with a marking key that had been validated by a reference panel. RESULTS: In the pre-clinical and clinical phases, the mean scores were 51.6% and 63.4% of the maximum possible scores, respectively. Comparison of the means of the responses among all the years showed that there were significant differences in 40% of the items. The panel marked all the possible answers in five items, while in one item, all the panelists marked a single answer. Cronbach’s alpha was 0.64. The results indicated that the more senior students performed better. Construction of an SCT with discriminative questions was not easy. The low reliability index may have occurred due to: a) problems with the construction of the questions; b) limitations of the reference panel; and/or c) the scoring key. CONCLUSION: This instrument is very difficult to construct, apply and correct. These difficulties may make application of an SCT as an assessment method unfeasible in units with limited resources.
Downloads
References
Piovezan RD, Custódio O, Cendoroglo MS, Batista NA. Teste de concordância de scripts: uma proposta para a avaliação do raciocínio clínico em contextos de incerteza [Script concordance test: an approach to the evaluation of clinical reasoning in uncertain contexts]. Rev Bras Educ Med. 2010;34(1):5-12.
Fournier JP, Demeester A, Charlin B. Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak. 2008;8:18.
Kreiter CD, Bergus G. The validity of performance-based measures of clinical reasoning and alternative approaches. Med Educ. 2009;43(4):320-5.
Lineberry M, Kreiter CD, Bordage G. Threats to validity in the use and interpretation of script concordance test scores. Med Educ. 2013;47(12):1175-83.
Aldekhayel SA, Alselaim NA, Magzoub ME, et al. Constructing a question bank based on script concordance approach as a novel assessment methodology in surgical education. BMC Med Educ. 2012;12:100.
Bland AC, Kreiter CD, Gordon JA. The psychometric properties of five scoring methods applied to the script concordance test. Acad Med. 2005;80(4):395-9.
Charlin B, Gagnon R, Pelletier J, et al. Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel. Med Educ. 2006;40(9):848-54.
Duggan P, Charlin B. Summative assessment of 5th year medical students’ clinical reasoning by Script Concordance Test: requirements and challenges. BMC Med Educ. 2012;12:29.
Groves M, Dick ML, McColl G, Bilszta J. Analysing clinical reasoning characteristics using a combined methods approach. BMC Med Educ. 2013;13:144.
Humbert AJ, Johnson MT, Miech E, et al. Assessment of clinical reasoning: A Script Concordance test designed for pre-clinical medical students. Med Teach. 2011;33(6):472-7.
Nouh T, Boutros M, Gagnon R, et al. The script concordance test as a measure of clinical reasoning: a national validation study. Am J Surg. 2012;203(4):530-4.
Sibert L, Charlin B, Corcos J, et al. Stability of clinical reasoning assessment results with the Script Concordance test across two different linguistic, cultural and learning environments. Med Teach. 2002;24(5):522-7.
Sibert L, Darmoni SJ, Dahamna B, et al. On line clinical reasoning assessment with Script Concordance test in urology: results of a French pilot study. BMC Med Educ. 2006;6:45.
Brailovsky C, Charlin B, Beausoleil S, Coté S, Van der Vleuten C. Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test. Med Educ. 2001;35(5):430-6.
Collard A, Gelaes S, Vanbelle S, et al. Reasoning versus knowledge retention and ascertainment throughout a problem-based learning curriculum. Med Educ. 2009;43(9):854-65.
Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten CPM. Script concordance testing: a review of published validity evidence. Med Educ. 2011;45(4):329-38.
Cronbach L. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. Available from: http://kttm. hoasen.edu.vn/sites/default/files/2011/12/22/cronbach_1951_ coefficient_alpha.pdf. Accessed in 2015 (Sep 8).
Gagnon R, Charlin B, Coletti M, Sauvé E, van der Vleuten C. Assessment in the context of uncertainty: how many members are needed on the panel of reference of a script concordance test? Med Educ. 2005;39(3):284-91.
