Reliability across content areas in progress tests assessing medical knowledge: a Brazilian cross-sectional study with implications for medical education assessments

Autores

Palavras-chave:

Internal medicine, Education, medical, Educational measurement, Reliability, Progress test, Medical education assessments

Resumo

BACKGROUND: Brazilian medical schools equitably divide their medical education assessments into five content areas: internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. However, this division does not follow international patterns and may threaten the examinations’ reliability and validity.

OBJECTIVE: To assess the reliability indices of the content areas of serial, cross-institutional progress test examinations.

DESIGN AND SETTINGS: This was an analytical, observational, and cross-sectional study conducted at nine public medical schools (mainly from the state of São Paulo) with progress test examinations conducted between 2017 and 2023.

METHODS: The examinations covered the areas of basic sciences, internal medicine, surgery, pediatrics, obstetrics and gynecology, and public health. We calculated reliability indices using Cronbach’s α, which indicates the internal consistency of a test. We used simple linear regressions to analyze temporal trends.

RESULTS: The results showed that the Cronbach’s α for basic sciences and internal medicine presented lower values, whereas gynecology, obstetrics, and public health presented higher values. After changes in the number of items and the exclusion of basic sciences as a separate content area, internal medicine ranked highest in 2023. Individually, all content areas except pediatrics remained stable over time.

CONCLUSIONS: Maintaining an equitable division in assessment content may lead to suboptimal results in terms of assessment reliability, especially for internal medicine. Therefore, content sampling of medical knowledge for general assessments should be reappraised.

Downloads

Não há dados estatísticos.

Biografia do Autor

Pedro Tadao, Universidade Estadual Paulista “Júlio de Mesquita Filho”, Botucatu, São Paulo, Brazil

PhD. Physician, Assistant Professor, Departament of Neurology, Psychology and Psychiatry, Faculdade de
Medicina de Botucatu, Universidade Estadual Paulista (UNESP), Botucatu (SP), Brazil.

Miriam, Universidade Estadual Paulista “Júlio de Mesquita Filho”, Botucatu, São Paulo, Brazil

PhD. Physician, Assistant Professor, Departament of Pediatrics, Faculdade de Medicina de Botucatu, Universidade Estadual Paulista (UNESP), Botucatu (SP), Brazil.

Alba Regina de Abreu, Faculdade de Medicina de São José do Rio Preto, São José do Rio Preto, São Paulo, Brazil

PhD. Biologist, Assistant Professor, Departament of Molecular Biology, Faculdade de Medicina de São José do
Rio Preto (FAMERP), São José do Rio Preto (SP), Brazil.

Leandro Arthur, Universidade Estadual de Londrina, Londrina, Paraná, Brazil

PhD. Physician, Assistant Professor, Departament of Internal Medicine, Universidade Estadual de Londrina (UEL), Londrina (PR), Brazil.

Neide Tomimura, Universidade Estadual de Londrina, Londrina, Paraná, Brazil

PhD. Physician, Assistant Professor, Departament of Internal Medicine, Universidade Estadual de Londrina (UEL), Londrina (PR), Brazil.

Patrícia Moretti, Universidade Estadual de Campinas, Campinas, São Paulo, Brazil

PhD. Physician, Assistant Professor, Departament of Tocogynecology, Universidade Estadual de Campinas
(UNICAMP), Campinas (SP), Brazil.

Samira, Universidade Federal de São Paulo, São Paulo, São Paulo, Brazil

PhD. Physician, Assistant Professor, Departament of Dermatology, Universidade Federal de São Paulo (UNIFESP), São Paulo (SP), Brazil.

Maria Cristina de, Universidade Federal de São Paulo, São Paulo, São Paulo, Brazil

PhD. Physician, Assistant Professor, Departament of Pediatrics, Universidade Federal de São Paulo (UNIFESP), São Paulo (SP), Brazil.

Maria de Lourdes Marmorato Botta, Faculdade de Medicina de Marília, Marília, São Paulo, Brazil

PhD. Physician, Assistant Professor, Academic Assessment Center, Faculdade de Medicina de Marília (FAMEMA), Marília (SP), Brazil.

Zilda Maria Tosta, Faculdade de Medicina de Marília, Marília, São Paulo, Brazil

PhD. Physician, Assistant Professor, Academic Assessment Center, Faculdade de Medicina de Marília (FAMEMA), Marília (SP), Brazil.

Júlio César, Universidade de São Paulo, Ribeirão Preto, São Paulo, Brazil

PhD. Physician, Associate Professor, Departament of Internal Medicine, Universidade de São Paulo (USP), Ribeirão Preto (SP), Brazil.

Angélica Maria, Universidade Estadual de Campinas, Campinas, São Paulo, Brazil

PhD. Physician, Full Professor. Departamentof Pediatrics, Universidade Estadual de Campinas (UNICAMP), Campinas (SP), Brazil.

Referências

Ministério da Educação. Resolução no 01, de 14 de agosto de 2000. Available from: https://www.gov.br/mec/pt-br/residencia-medica/ ementario-da-legislacao-da-residencia-medica. Accessed in 2023 (Nov. 01)

Bicudo AM, Hamamoto Filho PT, Abbade JF, Hafner MLMB, Maffei CML. Consortia of cross-institutional Progress Testing for all medical schools in Brazil. Rev Bras Educ Med. 2019;43(4):151-6. https://dx.doi. org/10.1590/1981-52712015v43n4RB20190018.

Zanolli MB, Streit DS, Maciel DT, et al. Differences in clerkship development between public and private Brazilian medical schools: an overview. BMC Med Educ. 2020 Sep 21;20(1):316. PMID: 32957972; https://doi.org/10.1186/s12909-020-02193-3.

Tio RA, Schutte B, Meiboom AA, Get al. The progress test of medicine: the Dutch experience. Perspect Med Educ. 2016;5(1):51-5. PMID: 26754310; https://doi.org/10.1007/s40037-015-0237-1.

Nouns ZM, Georg W. Progress testing in German speaking countries. Med Teach. 2010;32(6):467-70. PMID: 20515374. https://doi.org/10.31 09/0142159x.2010.485656.

United States Medical Licensing Examination. Step 2 CK Content Outline & Specifications. Available from: https://usmle.org/prepare-your-exam/ step-2-ck-materials/step-2-ck-content-outline-specifications. Accessed in 2023 (Nov. 01)

Downing SM. Reliability: on the reproducibility of assessment data. Med Educ. 2004;38(9):1006-12. PMID: 15327684. https://doi.org/10.1111/ j.1365-2929.2004.01932.x.

Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. https://dx.doi.org/10.1007/ BF02310555.

Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. 2011;2:53-55. PMID: 28029643; https://doi.org/10.5116/ijme.4dfb.8dfd.

Cecilio-Fernandes D, Bicudo AM, Hamamoto Filho PT. Progress testing as a pattern of excellence for the assessment of medical students’ knowledge - concepts, history, and perspective. Medicina (Ribeirão Preto). 2021;54(1):e-173770. http://dx.doi.org/10.11606/issn.2176-7262. rmrp.2021.173770.

Troncon LEA, Elias LLK, Osako MK, et al. Reflections on the use of the Progress Test in the programmatic student assessment. Rev Bras Educ Med. 2023;47(2):e076. https://dx.doi.org/10.1590/19815271v47.2-2022-0334.ing.

Tomic ER, Martins MA, Lotufo PA, Benseñor IM. Progress testing: evaluation of four years of application in the school of medicine, University of São Paulo. Clinics (Sao Paulo). 2005;60(5):389-96. PMID: 16254675. https://doi.org/10.1590/s1807-59322005000500007.

Sakai MH, Ferreira Filho OF, Almeida MJ, Mashima DA, Marchese MC. Progress testing and course evaluation: ten years of experience in the undergraduate medical course at the State University of Londrina. Rev Bras Educ Med. 2008;32(2):254-63. https://dx.doi.org/10.1590/S010055022008000200014.

Machado JLM. A participação da ANDES-SN na Cinaem: o processo de transformação das escolas médicas brasileiras. Interface (Botucatu). 1999;3(5):177–80. https://doi.org/10.1590/S1414-32831999000200025.

Falavigna A, Canabarro CT, Medeiros GS. Health system and medical education in Brazil: history, principles, and organization. World Neurosurg. 2013;80(6):723-7. PMID: 23333994. https://doi.org/10.1016/j. wneu.2013.01.079.

Ministério da Educação. Resolução CNE/CES no 4, de 7 de Novembro de 2001. Brasilia: Ministério da Educação; 2001. Available from: http://portal. mec.gov.br/cne/arquivos/pdf/CES04.pdf. Accessed in 2023 (Nov. 01) 17. Ministério da Educação. Resolução CNE/CES no 3, de 20 de Junho de 2014. Brasilia: Ministério da Educação; 2014. Available from: http://portal. mec.gov.br/cne/arquivos/pdf/CES04.pdf. Accessed in 2023 (Nov. 01)

Antunes Dos Santos R, Nunes MDPT. Medical education in Brazil. Med Teach. 2019;41(10):1106-1111. PMID: 31282823. https://doi.org/10.10 80/0142159x.2019.1636955.

Agbo AA. Cronbach’s alpha: review of limitations and associated recommendations. J Psychol Africa. 2010; 20(2):233-39. https://dx.doi. org/10.1080/14330237.2010.10820371.

Charter RA. Sample size requirements for precise estimates of reliability, generalizability, and validity coefficients. J Clin Exp Neuropsychol. 1999;21(4):559-66. PMID: 10550813. https://doi.org/10.1076/ jcen.21.4.559.889.

Schuwirth LW, van der Vleuten CP. The use of progress testing. Perspect Med Educ. 2012;1(1):24-30. PMID: 23316456; https://doi.org/10.1007/ s40037-012-0007-2.

Hamamoto Filho PT, Bicudo AM. Improvement of faculty’s skills on the creation of items for progress testing through feedback to item writers: a successful experience. Rev Bras Educ Med. 2020;44(1):e018.

https://dx.doi.org/10.1590/1981-5271v44.1-20190130.ING.

Feliciano CS, Elias LLK, Osako MK, Guimarães FS, Troncon LEA, Bollela VR. Workshop on item writing to assess basic sciences in health professional education: an experience report. Rev Bras Educ Med. 2023;47(2):e067. https://doi.org/10.1590/1981-5271v47.2-2022-0297.

Cecilio-Fernandes D, Kerdijk W, Bremers AJ, Aalders W, Tio RA. Comparison of level of cognitive process between case-based items and non-casebased items of the interuniversity progress test of medicine in the Netherlands. J Educ Eval Health Prof. 2018;15:28. PMID: 30541188; https://doi.org/10.3352/jeehp.2018.15.28.

Hamamoto Filho PT, Silva E, Ribeiro ZMT, Hafner MLMB, CecilioFernandes D, Bicudo AM. Relationships between Bloom’s taxonomy, judges’ estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study. Sao Paulo Med J. 2020;138(1):33-39. PMID: 32321103; https://doi. org/10.1590/1516-3180.2019.0459.r1.19112019.

Norman G, Neville A, Blake JM, Mueller B. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach. 2010;32(6):496-9. PMID: 20515380. https:// doi.org/10.3109/0142159x.2010.486063.

Kane MT. Validating the interpretations and uses of test scores. J Educ Meas. 2013;50(1):1-73. https://dx.doi.org/10.1111/jedm.12000.

Boursicot K, Kemp S, Wilkinson T, et al. Performance assessment: Consensus statement and recommendations from the 2020 Ottawa Conference. Med Teach. 2021;43(1):58-67. PMID: 33054524. https://doi. org/10.1080/0142159x.2020.1830052.

McLaughlin K, Lemaire J, Coderre S. Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach. 2005;27(6):544-7. PMID: 16199363. https://doi.org/10.1080/01421590500136113.

Downloads

Publicado

2025-06-06

Como Citar

1.
Pedro Tadao, Miriam, Alba Regina de Abreu, Leandro Arthur, Neide Tomimura, Patrícia Moretti, Samira, Maria Cristina de, Maria de Lourdes Marmorato Botta, Zilda Maria Tosta, Júlio César, Angélica Maria. Reliability across content areas in progress tests assessing medical knowledge: a Brazilian cross-sectional study with implications for medical education assessments. Sao Paulo Med J [Internet]. 6º de junho de 2025 [citado 16º de junho de 2025];142(6):1-6. Disponível em: https://periodicosapm.emnuvens.com.br/spmj/article/view/3225

Edição

v. 142 n. 6 (2024)

Seção

Artigo Original

Licença

Copyright (c) 2025 São Paulo Medical Journal

Creative Commons License
Este trabalho está licenciado sob uma licença Creative Commons Attribution 4.0 International License.