Validity and Reliability of Elasticity Multiple-Choice Items (EMCI) Using Rasch Model

Vivi Mardian, Achmad Samsudin, Judhistira Aria Utama, Irma Rahma Suwarma, Bayram Coştu

Abstract


Assessment in the form of an instrument is very important to test for validity and reliability so that it can measure student learning outcomes. We implemented a measurement instrument designed to assess students' understanding of elasticity. The measurement instrument consisted of 15 items. The instrument was administered to students who were taking physics subjects in senior high school in their second year. In total, 74 students were taken from two classes in Padang, Indonesia. In this study, the students' performance was collected as quantitative data and evaluated using the Rasch model. The results showed that the data matched the Rasch model measurements. Moreover, female students answered 53% of the questions better than male students. Furthermore, the DIF plot shows that S8 is the most difficult problem, while S14 is the easiest. There are two gender bias questions, namely S12 and S14. Both questions were easily solved by female students. However, all questions can be used to measure students' abilities in elasticity and Hooke's law. This study seeks to make a contribution to the literature on EMCI evaluations by offering a case study for academics and researchers to use in assessing students' elasticity skills

Keywords


Elasticity Concept; Multiple Choice Items; Rasch Model

Full Text:

PDF

References


Akhtar, H., & Sumintono, B. (2023). Rasch analysis of the International Personality Item Pool Big Five Markers Questionnaire: Is longer better? Primenjena Psihologija, 16(1), 3–28. https://doi.org/10.19090/pp.v16i1.2401

Al-Owidha, A. A. (2018). Investigating the psychometric properties of the Qiyas for L1 Arabic language test using a Rasch measurement framework. Language Testing in Asia, 8(1), 12. https://doi.org/10.1186/s40468-018-0064-5

Borda, E., Schumacher, E., Hanley, D., Geary, E., Warren, S., Ipsen, C., & Stredicke, L. (2020). Initial implementation of active learning strategies in large, lecture STEM courses: Lessons learned from a multi-institutional, interdisciplinary STEM faculty development program. International Journal of STEM Education, 7(1), 4. https://doi.org/10.1186/s40594-020-0203-2

Brassil, C. E., & Couch, B. A. (2019). Multiple-true-false questions reveal more thoroughly the complexity of student thinking than multiple-choice questions: A Bayesian item response model comparison. International Journal of STEM Education, 6(1), 16. https://doi.org/10.1186/s40594-019-0169-0

Carotenuto, G., Di Martino, P., & Lemmi, M. (2021). Students’ suspension of sense making in problem solving. ZDM – Mathematics Education, 53(4), 817–830. https://doi.org/10.1007/s11858-020-01215-0

Chan, S.-W., Looi, C.-K., & Sumintono, B. (2021). Assessing computational thinking abilities among Singapore secondary students: A Rasch model measurement analysis. Journal of Computers in Education, 8(2), 213–236. https://doi.org/10.1007/s40692-020-00177-2

Chen, Q., Zhu, G., Liu, Q., Han, J., Fu, Z., & Bao, L. (2020). Development of a multiple-choice problem-solving categorization test for assessment of student knowledge structure. Physical Review Physics Education Research, 16(2), 020120. https://doi.org/10.1103/PhysRevPhysEducRes.16.020120

Dahl, L. S., Staples, B. A., Mayhew, M. J., & Rockenbach, A. N. (2023). Meeting Students Where They Are: Using Rasch Modeling for Improving the Measurement of Active Research in Higher Education. Innovative Higher Education, 48(3), 557–577. https://doi.org/10.1007/s10755-022-09643-4

Diantoro, M., Wartono, W., Leasa, M., & Batlolona, J. R. (2020). Students Mental Models of Solid Elasticity: A Mixed Method Study. Turkish Journal of Science Education, 17(2), 199–209. https://doi.org/10.36681/tused.2020.21

Dziob, D. (2020). Board Game in Physics Classes—A Proposal for a New Method of Student Assessment. Research in Science Education, 50(3), 845–862. https://doi.org/10.1007/s11165-018-9714-y

Gamage, S. H. P. W., Ayres, J. R., & Behrend, M. B. (2022). A systematic review on trends in using Moodle for teaching and learning. International Journal of STEM Education, 9(1), 9. https://doi.org/10.1186/s40594-021-00323-x

Gladys, M. J., Furst, J. E., Holdsworth, J. L., & Dastoor, P. C. (2023). Gender bias in first-year multiple-choice physics examinations. Physical Review Physics Education Research, 19(2), 020109. https://doi.org/10.1103/PhysRevPhysEducRes.19.020109

Grover, R., & Wright, A. (2023). Shutting the studio: The impact of the Covid-19 pandemic on architectural education in the United Kingdom. International Journal of Technology and Design Education, 33(3), 1173–1197. https://doi.org/10.1007/s10798-022-09765-y

Ha, H. T. (2021). A Rasch-based validation of the Vietnamese version of the Listening Vocabulary Levels Test. Language Testing in Asia, 11(1), 16. https://doi.org/10.1186/s40468-021-00132-7

Hasanah, D., & Purwanto, J. (2023). Rasch model analysis of physics test of HOTS on the topic of elasticity and hooke’s law. THABIEA : JOURNAL OF NATURAL SCIENCE TEACHING, 6(1), 25. https://doi.org/10.21043/thabiea.v6i1.20422

Hedgeland, H., Dawkins, H., & Jordan, S. (2018). Investigating male bias in multiple choice questions: Contrasting formative and summative settings. European Journal of Physics, 39(5), 055704. https://doi.org/10.1088/1361-6404/aad169

Henry, M. A., Shorter, S., Charkoudian, L. K., Heemstra, J. M., Le, B., & Corwin, L. A. (2021). Quantifying fear of failure in STEM: Modifying and evaluating the Performance Failure Appraisal Inventory (PFAI) for use with STEM undergraduates. International Journal of STEM Education, 8(1), 43. https://doi.org/10.1186/s40594-021-00300-4

Heritage, B., Ladeira, C., & Steele, A. R. (2023). The development and pilot of the university student embeddedness (USE) scale for student retention within universities: Validation with an Australian student sample. Higher Education, 85(1), 27–54. https://doi.org/10.1007/s10734-022-00813-z

Kabic, M., & Alexandrowicz, R. W. (2023). RMX/PIccc: An Extended Person–Item Map and a Unified IRT Output for eRm, psychotools, ltm, mirt, and TAM. Psych, 5(3), 948–965. https://doi.org/10.3390/psych5030062

Kashihara, S., & Fukaya, T. (2022). Does a self-report questionnaire predict strategy use in mathematical problem solving among elementary school children? Importance of question format depending on the grade. European Journal of Psychology of Education. https://doi.org/10.1007/s10212-022-00668-z

Khan, S., & Krell, M. (2019). Scientific Reasoning Competencies: A Case of Preservice Teacher Education. Canadian Journal of Science, Mathematics and Technology Education, 19(4), 446–464. https://doi.org/10.1007/s42330-019-00063-9

Khine, M. S. (2020). Objective Measurement in Psychometric Analysis. In M. S. Khine (Ed.), Rasch Measurement (pp. 3–7). Springer Singapore. https://doi.org/10.1007/978-981-15-1800-3_1

Larrain, M., & Kaiser, G. (2022). Interpretation of Students’ Errors as Part of the Diagnostic Competence of Pre-Service Primary School Teachers. Journal Für Mathematik-Didaktik, 43(1), 39–66. https://doi.org/10.1007/s13138-022-00198-7

McKenna, P. (2019). Multiple choice questions: Answering correctly and knowing the answer. Interactive Technology and Smart Education, 16(1), 59–73. https://doi.org/10.1108/ITSE-09-2018-0071

Mierke, C. T. (2020). Cellular Mechanics and Biophysics: Structure and Function of Basic Cellular Components Regulating Cell Mechanics. Springer International Publishing. https://doi.org/10.1007/978-3-030-58532-7

Park, M., & Liu, X. (2021). An Investigation of Item Difficulties in Energy Aspects Across Biology, Chemistry, Environmental Science, and Physics. Research in Science Education, 51(S1), 43–60. https://doi.org/10.1007/s11165-019-9819-y

Pedaste, M., Baucal, A., & Reisenbuk, E. (2021). Towards a science inquiry test in primary education: Development of items and scales. International Journal of STEM Education, 8(1), 19. https://doi.org/10.1186/s40594-021-00278-z

Podschuweit, S., & Bernholt, S. (2018). Composition-Effects of Context-based Learning Opportunities on Students’ Understanding of Energy. Research in Science Education, 48(4), 717–752. https://doi.org/10.1007/s11165-016-9585-z

Rodríguez-Mora, F., Cebrián-Robles, D., & Blanco-López, Á. (2022). An Assessment Using Rubrics and the Rasch Model of 14/15-Year-Old Students’ Difficulties in Arguing About Bottled Water Consumption. Research in Science Education, 52(4), 1075–1091. https://doi.org/10.1007/s11165-020-09985-z

Schwarz, G. (2023). Multiple-Choice Questions for Teaching Quantitative Instrumental Element Analysis: A Follow-Up. Journal of Chemical Education, acs.jchemed.3c00061. https://doi.org/10.1021/acs.jchemed.3c00061

Silva, O. H. M., Laburú, C. E., Camargo, S., & Christófalo, A. A. C. (2019). Epistemological Contributions Derived from an Investigative Method in an Experimental Class in the Study of Hooke’s Law. Acta Scientiae, 21(2). https://doi.org/10.17648/acta.scientiae.v21iss2id4695

Susanti, Y., Tokunaga, T., Nishikawa, H., & Obari, H. (2018). Automatic distractor generation for multiple-choice English vocabulary questions. Research and Practice in Technology Enhanced Learning, 13(1), 15. https://doi.org/10.1186/s41039-018-0082-z

Suwono, H., Prasetyo, T. I., Lestari, U., Lukiati, B., Fachrunnisa, R., Kusairi, S., Saefi, M., Fauzi, A., & Atho’Illah, M. F. (2021). Cell Biology Diagnostic Test (CBD-Test) portrays pre-service teacher misconceptions about biology cell. Journal of Biological Education, 55(1), 82–105. https://doi.org/10.1080/00219266.2019.1643765

Taber, K. S. (2018). The Use of Cronbach’s Alpha When Developing and Reporting Research Instruments in Science Education. Research in Science Education, 48(6), 1273–1296. https://doi.org/10.1007/s11165-016-9602-2

Treagust, D. F. (1988). Development and use of diagnostic tests to evaluate students’ misconceptions in science. International Journal of Science Education, 10(2), 159–169. https://doi.org/10.1080/0950069880100204

Tumanggor, A. M. R., Supahar, S., Ringo, E. S., & Harliadi, M. D. (2020). Detecting Students’ Misconception in Simple Harmonic Motion Concepts Using Four-Tier Diagnostic Test Instruments. Jurnal Ilmiah Pendidikan Fisika Al-Biruni, 9(1), 21–31. https://doi.org/10.24042/jipfalbiruni.v9i1.4571

Van Wijk, E. V., Janse, R. J., Ruijter, B. N., Rohling, J. H. T., Van Der Kraan, J., Crobach, S., Jonge, M. D., Beaufort, A. J. D., Dekker, F. W., & Langers, A. M. J. (2023). Use of very short answer questions compared to multiple choice questions in undergraduate medical students: An external validation study. PLOS ONE, 18(7), e0288558. https://doi.org/10.1371/journal.pone.0288558

Vázquez-Bernal, B., & Jiménez-Pérez, R. (2023). Modeling a Theoretical Construct on Pupils’ Difficulties in Problem Solving. Science & Education, 32(1), 199–229. https://doi.org/10.1007/s11191-021-00289-w

Wammes, D., Slof, B., Schot, W., & Kester, L. (2022). Pupils’ prior knowledge about technological systems: Design and validation of a diagnostic tool for primary school teachers. International Journal of Technology and Design Education, 32(5), 2577–2609. https://doi.org/10.1007/s10798-021-09697-z

Wind, S. A., Alemdar, M., Lingle, J. A., Moore, R., & Asilkalkan, A. (2019). Exploring student understanding of the engineering design process using distractor analysis. International Journal of STEM Education, 6(1), 4. https://doi.org/10.1186/s40594-018-0156-x




DOI: http://dx.doi.org/10.24042/jipfalbiruni.v12i2.17037

Refbacks

  • There are currently no refbacks.


Creative Commons License

Jurnal ilmiah pendidikan fisika Al-Biruni is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.