Assessing Item Reliability, Differential Item Functioning, and Wright Map Analysis of the GSP122 Test at a Public University in Nigeria
DOI:
https://doi.org/10.56916/jesi.v2i2.934Keywords:
Item reliability, Differential item functioning, Wright map, GSP122 test, ICT and Rasch modelAbstract
This study examines the psychometric properties of the GSP122 test, an Information and Communication Technology (ICT) knowledge assessment administered at a public university in Nigeria. Despite its importance in evaluating students' ICT competencies, no prior attempt has been made to investigate the test's psychometric qualities. The research focuses on three key aspects: item reliability, Differential Item Functioning (DIF), and Wright Map analysis. The study employs Rasch analysis to evaluate these properties. A sample of 600 GSP122 test scripts was randomly selected from undergraduate students across various departments to ensure a representative assessment. Findings reveal that the test possesses strong item reliability, indicating consistency in measuring the intended construct. Furthermore, all items are found to be DIF-free, suggesting fairness across different subgroups of test-takers. The Wright Map analysis, however, indicates that the test doesn't accurately target the abilities of students at the extreme ends and bottom of the proficiency spectrum. Specifically, some items are identified as too difficult and too easy relative to the students' ability levels. These results provide valuable insights into the GSP122 test's strengths and areas for improvement. While the test demonstrates robustness in reliability and fairness, adjustments in item difficulty could enhance its effectiveness in assessing students across all proficiency levels. This comprehensive analysis contributes to the validation of the GSP122 test and offers a foundation for evidence-based refinements in ICT assessment practices within the Nigerian higher education context.
References
Adebayo, F., & Eze, U. (2024). ICT proficiency and national development: A study of Nigerian university graduates. African Journal of Education and Technology, 15(2), 123-140.
Andrich, D., & Styles, I. (2004). Final report on the psychometric analysis of the Early Development Instrument (EDI) using the Rasch model: A technical paper commissioned for the development of the Australian Early Development Instrument (AEDI). Perth, Australia: Murdoch University
Adebayo, F. (2021). ICT skills and national development in Nigeria: A critical analysis. Journal of African Studies, 45(3), 278-292.
Baghaei, P., & Amrahi, N. (2011). Validation of a multiple-choice English vocabulary test with the Rasch Model. Journal of Language Teaching and Research 2(5):1052-1060. DOI: 10.4304/jltr.2.5.1052-1060
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.
Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Lawrence Erlbaum Associates. https://psycnet.apa.org/record/2001-06187-000
Baghaei, P. (2008). The Rasch Model as a construct validation tool. Rasch Measurement Transactions, 22(1), 1145–1146.
Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch analysis in the human sciences. Springer.
Cooper, M., & Weaver, K. (2022). Gender Bias in STEM Education: Implications for Educational Practices. Journal of Educational Research, 115(3), 295-310.
DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Sage Publications.
Fisher, W.P. (2007). Rating scale instrument quality criteria. Rasch measurement transactions, 21(1), 1095.
Holland, P. W., & Wainer, H. (Eds.). (2012). Differential item functioning. Routledge.
Hill, C., Koekemeor, E. (2013). The development of the MACE work-family enrichment instrument. SA Journal of Industrial Psychology 39(2):1147–1162. 10.4102/sajip.v39i2.1147.
Hambleton, R. K., & Jones, R. W. (2021). Principles and Practices of Test Calibration and Linking. Springer.
Hamad, A. A. (2021). The psychometric properties of measurement of the mathematics teachers’ professional identity in Saudi Arabia (Unpublished doctoral dissertation). USM.
Hur, Y., Te Nijenhuis, J., & Jeong, H. (2017). Testing Lynn’s theory of sex differences in intelligence in a large sample of Nigerian school-aged children and adolescents (N > 11 000) using Raven’s standard progressive matrices plus. Mankind Quarterly 573(3):428-437. Doi: 10.46469/mq.2017.57.3.11
Johnson, R., Smith, A., & Garcia, M. (2023). A review of ICT assessment methods in global higher education: Trends and challenges. Journal of Educational Technology & Society, 26(1), 45-60.
Johnson, M., & Lee, S. (2019). The role of ICT literacy in higher education: A global perspective. International Journal of Educational Technology, 12(2), 145-163.
Kumar, V., & Singh, R. (2022). Employer expectations of ICT skills in recent graduates: A cross-sectional study. Journal of Vocational Education & Training, 74(4), 512-528.
Lee, J., & Park, S. (2024). Applying the Rasch model to evaluate ICT literacy: A case study from South Korea. Educational and Psychological Measurement, 84(2), 301-318.
Liu, X., & Wilson, M. (2019). Ensuring Fairness in Educational Assessment: The Role of Test Targeting. Educational Measurement: Issues and Practice, 38(2), 25-33.
Linacre, J. M. (2012). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.
Linacre, J. M. (2006). A user’s guide to WINSTEPS/ MINISTEPS: A Rasch model computer programs. Chicago, USA: Winsteps.com.
Linacre, J. M. (1994). Sample size and item calibration stability. Rasch Measurement Transactions, 7(4), 328.
Mustafa, A. K., & Ehab, M. N. (2022). Rasch analysis and differential item functioning of English language anxiety scale (ELAS) across sex in Egyptian context. BMC Psychology, 10(242), 4. https://doi.org/10.1186/s40359-022-00955-w
Oladipo, A., Nwosu, L., & Eze, U. (2023). ICT proficiency and academic performance: A longitudinal study of Nigerian university students. International Journal of Educational Research, 112, 101742.
Oladipo, A., & Eze, U. (2022). Evaluating ICT assessment tools in Nigerian universities: Challenges and opportunities. African Journal of Educational Assessment, 18(4), 412-428.
Oluwatobi, S., Efobi, U., Olurinola, I., & Alege, P. (2019). ICT and higher education in Nigeria: The way forward. Journal of Educational Innovation, 32(1), 87-103.
Pellegrino, J. W., DiBello, L. V., & Goldman, S. R. (2016). A framework for conceptualizing and evaluating the validity of instructionally relevant assessments. Educational Psychologist, 51(1), 59-81.
Paek, I., & Wilson, M. (2018). A Rasch analysis of the Test of English as a Foreign Language (TOEFL). Language Testing, 35(2), 147-164.
Richard, M. W., Peter, M. A., & Jotham, N. D. (2023). Psychometric properties of a test anxiety scale for use in computer-based testing in Kenya. The International Journal of Assessment and Evaluation, 30(1), 2327-8692. http://dx.doi.org/10.18848/2327-7920/CGP/v31i01/1-18.
Redecker, C., & Johannessen, Ø. (2013). Changing assessment — Towards a new assessment paradigm using ICT. European Journal of Education, 48(1), 79-96.
Smith, J., Brown, A., & Garcia, C. (2020). The importance of ICT skills in the 21st-century workforce. Journal of Vocational Education & Training, 72(3), 312-328.
Wang, W. (2008). Assessment of differential item functioning. Journal of Applied Measurement, 9(4), 387–408.
Wright, B. D., & Masters, G. N. (2018). Rating scale analysis: Rasch measurement. SAGE Publications.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Lawrence Erlbaum Associates.
Wilson, M. (2018). Making measurements important for education: The crucial role of classroom assessment. Educational Measurement: Issues and Practice, 37(1), 5-20.
Wu, M., & Adams, R. (2020). Applying the Rasch Model to Evaluate Fairness in Testing. Educational Assessment, 25(4), 287-305.
Wilson, M. (2022). Constructing Measures: An Item Response Modeling Approach. Routledge.
Yang, P., et al. (2020). A Rasch analysis of the Chinese version of the Beck Depression Inventory-II. Journal of Clinical Psychology, 76(1), 35-47.
Zumbo, B. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessment Quarterly, 4(2), 223-233.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Abubakar Rabiu Uba, Ahmad Zamri Khairani
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
