Weighted‐scoring scientific literacy instrument: development and validation of a contextualized assessment for junior high school students

Nurgan Tadeko; Dadan Rosana; Lusila Andriani Purwastuti

doi:10.31637/epsir-2026-2149

Autores/as

Nurgan Tadeko Yogyakarta State University https://orcid.org/0000-0003-3805-9078
Dadan Rosana Yogyakarta State University https://orcid.org/0000-0003-4987-7420
Lusila Andriani Purwastuti Yogyakarta State University https://orcid.org/0000-0003-3385-8700

DOI:

https://doi.org/10.31637/epsir-2026-2149

Palabras clave:

Science literacy, development of educational instruments, Instrument Validation, confirmatory factor analysis, science education, multiple-choice test

Resumen

Introduction: This study presents the development and validation of a contextualized scientific literacy assessment instrument for secondary school students, designed to overcome the limitations of traditional multiple-choice tests with dichotomous scoring. Weighted scoring allows for partial credit to be assigned to responses that demonstrate partial understanding, thus enabling a fairer evaluation. Methodology: A design-and-validation framework was employed to construct a multidimensional instrument based on PISA competencies and the Test of Scientific Literacy Skills, adapted to the national curriculum. Content validity was assessed by expert judgment, and the instrument was administered to 408 secondary students from 11 schools. Results: The Aiken’s V coefficients for content validity ranged from 0.73 to 0.87. Confirmatory factor analysis revealed strong loadings on a single factor (≥ 0.70; AVE = 0.665). After minor model adjustments, the fit indices were excellent (CFI = 0.999; RMSEA = 0.025), and internal consistency was high (composite reliability = 0.947). Discussion: These results indicate that the instrument meets statistical validity standards and supports the identification of key areas for improvement in scientific literacy. Conclusions: The instrument is valid and reliable for diagnosing levels of scientific literacy and for evaluating the effectiveness of educational interventions.

Descargas

Los datos de descargas todavía no están disponibles.

Biografía del autor/a

Nurgan Tadeko, Yogyakarta State University

is a student at Yogyakarta State University, as well as a researcher and lecturer in the field of science education. He specializes in the development and validation of instruments for measuring scientific literacy in multicultural contexts, employing confirmatory factor analysis and advanced statistical modeling techniques for the empirical validation of integrated science learning media. At Tadulako University in Indonesia, he teaches quantitative methodology courses, coordinates multidisciplinary teams with students and cultural tutors, and conducts workshops on SEM/AMOS analysis and the development of learning media.

Dadan Rosana, Yogyakarta State University

Dadan Rosana is a professor in the Department of Physical Education at the Faculty of Mathematics and Natural Sciences, Yogyakarta State University, Indonesia. Since 2023, she has served as Dean of the Faculty of Mathematics and Natural Sciences at Yogyakarta State University. She holds a Bachelor of Science from Bandung Teachers’ Training Institute (IKIP Bandung), a Master of Science from Bandung Institute of Technology, and a Ph.D. in Educational Measurement and Evaluation from Yogyakarta State University. His research interests include creativity and higher-order thinking skills in science education.

Lusila Andriani Purwastuti, Yogyakarta State University

Lusila Andriani Purwastuti has been an Associate Professor in the Faculty of Education, Yogyakarta State University, Indonesia, since 2016. She earned her Ph.D. in Social Pedagogy from Gadjah Mada University in 2015. Her research interests include inclusive curriculum design, formative assessment of transversal competencies, and critical pedagogy in multicultural contexts. Among her recent publications is a study on the reliability of environmental attitude scales and the application of critical pedagogy in Indonesian schools. She also conducts workshops on competency-based assessment grounded in sociocultural theory and coordinates international collaborations focused on social justice and educational equity.

Citas

AAAS. (2011). Vision & change in undergraduate biology education. American Association for the Advancement of Science. http://visionandchange.org/

Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012 DOI: https://doi.org/10.1177/0013164485451012

Alsubait, T., Parsia, B., & Sattler, U. (2013). A similarity-based theory of controlling MCQ difficulty. 2013 Second International Conference on E-Learning and E-Technologies in Education (ICEEE), 283-288. https://doi.org/10.1109/ICeLeTE.2013.6644389 DOI: https://doi.org/10.1109/ICeLeTE.2013.6644389

Anderson, D. L., Fisher, K. M., & Norman, G. J. (2002). Development and evaluation of the conceptual inventory of natural selection. Journal of Research in Science Teaching, 39(10), 952-978. https://doi.org/10.1002/tea.10053 DOI: https://doi.org/10.1002/tea.10053

Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.

Angoff, W. H. (1984). Scales, norms, and equivalent scores. Educational Testing Service.

Atta, H. B., Vlorensius, A., Aras, I., & Ikhsanudin. (2020). Developing an instrument for students’ scientific literacy. Journal of Physics: Conference Series, 1422(1), 012-019. https://doi.org/10.1088/1742-6596/1422/1/012019 DOI: https://doi.org/10.1088/1742-6596/1422/1/012019

Bachman, L. F., Lynch, B. K., & Mason, M. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12(2), 238–257. https://doi.org/10.1177/026553229501200206 DOI: https://doi.org/10.1177/026553229501200206

Bauer, M. W. (2009). The evolution of public understanding of science: Discourse and comparative evidence. Science, Technology and Society, 14(2), 221-240. https://doi.org/10.1177/097172180901400202 DOI: https://doi.org/10.1177/097172180901400202

Bauer, M. W., Allum, N., & Miller, S. (2007). What can we learn from 25 years of PUS survey research? Liberating and expanding the agenda. Public Understanding of Science, 16(1),

79-95. https://doi.org/10.1177/0963662506071287 DOI: https://doi.org/10.1177/0963662506071287

Bybee, R., & McCrae, B. (2011). Scientific literacy and student attitudes: Perspectives from PISA 2006 science. International Journal of Science Education, 33(1), 7-26. https://doi.org/10.1080/09500693.2010.518644 DOI: https://doi.org/10.1080/09500693.2010.518644

Bybee, R., McCrae, B., & Laurie, R. (2009). PISA 2006: An assessment of scientific literacy. Journal of Research in Science Teaching, 46(8), 865-883. https://doi.org/10.1002/tea.20333 DOI: https://doi.org/10.1002/tea.20333

Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications, and programming (2nd ed.). Lawrence Erlbaum Associates. https://psycnet.apa.org/record/2009-14446-000

Chevalier, C. D., Ashley, D. C., & Rushin, J. W. (2007). Acquisition and retention of quantitative communication skills in an undergraduate biology curriculum: Long‐term retention results. Journal of College Science Teaching, 39(2000), 64-71. https://www.jstor.org/stable/42993821

Colon‐Berlingeri, M., & Burrowes, P. A. (2011). Teaching biology through statistics: Application of statistical methods in genetics and zoology courses. CBE—Life Sciences Education, 10(3), 259–267. https://doi.org/10.1187/cbe.10-11-0137 DOI: https://doi.org/10.1187/cbe.10-11-0137

Coppi, M., Fialho, I., & Cid, M. (2023). Developing a scientific literacy assessment instrument for Portuguese 3rd cycle students. Education Sciences, 13(9), 941. https://doi.org/10.3390/educsci13090941 DOI: https://doi.org/10.3390/educsci13090941

Crocker, L., Algina, J., Staudt, M., Mercurio, S., Hintz, K., & Walker, R. A. (2008). Introduction to classical and modern test theory. University Press.

Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39-50. https://doi.org/10.1177/002224378101800104 DOI: https://doi.org/10.1177/002224378101800104

Garvin‐Doxas, K., & Klymkowsky, M. W. (2008). Understanding randomness and its impact on student learning: Lessons learned from building the Biology Concept Inventory (BCI). CBE—Life Sciences Education, 7(2), 227-233. https://doi.org/10.1187/cbe.07-08-0063 DOI: https://doi.org/10.1187/cbe.07-08-0063

Gormally, C., Brickman, P., & Lutz, M. (2012). Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates’ evaluation of scientific information and arguments. CBE—Life Sciences Education, 11(4), 364-377. https://doi.org/10.1187/cbe.12-03-0026 DOI: https://doi.org/10.1187/cbe.12-03-0026

Gottlieb, M., Bailitz, J., Fix, M., Shappell, E., & Wagner, M. J. (2023). Educator’s blueprint: A how‐to guide for developing high‐quality multiple‐choice questions. AEM Education and Training, 7(1), e10836. https://doi.org/10.1002/aet2.10836 DOI: https://doi.org/10.1002/aet2.10836

Hager, P., & Beckett, D. (2020). The emergence of complexity: Rethinking education as a social science. International Journal of Lifelong Education, 39(5-6), 453-467. https://doi.org/10.1080/02601370.2020.1860558 DOI: https://doi.org/10.1080/02601370.2020.1860558

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate statistical analysis (7th ed.). Pearson.

Helendra, H., & Sari, D. R. (2021). Pengembangan instrumen asesmen berbasis literasi sains tentang materi sistem ekskresi dan sistem pernapasan. Jurnal Ilmiah Pendidikan Profesi Guru, 4(1), 17-25. https://doi.org/10.23887/jippg.v4i1.29860 DOI: https://doi.org/10.23887/jippg.v4i1.29860

Istyadji, M., & Sauqina. (2023). Conception of scientific literacy in the development of scientific literacy assessment tools: A systematic theoretical review. Journal of Turkish Science Education, 20(2), 281-308. https://doi.org/10.36681/tused.2023.016 DOI: https://doi.org/10.36681/tused.2023.016

Jia, B., He, D., & Zhu, Z. (2020). Quality and feature of multiple‐choice questions in education. Problems of Education in the 21st Century, 78(4), 576-594. https://doi.org/10.33225/pec/20.78.576 DOI: https://doi.org/10.33225/pec/20.78.576

Kingston, N. M., & Dorans, N. J. (1984). Item location effects and their implications for IRT equating and adaptive testing. Applied Psychological Measurement, 8(2), 147-154. https://doi.org/10.1177/014662168400800202 DOI: https://doi.org/10.1177/014662168400800202

Kline, R. B. (2016). Principles and practice of structural equation modeling (4th ed.). The Guilford Press.

Kutner, M., Greenberg, E., Jin, Y., & Christine. (2002). The health literacy of America’s adults: Results from the 2003 National Assessment of Adult Literacy. Pharmacotherapy, 22(5),

282-302. https://doi.org/10.1592/phco.22.5.282.33191 DOI: https://doi.org/10.1592/phco.22.5.282.33191

Levy, E. H. (2010). Science in the liberal arts and sciences. In J. Meinwald & J. G. Hildebrand (Eds.), Science and the educated American: A core component of liberal education (41-56). American Academy of Arts and Sciences. https://www.amacad.org/content/publications/publication.aspx?d=333

Maienschein, J. (1998). Scientific literacy. Science, 281(5379), 917. https://acortar.link/oUUDfp DOI: https://doi.org/10.1126/science.281.5379.917

Melser, M. C., Steiner-Hofbauer, V., Lilaj, B., Agis, H., Knaus, A., & Holzinger, A. (2020). Knowledge, application, and how about competence? Qualitative assessment of multiple‐choice questions for dental students. Medical Education Online, 25(1). https://doi.org/10.1080/10872981.2020.1714199 DOI: https://doi.org/10.1080/10872981.2020.1714199

Marsteller, P., de Pillis, L., Findley, A., Joplin, K., Pelesko, J., Nelson, K., Thompson, K., Usher, D., & Watkins, J. (2010). Toward integration: From quantitative biology to mathbio‐biomath? CBE—Life Sciences Education, 9(3), 165-171. https://doi.org/10.1187/cbe.10-03-0053 DOI: https://doi.org/10.1187/cbe.10-03-0053

Mesci, G., Erdas‐Kartal, E., & Yeşildağ‐Hasançebi, F. (2025). The effect of argumentation-based inquiry on preservice teachers’ understanding of the nature of science, inquiry, and scientific literacy. Teaching and Teacher Education, 162, 105064. https://doi.org/10.1016/j.tate.2025.105064 DOI: https://doi.org/10.1016/j.tate.2025.105064

Moreno, R., Martínez, R., & Muñiz, J. (2015). Guidelines based on validity criteria for the development of multiple‐choice items. Psicothema, 27(4), 388-394. https://doi.org/10.7334/psicothema2015.110 DOI: https://doi.org/10.7334/psicothema2015.110

Mulyana, V., & Desnita, D. (2023a). Empirical validity and reliability of the scientific literacy assessment instrument based on the Tornado Physics Enrichment Book. Jurnal Penelitian Pendidikan IPA, 9(5), 3961-3967. https://doi.org/10.29303/jppipa.v9i5.3290 DOI: https://doi.org/10.29303/jppipa.v9i5.3290

Mulyana, V., & Desnita, D. (2023b). Validity and practicality of scientific literacy assessment instruments based on Tornado Physics Enrichment Book. Jurnal Ilmiah Pendidikan Fisika Al‐Biruni, 12(1), 11. https://doi.org/10.24042/jipfalbiruni.v12i1.15934 DOI: https://doi.org/10.24042/jipfalbiruni.v12i1.15934

Nasution, I. B., Liliawati, W., & Hasanah, L. (2019). Development of scientific literacy instruments based on PISA framework for high school students on global warming topic. Journal of Physics: Conference Series, 1157, 032063.

https://doi.org/10.1088/1742-6596/1157/3/032063 DOI: https://doi.org/10.1088/1742-6596/1157/3/032063

National Committee on Science Education Standards. (1996). National Science Education Standards (Vol. 26, No. 4). http://eprints.uanl.mx/5481/1/1020149995.PDF

National Research Council. (1999). Being fluent with information technology. National Academy Press.

Nuraini, N., & Waluyo, E. (2021). Pengembangan desain instruksional model project-based learning terintegrasi keterampilan proses sains untuk meningkatkan literasi sains. Jurnal IPA & Pembelajaran IPA, 5(1), 101-111. https://doi.org/10.24815/jipi.v5i1.20145 DOI: https://doi.org/10.24815/jipi.v5i1.20145

OECD. (2019). PISA 2018 results: Combined executive summaries (Vols. I–III). Organization for Economic Co-operation and Development. https://doi.org/10.1787/g222d18af-en DOI: https://doi.org/10.1787/g222d18af-en

OECD. (2022). PISA 2022 results (Volume I): The state of learning and equity in education. Organization for Economic Co-operation and Development. https://acortar.link/2N4P75

Pellegrino, J. W., & Quellmalz, E. S. (2010). Perspectives on the integration of technology and assessment. Journal of Research on Technology in Education, 43(2), 119-134. https://doi.org/10.1080/15391523.2010.10782565 DOI: https://doi.org/10.1080/15391523.2010.10782565

Pimentel‐Velázquez, C. (2025). CLIL and materials for learning: A rubric for analysis. European Public & Social Innovation Review, 10, 1–19. https://doi.org/10.31637/EPSIR-2025-1564 DOI: https://doi.org/10.31637/epsir-2025-1564

Pratiwi, S. A., Syamsiah, S., & Bahri, A. (2022). Pengembangan instrumen penilaian kemampuan literasi sains pada pembelajaran biologi. Biology Teaching and Learning, 5(1), 1-10. https://doi.org/10.35580/btl.v5i1.33834 DOI: https://doi.org/10.35580/btl.v5i1.33834

Quitadamo, I. J., Faiola, C. L., Johnson, J. E., & Kurtz, M. J. (2008). Community-based inquiry improves critical thinking in general education biology. CBE—Life Sciences Education, 7(3), 327–337. https://doi.org/10.1187/cbe.07-11-0097 DOI: https://doi.org/10.1187/cbe.07-11-0097

Rusilowati, A. (2016). Pengembangan bahan ajar berbasis literasi sains materi fluida statis. UPEJ Unnes Physics Education Journal, 5(3), 25-31. https://doi.org/10.15294/upej.v5i3.13726

Rutherford, J., & Ahlgren, A. (1990). Science for all Americans. Oxford University Press.

Santiago, B. J., Ramírez, J. M. O., Rodríguez‐Reséndiz, J., Dector, A., García, R. G., González‐Durán, J. E. E., & Sánchez, F. F. (2020). Learning management system‐based evaluation to determine academic efficiency performance. Sustainability, 12(10), 4256. https://doi.org/10.3390/su12104256 DOI: https://doi.org/10.3390/su12104256

Schraw, G. (2010). Measuring self‐regulation in computer‐based learning environments. Educational Psychologist, 45(4), 258-266. https://doi.org/10.1080/00461520.2010.515936 DOI: https://doi.org/10.1080/00461520.2010.515936

Shi, J., Wood, W. B., Martin, J. M., Guild, N. A., Vicens, Q., & Knight, J. K. (2010). A diagnostic assessment for introductory molecular and cell biology. CBE—Life Sciences Education, 9(4), 453-461. https://doi.org/10.1187/cbe.10-04-0055 DOI: https://doi.org/10.1187/cbe.10-04-0055

Smith, M. K., Wood, W. B., & Knight, J. K. (2008). The genetics concept assessment: A new concept inventory for gauging student understanding of genetics. CBE Life Sciences Education, 7(4), 422-430. https://doi.org/10.1187/cbe.08-08-0045 DOI: https://doi.org/10.1187/cbe.08-08-0045

Tsui, C. Y., & Treagust, D. (2010). Evaluating secondary students’ scientific reasoning in genetics using a two‐tier diagnostic instrument. International Journal of Science Education, 32(8), 1073-1098. https://doi.org/10.1080/09500690902951429 DOI: https://doi.org/10.1080/09500690902951429