Does generative-ai feedback in formative assessment improve higher-order thinking: motivational pathways in vocational safety education

Jurnal Riset Tindakan Indonesia October 17, 2026 Volume 11, Number 2 4 authors

Abstract

Occupational health and safety (OHS) education in vocational high schools requires higher-order thinking skills (HOTS), yet formative assessment is often limited by delayed teacher feedback. This study examined whether generative AI-supported formative assessment using ChatGPT and Gemini improved HOTS and learning motivation compared with conventional teacher-led assessment, and explored the relationship between perceptions of AI-assisted assessment, motivation, and HOTS. A posttest-only nonequivalent control group quasi-experimental design involved 60 Grade X Machining Engineering students at SMK Negeri 1 Padang, divided into two intact classes of 30 students each. HOTS was measured using a 20-item reasoned multiple-choice test covering Bloom’s cognitive levels C4–C6, while motivation and perceptions of AI-assisted assessment were measured using validated questionnaires. Independent-samples t-tests were used for the primary analysis, followed by exploratory partial least squares structural equation modelling (PLS-SEM) in the experimental group. No significant differences were found between the AI-assisted and conventional assessment groups in HOTS scores (81.73 vs. 80.87, p = .417, d = 0.21) or motivation (54.03 vs. 54.83, p = .497, d = −0.18). Within the experimental group, perceptions of AI-assisted assessment were positively associated with motivation (β = 0.743, p < .001), and motivation was positively associated with HOTS (β = 0.476, p = .023). Although the indirect effect was statistically significant (β = 0.354, p = .042), its bias-corrected confidence interval included zero, making the mediation evidence inconclusive. These findings do not demonstrate that AI-assisted formative assessment improves HOTS or motivation beyond conventional teacher feedback. However, the observed motivational relationships warrant further investigation using larger samples, baseline measurements, and longer interventions.

Authors

  1. Tri Juliantika

    Universitas Negeri Padang

  2. Waskito Waskito

    Universitas Negeri Padang

  3. Refdinal Refdinal

    Universitas Negeri Padang

  4. Yeka Hendriyani

    Universitas Negeri Padang

Keywords

Formative assessmentArtificial intelligenceLearning motivationHigher-order thinking skillsOccupational health and safetyPLS-SEM

References

  1. Adaby, M. A., Sugiarto, T., Arif, A., & Muslim. (2026). Analisis tingkat pengetahuan pada soal UAS produktif otomotif semester genap 2024/2025 kelas XII TKR SMK Negeri Teknologi di Kota Padang [Analysis of knowledge levels in productive automotive final-semester examination items for Grade XII light-vehicle engineering students]. Ensiklopedia of Journal, 8(2), 98–105. https://doi.org/10.33559/eoj.v8i2.3710
  2. Aldahlaki, Z. H. A. (2026). The role of artificial intelligence in enhancing learning motivation through self and immediate feedback. Journal of Technology and System Information, 3(1), 1–7. https://doi.org/10.47134/jtsi.v3i1.5570
  3. An, P. M. N., Anh, T. N. P., & Dinh, C.-T. (2025). Investigating the mediating effects of motivation on academic performance in a synchronous EFL online learning. Journal of e-Learning and Knowledge Society, 21(1), 224–235. https://doi.org/10.20368/1971-8829/1135976
  4. Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
  5. Cao, C., Taddese, E. T., Gebresilase, B. M., Shorouk, A., & Berhanu, K. Z. (2025). The influence of learning motivation on academic performance in Chinese vocational college students: A self-determination theory perspective. International Journal of Innovative Research and Scientific Studies, 8(6), 3297–3309. https://doi.org/10.53894/ijirss.v8i6.10339
  6. Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155
  7. Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153
  8. Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
  9. Fidan, M., & Gencel, N. (2022). Supporting the instructional videos with chatbot and peer feedback mechanisms in online learning: The effects on learning performance and intrinsic motivation. Journal of Educational Computing Research, 60(7), 1716–1741. https://doi.org/10.1177/07356331221077901
  10. Garzón, J., Patiño, E., & Marulanda, C. (2025). Systematic review of artificial intelligence in education: Trends, benefits, and challenges. Multimodal Technologies and Interaction, 9(8), 84. https://doi.org/10.3390/mti9080084
  11. Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2017). A primer on partial least squares structural equation modeling (PLS-SEM) (2nd ed.). SAGE Publications.
  12. Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of PLS-SEM. European Business Review, 31(1), 2–24. https://doi.org/10.1108/EBR-11-2018-0203
  13. Hanazahra, A., Efriyanti, L., Satria, E., & Indra, Y. (2025). Evaluasi implementasi pembelajaran adaptif berbasis artificial intelligence (AI) di SMKN 1 Bukittinggi [Evaluation of the implementation of artificial intelligence-based adaptive learning at SMKN 1 Bukittinggi]. Jurnal Pendidikan Teknologi Informasi dan Vokasional, 7(2), 1–13. https://doi.org/http://dx.doi.org/10.23960/jpvti
  14. Hanis, M., & Wahyudin, D. (2024). Pemanfaatan artificial intelligence (AI) dalam penyusunan asesmen pembelajaran bagi guru sekolah dasar [Utilizing artificial intelligence (AI) in developing learning assessments for elementary school teachers]. Jurnal Ilmiah Profesi Pendidikan, 9(2). https://doi.org/10.29303/jipp.v9i2.2252
  15. Hapsari, D. D., Ramadhani, G. Y., & Ikramullah, N. I. (2024). Literature review: Pengaruh artificial intelligence (AI) terhadap motivasi belajar peserta didik [Literature review: The effect of artificial intelligence (AI) on students’ learning motivation]. Jurnal Empati, 13(4), 313–324. https://doi.org/10.14710/empati.2024.46697
  16. Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
  17. Hidayati, H. N. (2025). Pemanfaatan artificial intelligence pada penilaian pembelajaran mendalam [Utilizing artificial intelligence in deep learning assessment]. Afeksi: Jurnal Penelitian dan Evaluasi Pendidikan, 6(5), 900–908. https://doi.org/10.59698/afeksi.v6i5.531
  18. Hukom, J. (2025). Model integrasi feedback digital berbasis AI untuk pembelajaran berorientasi kompetensi [An AI-based digital feedback integration model for competency-oriented learning]. IDENTIK: Jurnal Ilmu Ekonomi, Pendidikan dan Teknik, 2(4), 169–174.
  19. Ji, H., Suo, L., & Chen, H. (2024). AI performance assessment in blended learning: Mechanisms and effects on students’ continuous learning motivation. Frontiers in Psychology, 15, 1447680. https://doi.org/10.3389/fpsyg.2024.1447680
  20. Jian, Z. (2022). Sustainable engagement and academic achievement under impact of academic self-efficacy through mediation of learning agility: Evidence from music education students. Frontiers in Psychology, 13, 899706. https://doi.org/10.3389/fpsyg.2022.899706
  21. Keller, J. M. (1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2–10. https://doi.org/10.1007/BF02905780
  22. Keong, L. M. (2025). AI-personalised learning in higher education: A study on learning outcomes and motivation among university students. International Journal of Research and Innovation in Social Science, 9(5), 6075–6085. https://doi.org/10.47772/IJRISS.2025.905000473
  23. Kock, N., & Hadaya, P. (2018). Minimum sample size estimation in PLS-SEM: The inverse square root and gamma-exponential methods. Information Systems Journal, 28(1), 227–261. https://doi.org/10.1111/isj.12131
  24. Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory Into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2
  25. Larasati, S. P., Sukma, D., Karneli, Y., Fikri, M., Amat, M. A. C., & Rismi, R. (2026). AI-driven feedback and its impact on student motivation in online learning. G-COUNS: Jurnal Bimbingan dan Konseling, 10(2), 1071–1088. https://doi.org/10.31316/g-couns.v10i02.8584
  26. Nafiati, D. A. (2021). Revisi taksonomi Bloom: Kognitif, afektif, dan psikomotorik [Revision of Bloom’s taxonomy: Cognitive, affective, and psychomotor]. Humanika: Kajian Ilmiah Mata Kuliah Umum, 21(2), 151–172. https://doi.org/10.21831/hum.v21i2.29252
  27. Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090
  28. Pratama, D. H., & Sugiarto, T. (2025). Literasi keselamatan dan kesehatan kerja sebagai prediktor kedisiplinan praktikum siswa SMK program Teknik Kendaraan Ringan [Occupational health and safety literacy as a predictor of practical-work discipline among light-vehicle engineering students]. JTPVI: Jurnal Teknologi dan Pendidikan Vokasi Indonesia, 3(4), 891–902. https://doi.org/10.24036/jtpvi.v3i4.222
  29. Rahmawati, Y. (2025). Kecerdasan buatan di ruang kelas: Analisis penurunan motivasi belajar dan kemampuan berpikir kritis siswa SMP Negeri 1 Ganding Sumenep [Artificial intelligence in the classroom: An analysis of declining learning motivation and critical-thinking ability among students of SMP Negeri 1 Ganding Sumenep]. Jurnal Teknologi Pendidikan, 2(4), 1–8. https://doi.org/10.47134/jtp.v2i4.1640
  30. Roediger, H. L., III, & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x
  31. Sari, I. P., Hatta, T. A., Manurung, A. A., & Dachi, S. W. (2024). The influence of a high-order thinking (HOTS)-based learning model on students’ midterm question analysis. Aksaqila International Humanities and Social Sciences Journal, 3(2), 58–66. https://doi.org/10.30596/aihss.v3i2.505
  32. Simões, S., Oliveira, T., & Nunes, C. (2022). Influence of computers in students’ academic achievement. Heliyon, 8, e09004. https://doi.org/10.1016/j.heliyon.2022.e09004
  33. Wahyu, R., Pertiwi, L., Kulsum, L. U., & Hanifah, I. A. (2024). Evaluating the impact of artificial intelligence-based learning methods on students’ motivation and academic achievement. International Journal of Post Axial: Futuristic Teaching and Learning, 2(1), 49–58. https://doi.org/10.59944/postaxial.v2i1.279
  34. Waskito, Irzal, Wulansari, R. E., & Ya, K. Z. (2022). The adventure of formative assessment with active feedback in the vocational learning: The empirical effect for increasing students’ achievement. Journal of Technical Education and Training, 14(1), 54–62. https://doi.org/10.30880/jtet.2022.14.01.005
  35. Widyawati, N. K. (2020). Pentingnya penguasaan konsep keselamatan dan kesehatan kerja (K3) dalam mendukung kinerja calon lulusan pendidikan kejuruan di dunia kerja [The importance of mastering occupational health and safety (OHS) concepts in supporting the workplace performance of prospective vocational graduates]. Jurnal Bosaparis: Pendidikan Kesejahteraan Keluarga, 11(3), 87–93.
  36. Wu, J., Tlili, A., Salha, S., Mizza, D., Saqr, M., López-Pernas, S., & Huang, R. (2025). Unlocking the potential of artificial intelligence in improving learning achievement in blended learning: A meta-analysis. Frontiers in Psychology, 16, 1691414. https://doi.org/10.3389/fpsyg.2025.1691414
  37. Yin, J., Goh, T.-T., & Hu, Y. (2024). Using a chatbot to provide formative feedback: A longitudinal study of intrinsic motivation, cognitive load, and learning performance. IEEE Transactions on Learning Technologies, 17, 1378–1389. https://doi.org/10.1109/TLT.2024.3364015