Evaluating the performance of 3 large language models in higher education essay-like assessments in 2024 and 2026

Authors

DOI:

https://doi.org/10.32674/tfptev94

Keywords:

large language models, artificial intelligence, essays, assessments

Abstract

Recent advances in artificial intelligence, especially generative large language models (LLMs), have transformed the higher education sector, raising concerns with academic integrity. The current literature lacks direct comparative analyses between LLMs. In the current study, we evaluated and compared the performance of ChatGPT, Gemini and Copilot (free versions) in 2024 and 2026, following prompts related to coursework essay assessments in computer science education or biomedical science. Our results indicate that LLMs struggle to abide by the word count beyond 1000 words, with Gemini presenting greater deviations. Copilot presented the lowest frequency of reference hallucinations. Overall performance of the 3 LLMs did not reveal any statistically significant differences. Quality assessments of the outputs revealed issues with content and criticality for all the LLMs. Similar performances were observed in 2024 and 2026.                 

Author Biographies

  • David Hunt, University of Worcester

    DAVID HUNT, M.A., is a Senior Lecturer in Secondary Education (Computer Science) at the University of Worcester. Email: d.hunt@worc.ac.uk

  • Mathieu Di Miceli, University of Leeds

    MATHIEU DI MICELI, Ph.D., is a Lecturer in Human Anatomy and Physiology at the University of Leeds. Email: m.c.dimiceli@leeds.ac.uk  

References

Aljamaan, F., Temsah, M. H., Altamimi, I., Al-Eyadhy, A., Jamal, A., Alhasan, K., Mesallam, T. A., Farahat, M., & Malki, K. H. (2024). Reference hallucination score for medical artificial intelligence chatbots: Development and usability study. JMIR Medical Informatics, 12(1), e54345. https://doi.org/10.2196/54345 DOI: https://doi.org/10.2196/54345

Amin, K. S., Mayes, L. C., Khosla, P., & Doshi, R. H. (2024). Assessing the Efficacy of Large Language Models in Health Literacy: A Comprehensive Cross-Sectional Study. The Yale Journal of Biology and Medicine, 97(1), 17–27. https://doi.org/10.59249/ZTOZ1966 DOI: https://doi.org/10.59249/ZTOZ1966

Athaluri, S. A., Manthena, S. V., Kesapragada, V. S. R. K. M., Yarlagadda, V., Dave, T., & Duddumpudi, R. T. S. (2023). Exploring the boundaries of reality: investigating the phenomenon of artificial intelligence hallucination in scientific writing through ChatGPT references. Cureus, 15(4), e37432. https://doi.org/10.7759/cureus.37432 DOI: https://doi.org/10.7759/cureus.37432

Azamfirei, R., Kudchadkar, S. R., & Fackler, J. (2023). Large language models and the perils of their hallucinations. Critical Care, 27(120). https://doi.org/10.1186/s13054-023-04393-x DOI: https://doi.org/10.1186/s13054-023-04393-x

Azevedo, L., Mallinson, D. J., Wang, J., Robles, P., & Best, E. (2024). AI policies, equity, and morality and the implications for faculty in higher education. Public Integrity, 28(2), 186-201. https://doi.org/10.1080/10999922.2024.2414957 DOI: https://doi.org/10.1080/10999922.2024.2414957

Bai, Y., Zhang, J., Lv, X., Zheng, L., Zhu, S., Hou, L., Dong, Y., Tang, J., & Li, J. (2024). LongWriter: unleashing 10,000+ word generation from long context LLMs (arXiv:2408.07055). arXiv. https://doi.org/10.48550/arXiv.2408.07055

Benton, T. (2019). Which is better: One experienced marker or many inexperienced markers? Research Matters: A Cambridge Assessment Publication, 28. https://doi.org/10.17863/CAM.100393

Bernabei, M., Colabianchi, S., Falegnami, A., & Costantino, F. (2023). Students’ use of large language models in engineering education: A case study on technology acceptance, perceptions, efficacy, and detection chances. Computers and Education: Artificial Intelligence, 5, 100172. https://doi.org/10.1016/j.caeai.2023.100172 DOI: https://doi.org/10.1016/j.caeai.2023.100172

Beutel, G., Geerits, E., & Kielstein, J. T. (2023). Artificial hallucination: GPT on LSD? Critical Care, 27, 148. https://doi.org/10.1186/s13054-023-04425-6 DOI: https://doi.org/10.1186/s13054-023-04425-6

Bewersdorff, A., Hartmann, C., Hornberger, M., Seßler, K., Bannert, M., Kasneci, E., Kasneci, G., Zhai, X., & Nerdel, C. (2025). Taking the next step with generative artificial intelligence: The transformative role of multimodal large language models in science education. Learning and Individual Differences, 118, 102601. https://doi.org/10.1016/j.lindif.2024.102601 DOI: https://doi.org/10.1016/j.lindif.2024.102601

Bhattacharyya, M., Miller, V. M., Bhattacharyya, D., & Miller, L. E. (2023). High rates of fabricated and inaccurate references in ChatGPT-generated medical content. Cureus, 15(5), e39238. https://doi.org/10.7759/cureus.39238 DOI: https://doi.org/10.7759/cureus.39238

Bird, F. L., & Yucel, R. (2013). Improving marking reliability of scientific writing with the Developing Understanding of Assessment for Learning programme. Assessment & Evaluation in Higher Education, 38(5), 536–553. https://doi.org/10.1080/02602938.2012.658155 DOI: https://doi.org/10.1080/02602938.2012.658155

Bittle, K., & El-Gayar, O. (2025). Generative AI and Academic Integrity in Higher Education: A Systematic Review and Research Agenda. Information, 16(4), 296. https://doi.org/10.3390/info16040296 DOI: https://doi.org/10.3390/info16040296

Brookhart, S. M. (2018). Appropriate Criteria: Key to Effective Rubrics. Frontiers in Education, 3, 22. https://doi.org/10.3389/feduc.2018.00022 DOI: https://doi.org/10.3389/feduc.2018.00022

Campbell, H., Bluck, T., Curry, E., Harris, D., Pike, B., & Wright, B. (2024). Should we still teach or learn coding? A postgraduate student perspective on the use of large language models for coding in ecology and evolution. Methods in Ecology and Evolution, 15(10), 1767–1770. https://doi.org/10.1111/2041-210X.14396 DOI: https://doi.org/10.1111/2041-210X.14396

Caucheteux, C., & King, J. R. (2022). Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1), 134. https://doi.org/10.1038/s42003-022-03036-1 DOI: https://doi.org/10.1038/s42003-022-03036-1

Chakraborty, S., Dann, C., Mandal, A., Dann, B., Paul, M., & Hafeez-Baig, A. (2021). Effects of rubric quality on marker variation in higher education. Studies in Educational Evaluation, 70, 100997. https://doi.org/10.1016/j.stueduc.2021.100997 DOI: https://doi.org/10.1016/j.stueduc.2021.100997

Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20(1), 43. https://doi.org/10.1186/s41239-023-00411-8 DOI: https://doi.org/10.1186/s41239-023-00411-8

Chelli, M., Descamps, J., Lavoué, V., Trojani, C., Azar, M., Deckert, M., Raynier, J. L., Clowez, G., Boileau, P., & Ruetsch-Chelli, C. (2024). Hallucination rates and reference accuracy of ChatGPT and Bard for systematic reviews: Comparative Analysis. Journal of Medical Internet Research, 26(1), e53164. https://doi.org/10.2196/53164 DOI: https://doi.org/10.2196/53164

Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. (2022). ChatGPT goes to law school. Journal of Legal Education, 71(3), 387–400. https://doi.org/10.2139/ssrn.4335905 DOI: https://doi.org/10.2139/ssrn.4335905

Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J. N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S. C., Unger, M., Veldhuizen, G. P., Wagner, S. J., & Kather, J. N. (2023). The future landscape of large language models in medicine. Communications Medicine, 3(1), 1–8. https://doi.org/10.1038/s43856-023-00370-1 DOI: https://doi.org/10.1038/s43856-023-00370-1

Coen, A., & Cuddapah, J. L. (2026). Artificial intelligence in action: How preservice teachers embark on the fast-paced journey of mindful AI use. American Journal of STEM Education, 20, 111–132. https://doi.org/10.32674/keqjjt69 DOI: https://doi.org/10.32674/keqjjt69

Contreras Kallens, P., Kristensen-McLachlan, R. D., & Christiansen, M. H. (2023). Large language models demonstrate the potential of statistical learning in language. Cognitive Science, 47(3), e13256. https://doi.org/10.1111/cogs.13256 DOI: https://doi.org/10.1111/cogs.13256

Cooper, N., Clark, A. T., Lecomte, N., Qiao, H., & Ellison, A. M. (2024). Harnessing large language models for coding, teaching and inclusion to empower research in ecology and evolution. Methods in Ecology and Evolution, 15(10), 1757–1763. https://doi.org/10.1111/2041-210X.14325 DOI: https://doi.org/10.1111/2041-210X.14325

Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228-239. https://doi.org/10.1080/14703297.2023.2190148 DOI: https://doi.org/10.1080/14703297.2023.2190148

Deep, P. D., Edgington, W. D., Ghosh, N., & Rahaman, M. S. (2025). Evaluating the Effectiveness and Ethical Implications of AI Detection Tools in Higher Education. Information, 16(10), 905. https://doi.org/10.3390/info16100905 DOI: https://doi.org/10.3390/info16100905

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: pre-training of deep bidirectional transformers for language understanding (arXiv:1810.04805). arXiv. https://doi.org/10.48550/arXiv.1810.04805

Dilling, M., & Owen, L. (2024). Designing politics and IR assessments in the era of AI: an empirical investigation into ChatGPT’s output across Bloom’s revised taxonomy. Journal of Political Science Education, 21(2), 290-309. https://doi.org/10.1080/15512169.2024.2408767 DOI: https://doi.org/10.1080/15512169.2024.2408767

Eke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? Journal of Responsible Technology, 13, 100060. https://doi.org/10.1016/j.jrt.2023.100060 DOI: https://doi.org/10.1016/j.jrt.2023.100060

El Fathi, T., Saad, A., Larhzil, H., Lamri, D., & Al Ibrahmi, E. M. (2025). Integrating generative AI into STEM education: Enhancing conceptual understanding, addressing misconceptions, and assessing student acceptance. Disciplinary and Interdisciplinary Science Education Research, 7(1), 6. https://doi.org/10.1186/s43031-025-00125-z DOI: https://doi.org/10.1186/s43031-025-00125-z

Erez, Y., & Hazzan, O. (2025, May 22). Students in advanced computational fields are accelerated early adapters of generative AI technology – Communications of the ACM. https://cacm.acm.org/blogcacm/students-in-advanced-computational-fields-are-accelerated-early-adapters-of-generative-ai-technology/

Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017), 625–630. https://doi.org/10.1038/s41586-024-07421-0 DOI: https://doi.org/10.1038/s41586-024-07421-0

Farrelly, T., & Baker, N. (2023). Generative Artificial Intelligence: Implications and Considerations for Higher Education Practice. Education Sciences, 13(11), Article 1109. https://doi.org/10.3390/educsci13111109 DOI: https://doi.org/10.3390/educsci13111109

Feng, H., Ronzano, F., LaFleur, J., Garber, M., de Oliveira, R., Rough, K., Roth, K., Nanavati, J., Abidine, K. Z. E., & Mack, C. (2024). Evaluation of large language model performance on the biomedical language understanding and reasoning benchmark: comparative study. medRxiv. https://doi.org/10.1101/2024.05.17.24307411 DOI: https://doi.org/10.1101/2024.05.17.24307411

Giannakopoulos, K., Kavadella, A., Aaqel Salim, A., Stamatopoulos, V., & Kaklamanos, E. G. (2023). Evaluation of the performance of generative AI large language models ChatGPT, Google Bard, and Microsoft Bing chat in supporting evidence-based dentistry: comparative mixed methods study. Journal of Medical Internet Research, 25, e51580. https://doi.org/10.2196/51580 DOI: https://doi.org/10.2196/51580

Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How does ChatGPT perform on the United States medical licensing examination (USMLE)? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9(1), e45312. https://doi.org/10.2196/45312 DOI: https://doi.org/10.2196/45312

Giuffrè, M., You, K., & Shung, D. L. (2024). Evaluating ChatGPT in Medical Contexts: The Imperative to Guard Against Hallucinations and Partial Accuracies. Clinical Gastroenterology and Hepatology, 22(5), 1145–1146. https://doi.org/10.1016/j.cgh.2023.09.035 DOI: https://doi.org/10.1016/j.cgh.2023.09.035

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., Jansen, A., Gazula, H., Choe, G., Rao, A., Kim, C., Casto, C., Fanda, L., Doyle, W., Friedman, D., … Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3), 369–380. https://doi.org/10.1038/s41593-022-01026-4 DOI: https://doi.org/10.1038/s41593-022-01026-4

Google AI. (2024). Gemini. Language Model. https://gemini.google.com/app

Google LLC. (2024a). Generative AI. Google Cloud. https://cloud.google.com/vertex-ai/generative-ai/docs/learn/prompts/adjust-parameter-values

Google LLC. (2024b). ‎Gemini Apps’ release updates and improvements. Gemini. https://gemini.google.com/updates

Graham, A. I., Harner, C., & Marsham, S. (2022). Can assessment-specific marking criteria and electronic comment libraries increase student engagement with assessment and feedback? Assessment & Evaluation in Higher Education, 47(7), 1071–1086. https://doi.org/10.1080/02602938.2021.1986468 DOI: https://doi.org/10.1080/02602938.2021.1986468

Gravel, J., D’Amours-Gravel, M., & Osmanlliu, E. (2023). Learning to Fake It: Limited Responses and Fabricated References Provided by ChatGPT for Medical Questions. Mayo Clinic Proceedings: Digital Health, 1(3), 226–234. https://doi.org/10.1016/j.mcpdig.2023.05.004 DOI: https://doi.org/10.1016/j.mcpdig.2023.05.004

Hadra, M., Cambridge, K., & Mesbah, M. (2026). Evaluating the accuracy and reliability of AI content detectors in academic contexts. International Journal for Educational Integrity, 22, 4. https://doi.org/10.1007/s40979-026-00213-1 DOI: https://doi.org/10.1007/s40979-026-00213-1

Hager, P., Jungmann, F., Holland, R., Bhagat, K., Hubrecht, I., Knauer, M., Vielhauer, J., Makowski, M., Braren, R., Kaissis, G., & Rueckert, D. (2024). Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature Medicine, 30(9), 2613–2622. https://doi.org/10.1038/s41591-024-03097-1 DOI: https://doi.org/10.1038/s41591-024-03097-1

Haider, J., Söderström, K. R., Ekström, B., & Rödl, M. (2024). GPT-fabricated scientific papers on Google Scholar: Key features, spread, and implications for preempting evidence manipulation. Harvard Kennedy School Misinformation Review, 5(5), 1-16. https://doi.org/10.37016/mr-2020-156 DOI: https://doi.org/10.37016/mr-2020-156

Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023). Exploring the responses of large language models to beginner programmers’ help requests. ICER ’23 Proceedings of the 2023 ACM Conference on International Computing Education Research, 1, 93–105. https://doi.org/10.1145/3568813.3600139 DOI: https://doi.org/10.1145/3568813.3600139

Huang, X., & Qiao, C. (2024). Enhancing computational thinking Skills through artificial intelligence education at a STEAM high school. Science & Education, 33(2), 383–403. https://doi.org/10.1007/s11191-022-00392-6 DOI: https://doi.org/10.1007/s11191-022-00392-6

Huang, Y., Gomaa, A., Semrau, S., Haderlein, M., Lettmaier, S., Weissmann, T., Grigo, J., Tkhayat, H. B., Frey, B., Gaipl, U., Distel, L., Maier, A., Fietkau, R., Bert, C., & Putz, F. (2023). Benchmarking ChatGPT-4 on a radiation oncology in-training exam and Red Journal Gray Zone cases: Potentials and challenges for AI-assisted medical education and decision making in radiation oncology. Frontiers in Oncology, 13, 1265024. https://doi.org/10.3389/fonc.2023.1265024 DOI: https://doi.org/10.3389/fonc.2023.1265024

Hubert, K. F., Awa, K. N., & Zabelina, D. L. (2024). The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks. Scientific Reports, 14(1), 3440. https://doi.org/10.1038/s41598-024-53303-w DOI: https://doi.org/10.1038/s41598-024-53303-w

Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2024). A survey on large language models for code generation (arXiv:2406.00515). arXiv. https://doi.org/10.48550/arXiv.2406.00515

Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002 DOI: https://doi.org/10.1016/j.edurev.2007.05.002

Khalil, M., & Er, E. (2023). Will ChatGPT get you caught? Rethinking of plagiarism detection. In P. Zaphiris & A. Ioannou (Eds.), Learning and Collaboration Technologies (pp. 475–487). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-34411-4_32 DOI: https://doi.org/10.35542/osf.io/fnh48

Kobak, D., González-Márquez, R., Horvát, E.-Á., & Lause, J. (2024). Delving into ChatGPT usage in academic writing through excess vocabulary (arXiv:2406.07016). arXiv. https://doi.org/10.48550/arXiv.2406.07016

Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), 100779. https://doi.org/10.1016/j.patter.2023.100779 DOI: https://doi.org/10.1016/j.patter.2023.100779

Lubiana, T., Lopes, R., Medeiros, P., Silva, J. C., Goncalves, A. N. A., Maracaja-Coutinho, V., & Nakaya, H. I. (2023). Ten quick tips for harnessing the power of ChatGPT in computational biology. PLoS Computational Biology, 19(8), e1011319. https://doi.org/10.1371/journal.pcbi.1011319 DOI: https://doi.org/10.1371/journal.pcbi.1011319

Lund, B., Mannuru, N. R., Teel, Z. A., Lee, T. H., Ortega, N. J., Simmons, S., & Ward, E. (2026). Student perceptions of AI-assisted writing and academic integrity: ethical concerns, academic misconduct, and use of generative AI in higher education. AI in Education, 1(1), 2. https://doi.org/10.3390/aieduc1010002 DOI: https://doi.org/10.3390/aieduc1010002

Lyness, S. A., Peterson, K., & Yates, K. (2021). Low inter-rater reliability of a high stakes performance assessment of teacher candidates. Education Sciences, 11(10), 648. https://doi.org/10.3390/educsci11100648 DOI: https://doi.org/10.3390/educsci11100648

Mammides, C., & Papadopoulos, H. (2024). The role of large language models in interdisciplinary research: Opportunities, challenges and ways forward. Methods in Ecology and Evolution, 15(10), 1774–1776. https://doi.org/10.1111/2041-210X.14398 DOI: https://doi.org/10.1111/2041-210X.14398

Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50–60. https://doi.org/10.1214/aoms/1177730491 DOI: https://doi.org/10.1214/aoms/1177730491

Mao, J., Chen, B., & Liu, J. C. (2024). Generative Artificial Intelligence in Education and Its Implications for Assessment. TechTrends, 68(1), 58–66. https://doi.org/10.1007/s11528-023-00911-4 DOI: https://doi.org/10.1007/s11528-023-00911-4

McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024). Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences of the United States of America, 121(41), e2322420121. https://doi.org/10.1073/pnas.2322420121 DOI: https://doi.org/10.1073/pnas.2322420121

Meissner, R., Pögelt, A., Ihsberner, K., Grüttmüller, M., Tornack, S., Thor, A., Pengel, N., Wollersheim, H. W., & Hardt, W. (2024). LLM-generated competence-based e-assessment items for higher education mathematics: Methodology and evaluation. Frontiers in Education, 9, 1427502. https://doi.org/10.3389/feduc.2024.1427502 DOI: https://doi.org/10.3389/feduc.2024.1427502

Microsoft Corporation. (2024a). Microsoft CoPilot. https://copilot.microsoft.com

Microsoft Corporation. (2024b). Model selection and temperature settings. https://learn.microsoft.com/en-us/ai-builder/prompt-modelsettings

Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008). Predicting human brain activity associated with the meanings of nouns. Science, 320(5880), 1191–1195. https://doi.org/10.1126/science.1152876 DOI: https://doi.org/10.1126/science.1152876

Mugaanyi, J., Cai, L., Cheng, S., Lu, C., & Huang, J. (2024). Evaluation of large language model performance and reliability for citations and references in scholarly writing: cross-disciplinary study. Journal of Medical Internet Research, 26, e52935. https://doi.org/10.2196/52935 DOI: https://doi.org/10.2196/52935

Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024). Using an LLM to help with code understanding (arXiv:2307.08177). arXiv. https://doi.org/10.48550/arXiv.2307.08177 DOI: https://doi.org/10.1145/3597503.3639187

Nejjar, M., Zacharias, L., Stiehle, F., & Weber, I. (2024). LLMs for science: Usage for code generation and data analysis. Journal of Software: Evolution and Process, 35(1) e2723. https://doi.org/10.1002/smr.2723 DOI: https://doi.org/10.1002/smr.2723

OpenAI. (2022, November 30). Introducing ChatGPT. https://openai.com/index/chatgpt/

OpenAI. (2024). API reference. https://platform.openai.com

OpenAI. (2024, May 13). Introducing GPT-4o and more tools to ChatGPT free users. https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/

Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). The impact of AI on developer productivity: evidence from GitHub Copilot (arXiv:2302.06590). arXiv. https://doi.org/10.48550/arXiv.2302.06590

Petingola, M., Zhang, Y., Yan, Y., & Lin, W. (2025). Integrating ethical AI tools into educational practices for enhancing academic integrity. CUI ’25 Proceedings of the 7th ACM Conference on Conversational User Interfaces, 6, 1–6. https://doi.org/10.1145/3719160.3737626 DOI: https://doi.org/10.1145/3719160.3737626

Polyportis, A. (2023). A longitudinal study on artificial intelligence adoption: Understanding the drivers of ChatGPT usage behavior change in higher education. Frontiers in Artificial Intelligence, 6, 1324398. https://doi.org/10.3389/frai.2023.1324398 DOI: https://doi.org/10.3389/frai.2023.1324398

Poudel, P., Chouchen, M., Subedi, S., Gaulee, U., Bista, K., Bista, K., & Bhattarai, U. (2026). AI adoption and student productivity in higher education: A cross-institutional study. American Journal of STEM Education, 20, 117–134. https://doi.org/10.32674/2wda3n83 DOI: https://doi.org/10.32674/2wda3n83

R Core Team. (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

Radford, A., & Narasimhan, K. (2018). Improving Language Understanding by Generative Pre-Training. https://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035

Redmond-Sanogo, A., Burton, M., Ivy, J., & Maiorca, C. (2026). Generative AI in mathematics, science, and STEM education: research, applications, and emerging themes. School Science and Mathematics, 126(1), 3–8. https://doi.org/10.1111/ssm.70002 DOI: https://doi.org/10.1111/ssm.70002

Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021). The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences of the United States of America, 118(45), e2105646118. https://doi.org/10.1073/pnas.2105646118 DOI: https://doi.org/10.1073/pnas.2105646118

Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52(3/4), 591–611. https://doi.org/10.2307/2333709 DOI: https://doi.org/10.1093/biomet/52.3-4.591

Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023). ChatGPT and other large language models are double-edged swords. Radiology, 307(2), e230163. https://doi.org/10.1148/radiol.230163 DOI: https://doi.org/10.1148/radiol.230163

Šidák, Z. (1967). Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association, 62(318), 626–633. https://doi.org/10.2307/2283989 DOI: https://doi.org/10.1080/01621459.1967.10482935

Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Babiker, A., Schärli, N., Chowdhery, A., Mansfield, P., Demner-Fushman, D., … Natarajan, V. (2023). Large language models encode clinical knowledge. Nature, 620(7972), 172–180. https://doi.org/10.1038/s41586-023-06291-2 DOI: https://doi.org/10.1038/s41586-023-06291-2

Spataro, J. (2024, September 16). Microsoft 365 Copilot Wave 2: Pages, Python in Excel, and agents. Microsoft 365 Blog. https://www.microsoft.com/en-us/microsoft-365/blog/2024/09/16/microsoft-365-copilot-wave-2-pages-python-in-excel-and-agents/

Stone, A. (2022). Student perceptions of academic integrity: a qualitative study of understanding, consequences, and impact. Journal of Academic Ethics, 21, 357-375. https://doi.org/10.1007/s10805-022-09461-5 DOI: https://doi.org/10.1007/s10805-022-09461-5

Strzelecki, A. (2024). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments, 32(9), 5142–5155. https://doi.org/10.1080/10494820.2023.2209881 DOI: https://doi.org/10.1080/10494820.2023.2209881

Susnjak, T., & McIntosh, T. R. (2024). ChatGPT: the end of online exam integrity? Education Sciences, 14(6), 656. https://doi.org/10.3390/educsci14060656 DOI: https://doi.org/10.3390/educsci14060656

Tao, Z., Lin, T.-E., Chen, X., Li, H., Wu, Y., Li, Y., Jin, Z., Huang, F., Tao, D., & Zhou, J. (2024). A survey on self-evolution of large language models (arXiv:2404.14387). arXiv. https://doi.org/10.48550/arXiv.2404.14387

Tarabanis, C., Zahid, S., Mamalis, M., Zhang, K., Kalampokis, E., & Jankelson, L. (2024). Performance of publicly available large language models on internal medicine board-style questions. PLOS Digital Health, 3(9), e0000604. https://doi.org/10.1371/journal.pdig.0000604 DOI: https://doi.org/10.1371/journal.pdig.0000604

Taylor, B., Kisby, F., & Reedy, A. (2024). Rubrics in higher education: An exploration of undergraduate students’ understanding and perspectives. Assessment & Evaluation in Higher Education, 49(6), 799–809. https://doi.org/10.1080/02602938.2023.2299330 DOI: https://doi.org/10.1080/02602938.2023.2299330

Tian, S. (2024). The integration of computational thinking and artificial intelligence serves to enhance the cognitive processes and skill acquisition of students. ISAIE ’24 Proceedings of the 2024 International Symposium on Artificial Intelligence for Education, 564–567. https://doi.org/10.1145/3700297.3700394 DOI: https://doi.org/10.1145/3700297.3700394

Trenker, J., Menon, S. S., & Blumtritt, C. (2023, July). A steam-engine moment to the computer age, generative AI is boosting efficiency and creativity to unprecedented heights. Statista. https://www.statista.com/site/insights-compass-ai-generative-ai

Truhn, D., Weber, C. D., Braun, B. J., Bressem, K., Kather, J. N., Kuhl, C., & Nebelung, S. (2023). A pilot study on the efficacy of GPT-4 in providing orthopedic treatment recommendations from MRI reports. Scientific Reports, 13(1), 20159. https://doi.org/10.1038/s41598-023-47500-2 DOI: https://doi.org/10.1038/s41598-023-47500-2

Tsao, J. (2025). Trajectories of AI policy in higher education: Interpretations, discourses, and enactments of students and teachers. Computers and Education: Artificial Intelligence, 9, 100496. https://doi.org/10.1016/j.caeai.2025.100496 DOI: https://doi.org/10.1016/j.caeai.2025.100496

Tukey, J. W. (1949). Comparing individual means in the analysis of variance. Biometrics, 5(2), 99–114. https://doi.org/10.2307/3001913 DOI: https://doi.org/10.2307/3001913

Ülkü, A. (2023). Artificial intelligence-based large language models and integrity of exams and assignments in higher education: The case of tourism courses. Tourism & Management Studies, 19(4), 21–34. https://doi.org/10.18089/tms.2023.190402 DOI: https://doi.org/10.18089/tms.2023.190402

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. NIPS’17 Proceeding of the 31st International Conference on Neural Information Processing Systems, 6000-6010. https://papers.nips.cc/paper_files/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

Walters, W. H., & Wilder, E. I. (2023). Fabrication and errors in the bibliographic citations generated by ChatGPT. Scientific Reports, 13(1), 14045. https://doi.org/10.1038/s41598-023-41032-5 DOI: https://doi.org/10.1038/s41598-023-41032-5

Welch, B. L. (1951). On the comparison of several mean values: An alternative approach. Biometrika, 38(3/4), 330–336. https://doi.org/10.2307/2332579 DOI: https://doi.org/10.1093/biomet/38.3-4.330

Wolfe, C. R. (2024, August 22). LLMs evolve quickly. Their underlying architecture, not so much. https://stackoverflow.blog/2024/08/22/llms-evolve-quickly-their-underlying-architecture-not-so-much/

Xing, W., Zhu, T., Wang, J., & Liu, B. (2024). A survey on MLLMs in education: Application and future directions. Future Internet, 16(12), 467. https://doi.org/10.3390/fi16120467 DOI: https://doi.org/10.3390/fi16120467

Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., Lyu, Q., Hendryx, S., Kaplan, R., Lunati, M., & Yue, S. (2024). A careful examination of large language model performance on grade school arithmetic (arXiv:2405.00332). arXiv. https://doi.org/10.48550/arXiv.2405.00332 DOI: https://doi.org/10.52202/079017-1485

Additional Files

Published

2026-07-05

Issue

Vol. 25 (2026): American Journal of STEM Education

Section

STEM Education (regular)

License

Copyright (c) 2026 David Hunt, Mathieu Di Miceli

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright (c) [year] [author]
This work is licensed under a Creative Commons Attribution 4.0 International License.
This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. https://creativecommons.org/licenses/by/4.0

How to Cite

Hunt, D., & Di Miceli, M. (2026). Evaluating the performance of 3 large language models in higher education essay-like assessments in 2024 and 2026. American Journal of STEM Education, 25, 279-304. https://doi.org/10.32674/tfptev94