Analyzing GPT-5 performance in general chemistry
Best practices for assignment design
DOI:
https://doi.org/10.32674/ycyhab69Keywords:
artificial intelligence, ChatGPT, chemistry, general chemistry, higher education, large language modelAbstract
The ability of Large Language Models (LLMs) to outperform humans on many intellectual tasks has created a difficult landscape for assessing student work. One strategy is designing assignments to be specifically difficult for AI. However, LLMs have been shown to perform well on general chemistry problems across topics, and new models continue to make AI-proofing assignments more difficult. In this work, we evaluate the recently released GPT-5 Thinking model across a set of 1736 questions from the Openstax general chemistry textbook. While GPT-5 achieves near-perfect accuracy overall, categorizing questions by the types of cognitive skills required elucidates specific weaknesses. Notably, GPT-5 tends to make major errors when asked to create figures. These findings inform best practices for AI-resistant assignment design.
References
Alampara, N., Schilling-Wilhelmi, M., Ríos-García, M., Mandal, I., Khetarpal, P., Grover, H. S., Krishnan, N. A., & Jablonka, K. M. (2025). Probing the limitations of multimodal language models for chemistry and materials research. Nature Computational Science, 5, 952–961. https://doi.org/10.1038/s43588-025-00836-3 DOI: https://doi.org/10.1038/s43588-025-00836-3
Alasadi, E. A., & Baiz, C. R. (2024). Multimodal generative artificial intelligence tackles visual problems in chemistry. Journal of Chemical Education, 101(7), 2716–2729. https://doi.org/10.1021/acs.jchemed.4c00138 DOI: https://doi.org/10.1021/acs.jchemed.4c00138
Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32, 347–364. https://doi.org/10.1007/BF00138871 DOI: https://doi.org/10.1007/BF00138871
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain (pp. 1103–1133). Longman.
Boiko, D. A., MacKnight, R., Kline, B., & Gomes, G. (2023). Autonomous chemical research with large language models. Nature, 624(7992), 570–578. https://doi.org/10.1038/s41586-023-06792-0 DOI: https://doi.org/10.1038/s41586-023-06792-0
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024). Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6, 525–535. https://doi.org/10.1038/s42256-024-00832-8 DOI: https://doi.org/10.1038/s42256-024-00832-8
Brown, L. D., Cai, T. T., & DasGupta, A. (2001). Interval estimation for a binomial proportion. Statistical Science, 16(2), 101–133. https://doi.org/10.1214/ss/1009213286 DOI: https://doi.org/10.1214/ss/1009213286
Clark, T. M. (2023). Investigating the use of an artificial intelligence chatbot with general chemistry exam questions. Journal of Chemical Education, 100(5), 1905–1916. https://doi.org/10.1021/acs.jchemed.3c00027 DOI: https://doi.org/10.1021/acs.jchemed.3c00027
Clark, T. M., Anderson, E., Dickson-Karn, N. M., Soltanirad, C., & Tafini, N. (2023). Comparing the performance of college chemistry students with ChatGPT for calculations involving acids and bases. Journal of Chemical Education, 100(10), 3934–3944. https://doi.org/10.1021/acs.jchemed.3c00500 DOI: https://doi.org/10.1021/acs.jchemed.3c00500
Daher, W., Diab, H., & Rayan, A. (2023). Artificial intelligence generative tools and conceptual knowledge in problem solving in chemistry. Information, 14(7), 409. https://doi.org/10.3390/info14070409 DOI: https://doi.org/10.3390/info14070409
Emenike, M. E., & Emenike, B. U. (2023). Was this title generated by ChatGPT? considerations for artificial intelligence text-generation software programs for chemists and chemistry educators. Journal of Chemical Education, 100(4), 1413–1418. https://doi.org/10.1021/acs.jchemed.3c00063 DOI: https://doi.org/10.1021/acs.jchemed.3c00063
Fergus, S., Botha, M., & Ostovar, M. (2023). Evaluating academic answers generated using ChatGPT. Journal of Chemical Education, 100(4), 1672–1675. https://doi.org/10.1021/acs.jchemed.3c00087 DOI: https://doi.org/10.1021/acs.jchemed.3c00087
Genovese, D. (2025, August 7). OpenAI GPT-5 is here: Company unveils ’PhD-level experts in your pocket’. Fox Business. https://www.foxbusiness.com/technology/openai-gpt-5-here-company-unveils-phd-level-experts-your-pocket
GraphPad Software. (2025). GraphPad Prism (Version 10.5.0) [Computer software].
Guo, T., Guo, K., Nan, B., Liang, Z., Guo, Z., Chawla, N. V., Wiest, O., & Zhang, X. (2023). What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks. NeurIPS '23: Proceedings of the 37th International Conference on Neural Information Processing Systems, 59662 – 59688. https://doi.org/10.52202/075280-2607 DOI: https://doi.org/10.52202/075280-2607
Guo, Y., & Lee, D. (2023). Leveraging ChatGPT for enhancing critical thinking skills. Journal of Chemical Education, 100(12), 4876–4883. https://doi.org/10.1021/acs.jchemed.3c00505 DOI: https://doi.org/10.1021/acs.jchemed.3c00505
Hallal, K., Hamdan, R., & Tlais, S. (2023). Exploring the potential of AI-chatbots in organic chemistry: An assessment of ChatGPT and Bard. Computers and Education: Artificial Intelligence, 5, 100170. https://doi.org/10.1016/j.caeai.2023.100170 DOI: https://doi.org/10.1016/j.caeai.2023.100170
Hasrod, T., Nuapia, Y. B., & Tutu, H. (2024). ChatGPT helped me build a chemistry app, and here’s how you can make one also. Journal of Chemical Education, 101(2), 653-660. https://doi.org/10.1021/acs.jchemed.3c01170 DOI: https://doi.org/10.1021/acs.jchemed.3c01170
Humphry, T., & Fuller, A. L. (2023). Potential ChatGPT use in undergraduate chemistry laboratories. Journal of Chemical Education, 100(4), 1434–1436. https://doi.org/10.1021/acs.jchemed.3c00006 DOI: https://doi.org/10.1021/acs.jchemed.3c00006
Jones, R. M., Bryan, E. M., & Preston, C. (2024). Exploring the concept of valence and the nature of science via generative artificial intelligence and general chemistry textbooks. Journal of Chemical Education, 101(8), 3276–3283. https://doi.org/10.1021/acs.jchemed.4c00271 DOI: https://doi.org/10.1021/acs.jchemed.4c00271
Kim, S., Jung, Y., & Schrier, J. (2024). Large language models for inorganic synthesis predictions. Journal of the American Chemical Society, 146(29), 19654–19659. https://doi.org/10.1021/jacs.4c05840 DOI: https://doi.org/10.1021/jacs.4c05840
Kılınç, S. (2023). Embracing the future of distance science education: Opportunities and challenges of ChatGPT integration. Asian Journal of Distance Education, 18(1), 205–237. https://doi.org/10.5281/zenodo.7857396
Lear, B. J. (2024). Using ChatGPT-4 to teach the design of data visualizations. Journal of Chemical Education, 101(7), 2749–2756. https://doi.org/10.1021/acs.jchemed.4c00228 DOI: https://doi.org/10.1021/acs.jchemed.4c00228
Leon, A. J., & Vidhani, D. (2023). ChatGPT needs a chemistry tutor too. Journal of Chemical Education, 100(10), 3859–3865. https://doi.org/10.1021/acs.jchemed.3c00288 DOI: https://doi.org/10.1021/acs.jchemed.3c00288
Lopez-Gazpio, I. (2025). Integrating large language models into accessible and inclusive education: Access democratization and individualized learning enhancement supported by generative artificial intelligence. Information, 16(6), 473. https://doi.org/10.3390/info16060473 DOI: https://doi.org/10.3390/info16060473
Lopez-Perez, K., Benjamin, S. A., Veige, M. K. (2026). Outsmarting ChatGPT in Chemistry: Using Student-Formulated Questions for Critical AI Evaluation. Journal of Chemical Education. https://doi.org/10.1021/acs.jchemed.5c01611 DOI: https://doi.org/10.1021/acs.jchemed.5c01611
McLeod, G. M., Hanks, B., Ashby, H., Nolder, J., Sorenson, B., & Hanson, K. (2026). Generative AI vs Student Performance on Multiple-Choice, General Chemistry Exams: Insights into Question Design from Regression Analysis. Journal of Chemical Education, 103(2), 872–878. https://doi.org/10.1021/acs.jchemed.5c00979 DOI: https://doi.org/10.1021/acs.jchemed.5c00979
Mirza, A., Alampara, N., Kunchapu, S., Ríos-García, M., Emoekabu, B., Krishnan, A., Gupta, T., Schilling-Wilhelmi, M., Okereke, M., Aneesh, A., Asgari, M., Eberhardt, J., Elahi, A. M., Elbeheiry, H. M., Gil, M. V., Glaubitz, C., Greiner, M., Holick, C. T., Hoffmann, T., Ibrahim, A., … Jablonka, K. M. (2025). A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry. 17, 1027–1034. https://doi.org/10.1038/s41557-025-01815-x DOI: https://doi.org/10.1038/s41557-025-01815-x
Pereira, N. C. P., Bedouch, M. B., & Borges, E. M. (2026). Leveraging Microsoft Copilot (GPT-5) for Calculations and Interactive Data Visualization. Digital, 6(1), 16. https://doi.org/10.3390/digital6010016 DOI: https://doi.org/10.3390/digital6010016
Ruff, E. F., Engen, M. A., Franz, J. L., Mauser, J. F., West, J. K., & Zemke, J. M. O. (2024). ChatGPT writing assistance and evaluation assignments across the chemistry curriculum. Journal of Chemical Education. 101(6), 2483–2492. https://doi.org/10.1021/acs.jchemed.4c00248 DOI: https://doi.org/10.1021/acs.jchemed.4c00248
Ruff, E. F., Franz, J. L., & West, J. K. (2024). Using ChatGPT for method development and green chemistry education in upper-level laboratory courses. Journal of Chemical Education, 101(8), 3224–3232. https://doi.org/10.1021/acs.jchemed.4c00193 DOI: https://doi.org/10.1021/acs.jchemed.4c00193
Runcie N. T., Deane C. M., & Imrie F. (2026). Assessing the Chemical Intelligence of Large Language Models. Journal of Chemical Information and Modeling, 66(1), 216–227. https://doi.org/10.1021/acs.jcim.5c02145 DOI: https://doi.org/10.1021/acs.jcim.5c02145
Schrier, J. (2024). Comment on “comparing the performance of college chemistry students with ChatGPT for calculations involving acids and bases”. Journal of Chemical Education, 101(5), 1782–1784. https://doi.org/10.1021/acs.jchemed.4c00058 DOI: https://doi.org/10.1021/acs.jchemed.4c00058
Sharlin, S., Agbere, F., Ishimwe, K., Osifová, Z., Socha, O., Dračínský, M., & Josephson, T. R. (2026). NMR-challenge for LLMs: Evaluating chemical reasoning in humans and AI. Machine Learning and Deep Learning, 66(11), 6299–6312. https://doi.org/10.1021/acs.jcim.6c00481 DOI: https://doi.org/10.1021/acs.jcim.6c00481
Tyson, J. (2023). Shortcomings of ChatGPT. Journal of Chemical Education, 100(8), 3098–3101. https://doi.org/10.1021/acs.jchemed.3c00361 DOI: https://doi.org/10.1021/acs.jchemed.3c00361
Uçar, S. Ş., Lopez-Gazpio, I., & Lopez-Gazpio, J. (2025). Evaluating and challenging the reasoning capabilities of generative artificial intelligence for technology-assisted chemistry education. Education and Information Technologies, 30, 11463–11482. https://doi.org/10.1007/s10639-024-13295-6 DOI: https://doi.org/10.1007/s10639-024-13295-6
Watts, F. M., Dood, A. J., Shultz, G. V., & Rodriguez, J.-M. G. (2023). Comparing student and generative artificial intelligence chatbot responses to organic chemistry writing-to-learn assignments. Journal of Chemical Education, 100(10), 3806–3817. https://doi.org/10.1021/acs.jchemed.3c00664 DOI: https://doi.org/10.1021/acs.jchemed.3c00664
West, J. K., Franz, J. L., Hein, S. M., Leverentz-Culp, H. R., Mauser, J. F., Ruff, E. F., & Zemke, J. M. (2023). An analysis of AI-generated laboratory reports across the chemistry curriculum and student perceptions of ChatGPT. Journal of Chemical Education, 100(11), 4351–4359. https://doi.org/10.1021/acs.jchemed.3c00581 DOI: https://doi.org/10.1021/acs.jchemed.3c00581
Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212. https://doi.org/10.2307/2276774 DOI: https://doi.org/10.1080/01621459.1927.10502953
Yik, B. J., & Dood, A. J. (2024). ChatGPT convincingly explains organic chemistry reaction mechanisms slightly inaccurately with high levels of explanation sophistication. Journal of Chemical Education, 101(5), 1836– 1846. https://doi.org/10.1021/acs.jchemed.4c00235 DOI: https://doi.org/10.1021/acs.jchemed.4c00235
Yuriev, E., Wink, D. J., & Holme, T. A. (2024). The dawn of generative artificial intelligence in chemistry education. Journal of Chemical Education, 101(8), 2957−2959. https://doi.org/10.1021/acs.jchemed.4c00836 DOI: https://doi.org/10.1021/acs.jchemed.4c00836
Zhang, D., Liu, W., Tan, Q., Chen, J., Yan, H., Yan, Y., Li, J., Huang, W., Yue, X., Ouyang, W., Zhou, D., Zhang, S., Mao, S., Zhong, H. S., & Li, Y. (2024). ChemLLM: A chemical large language model. arXiv. https://doi.org/10.48550/arXiv.2402.06852
Additional Files
Published
Issue
Section
License
Copyright (c) 2026 Emilie V. Ernst, Jennifer R. DeRosa, Brian G. Ernst
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) [year] [author]
This work is licensed under a Creative Commons Attribution 4.0 International License.
This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. https://creativecommons.org/licenses/by/4.0
Call for Special Issue Proposals 
