Custom-Trained Large Language Models as Open Educational Resources: An Exploratory Research of a Business Management Educational Chatbot in Croatia and Bosnia and Herzegovina

Abstract

This paper explores the contribution of custom-trained Large Language Models (LLMs) to developing Open Education Resources (OERs) in higher education. Our empirical analysis is based on the case of a custom LLM specialized for teaching business management in higher education. This custom LLM has been conceptualized as a virtual teaching companion, aimed to serve as an OER, and trained using the authors’ licensed educational materials. It has been designed without coding or specialized machine learning tools using the commercially available ChatGPT Plus tool and a third-party Artificial Intelligence (AI) chatbot delivery service. This new breed of AI tools has the potential for wide implementation, as they can be designed by faculty using only conventional LLM prompting techniques in plain English. This paper focuses on the opportunities for custom-trained LLMs to create Open Educational Resources (OERs) and democratize academic teaching and learning. Our approach to AI chatbot evaluation is based on a mixed-mode approach, combining a qualitative analysis of expert opinions with a subsequent (quantitative) student survey. We have collected and analyzed responses from four subject experts and 204 business students at the Faculty of Economics, Business and Tourism Split (Croatia) and Faculty of Economics Mostar (Bosnia and Herzegovina). We used thematic analysis in the qualitative segment of our research. In the quantitative segment of empirical research, we used statistical methods and the SPSS 25 software package to analyze student responses to the modified BUS-15 questionnaire. Research results show that students positively evaluate the business management learning chatbot and consider it useful and responsive. However, interviewed experts raised concerns about the adequacy of chatbot answers to complex queries. They suggested that the custom-trained LLM lags behind the generic LLMs (such as ChatGPT, Gemini, and others). These findings suggest that custom LLMs might be useful tools for developing OERs in higher education. However, their training data, conversational capabilities, technical execution, and response speed must be monitored and improved. Since this research presents a novelty in the extant literature on AI in education, it requires further research on custom GPTs in education, including their use in multiple academic disciplines and contexts.