Rethinking what LLMs really do for education: The case of the artificial-intelligence chatbot, GPT-4, and the Khan Academy
Whether LLMs’ promise for education will ultimately outweigh the risks is still not clear. Lynch accepts that they are powerful tools, but also seeks to keep their shortcomings in focus. “It isn’t like overnight we’ve learnt to fly,” he says.
Many organizations use artificial intelligence to help students. The approach of the teaching assistant Khanmigo and the artificial intelligence tutor is the most used of all the LLM-based education tools. The tool is the result of a partnership between OpenAI and education non-profit organization Khan Academy in Mountain View, California. Using GPT-4, Khanmigo offers students tips as they work through an exercise, saving teachers time.
In a preprints in February, researchers described how in a benchmark set of simple mathematical problems, in which students aged 12–17 would typically answer, ChatGPT answered about half of the questions correctly. It was likely to fail if the problems were more complex and required four or more additions or subtractions in the calculation.
Teachers were spooked when ChatGPT was launched a year ago. The artificial-intelligence (AI) chatbot can write lucid, apparently well-researched essays in response to assignment questions, forcing educators around the world to rethink their evaluation methods. The pen-and-paper exams were brought back in a few countries. And some schools are ‘flipping’ the classroom model: students do their assignments at school, after learning about a subject at home.
Tawil, who has worked in education at UNESCO for more than two decades, says that understanding AI’s limitations is crucial. At the same time, LLMs are now so bound up in human endeavours that he says it is essential to rethink how to teach and assess learning. What makes us human is being redefined.
He likens the attention that they’re attracting to that previously lavished on massively online open courses and educational uses of the 3D virtual worlds known as the metaverse. Neither have the transformative power that some once predicted, but both have their uses. This is going to be the same. It’s not bad. It’s not perfect. It is not everything. He says it is a new thing.
Kristen DiCerbo, the academy’s chief learning officer, calls this process a “productive struggle”. But she acknowledges that Khanmigo is still in a pilot phase and that there is a fine line between a question that aids learning and one that’s so difficult that it makes students give up. “The trick is to figure out where that line is,” she says.
LLMs in Education: How Chatgt has entered the Classroom: how GPT-4, Google’s Bard and Merlyn Mind can help students
RAG is also being used by ASU, which is one of the most progressive universities for LLM adoption, says Claire Zau, vice-president of GSV Ventures, an investor in educational-technology companies in New York City. After an initial narrow release for testing, the faculty were able to experiment with LLMs in education through a web interface. This includes access to six LLMs, including GPT-3.5, GPT-4 and Google’s Bard, as well as RAG capabilities.
Another approach to creating an AI learning partner integrates the LLM with external, focused corpuses of knowledge — such as a textbook or a set of scientific papers — that have been rigorously verified. The goal of this method is to make it easier to verify the billions of sources that give an LLM its power.
But unlike ChatGPT, when the LLM answers a query, it does not rely just on what it has learnt in its training. Satya Nita, the company’s chief executive, says that it also refers to a specific corpus of information, which helps minimize hallucinations and other errors. Merlyn Mind fine-locks its graduates to confession if they don’t have a high-quality response and work on producing a better answer.
LLMs are being offered by other tutoring companies as assistants. The education technology firm Chegg in Santa Clara, California, launched an assistant based on GPT-4 in April. And TAL Education Group, a Chinese tutoring company based in Beijing, has created an LLM called MathGPT that it claims is more accurate than GPT-4 at answering maths-specific questions. By showing how to solve problems, MathGPM aims to help students.
Source: ChatGPT has entered the classroom: how LLMs could transform education
How Does Khanmigo Change Education? An Empirical Study of the Appearance, Performance, and Response of the AI Assistant to GPT-4
Lynch stresses the importance of carefully checking for accuracy and tone of the chatbot used in education, and that it does not insult or make students feel lost. “Emotion is key to learning. Lynch feels that you can help someone in a bad way and destroy their interest in learning.
It’s not clear whether Khanmigo can really change education. LLMs are trained to include only the next most likely word in a sentence, not to check facts. They therefore sometimes get things wrong. The right answers for guidance have been included in the prompt that Khanmigo sends to GPT-4. It still makes mistakes, however, and Khan Academy asks users to let the organization know when it does.
Khanmigo was first introduced in March, and more than 28,000 US teachers and 11–18-year-old students are piloting the AI assistant this school year, according to Khan Academy. Users include more than 30 school districts and private subscribers. Individuals pay US$99 a year to cover the computing costs of LLMs, and school districts pay $60 a year per student for access. To protect student privacy, OpenAI has agreed not to use Khanmigo data for training.
Khanmigo works differently from ChatGPT. It appears as a pop-up chatbot on a student’s computer screen. The problem that the students are working on can be discussed. The tool will add a prompt before it sends a question to GPT-4, which will ask lots of questions and not give away answers.
The scores of PyrEval help students to reflect on the work, if the computer ignores a theme the student thought they had included, or if the idea needs to be explained more clearly. The team is now comparing the results of the task done by other LLMs.
PyrEval has been helped by the help of a psychologist at the University of Wisconsin–Madison to score more than 2000 physics essays for middle school students a year. The PyrEval enables teachers to quickly check if assignments include key themes and provide feedback during the class even if they don’t get a traditional grade.
Companies are marketing commercial assistants, such as MagicSchool and Eduaide, that are based on OpenAI’s LLM technology and help schoolteachers to plan lesson activities and assess students’ work. The PyrEval4 created by Rebecca Passonneau and her team at Penn State University was a tool to read essays and extract the key ideas.
A computer scientist at NC State University who specializes in educational systems asks, “Are there positive uses?” “Absolutely. Is there any risks? There are a great number of risks and concerns. But I think there are ways to mitigate those.”
Students working with LLMs would be put off if they were aware that everything they type into them was being stored by Openai and could be used to train models.
Much of the attention has been negative about the use of the new technology in education. Training data containing billions of examples show how words and phrases relate to each other. They respond to user questions with responses that include an answer to an assignment question and whole essays.
The rise of ChatGTrust will make it easier for students to cheat on assignments according to some educators. Beghetto, who is based in Arizona, and others are exploring the potential of large language models to enhance education.
Ronald Beghetto asked a group of graduate students and teaching professionals to talk about their work in an unusual way. They conversed with a collection of creativity focused chatbot that Beghetto had designed and a platform that will be hosted on by his institute at Arizona State University.