ChatGPT: Brazilian colleges prepare to adapt test format; You understand

Launched at the end of November last year, ChatGPT is an artificial intelligence that generates text (GPT-3) in a user-friendly interface (chat, as the name implies), developed by OpenAI, has quickly become popular and is already causing headaches for universities in the United States. There, some Teachers are restructuring their courses, making changes that include more oral exams, group work, and handwritten assignments rather than written assignments..

He arrived in Brazil at the end of the semester, but now professors from institutions such as the Getulio Vargas Foundation (FGV), the Graduate School of Advertising and Marketing (ESPM), Insper, Mackenzie Presbyterian University and the State University of Rio de Janeiro (Uerj) are already studying how to adapt teaching plans and also how to assess students , particularly in “content-oriented” disciplines. However, they understand that it is impossible to predict everything that can happen with the introduction of new technology into the educational scenario, it will be necessary to “change the wheel, as the car moves,” as they say.

Their assessment is that “you can’t swim against the tide” and that it’s better to have ChatGPT as an ally rather than an enemy. This is taking into account that new technology should not only cause negative effects but, on the contrary, if it is used well, it can improve the teaching and learning process.

“We already have a roundtable scheduled, which will take place on February 8, about ChatGPT and its possibilities. Some teachers are already including ChatGPT in their lesson plans, and everything is being done experimentally, because we have no way of knowing its impact on teaching and learning experiences,” he says. Tiago Tavares, PhD in Electrical Engineering with a focus on Machine Learning and Professor at Insper. to EstadaoThe State University of São Paulo (Unesp), through its consultant, stated that this artificial intelligence was already the subject of an “institutional meeting with the aim of addressing the reach of this technology”.

To use ChatGPT, for now, you just need to create an account, login to your profile, and ask questions. The web interface is still entirely in English, but the little bot understands and gives answers in very reasonable Portuguese, with the exception of some spelling slips. They are used to using search engines and intelligent virtual assistants. From this description, it probably doesn’t seem surprising, after all, these other tools do it too.

The difference, says McKinsey School of Computing and Informatics professor Rodrigo Cardoso Silva, is the fact that he manages to fit his answer into the context of what he’s being asked. “Within the form of text it fetches, it will try to contextualize the user in a more granular way.” Trained on billions of parameters, Carlos Rafael, a professor of information systems at ESPM, added, “He is able to understand textual properties and extract subjects on an infinite number of occasions.”

With this response, the AI ​​proved to be a good student and “Passed” the Wharton University MBA final exam, the Bar Examination (MBE) and also the United States Medical Licensing Examination (USMLE). And in the National Secondary School Examination (ENEM), he was not embarrassed either.

Tavares copied and pasted the questions into a ChatGPT box, typed the answers and then placed them on a platform that estimates the score using Item Response Theory (IRT). “I found the system’s performance impressive on the humanities and language tests. I think the good performance occurred because more often than not the answer to the question was in the statement itself, or it’s a paraphrase of the same statement,” he says.

“I don’t think this is a flaw in the enemy, because it is designed to evaluate people. When we talk about human thinking, paraphrasing means understanding what has been read, building a mental model of it, and then using other words to describe the same thing. In ChatGPT’s mathematical operation, it is found The answer is by checking which words are most likely to be part of the same group of words as in speech, which is quite different,” he adds.

“On a math test, it becomes very clear: ChatGPT is incapable of doing algebraic syllables, even the simplest ones. The equivalent to this, on a humanities test, is that ChatGPT tends to make mistakes when the answer depends on understanding a broader context than the context explicitly described in speech.” , he says.

Obviously, the man-made technology has limitations, which ChatGPT isn’t shy about admitting (there are a number of caveats that “the system may occasionally generate incorrect or misleading information and produce offensive or biased content”). “He is not online, so he is not able to talk about the issues that are happening today, and his knowledge base has a strict deadline, which is 2021,” says Carlos Rafael.

It continues after the announcement

In New York, some schools have been completely categorical and ChatGPT has been banned from use. However, the professors Estadão interviewed don’t think this is the way to tackle the problem and prefer a less hard-line approach. It is the road of no return. Technology is developing. The only constant is change. I talk a lot about the Darwinian principle: only the strongest survive. So we have to think about this course of action,” says Flávio Marques Azevedo, coordinator of the Information Systems course at ESPM.

Ignoring new technology can also mean missing out on an opportunity to improve teaching and learning. “I recently read an article where a professor at a university in Ecuador did an experiment, not using ChatGPT, but the university created an app that was a bot, like Siri or Alexa, that was used as a tutor for a student. They divided the class into two groups. Half of the students didn’t use the bot and the other half Use it Marcos Vaco, director of communications and marketing at FGV, said the combination I used during the program had 20% higher performance.

In this sense, the researchers interviewed Estadao List a number of benefits of technology, for both students and teachers, such as: marking tests; work as a “teacher” or “assistant teacher”; help with boring tasks (converting an Excel spreadsheet to PDF, for example); simplify search and write processes; Refine generating bibliographies and helping to think of questions for tests. They also point out risks, such as a lack of accuracy, not quite right answers, and the presence of payout biases. After all, AI is trained by humans (the platform itself assumes this, although it strives for it). reflect “prejudice” and “inequality”, with small social groups underrepresented from their sources).

On accuracy, ESPM’s Carlos Rafael commented that in the tests he conducted, he noticed that for simple questions he had a success rate of 100%, but when the complexity is medium, the story is already different. “It does something very, very dangerous,” he says, “and gives an answer that’s not wrong, but it’s not 100 percent correct.”

“I ran a test where I asked ChatGPT to determine whether a geometry intersects another. There are three possible cases: the two geometries that don’t have an intersection; I got the code right in 2/3 of the cases. It speaks when it is not and when it collides, but not when it is inside. That is, if the student takes this code and performs a superficial test, without effort, at some point he will make a mistake “, Add.

“The biggest danger is that people will completely trust what he offers. There must be confirmation,” Facco sums up.

The danger that is perhaps the cause of the greatest panic in the United States is related to academic ethics and plagiarism on the part of students. Notably, this is a problem that predates the technology in question and relates to behavior and the way students see their role in education.

However, the catch is that it is much more difficult to detect the version when using ChatGPT. “As far as I know, spoofing programs don’t catch ChatGTP,” adds Carlos Afonso Souza, professor at Uerj and director of the Institute of Technology and Society (ITS Rio), who tested the texts prepared by the AI ​​on the platform it usually uses.

However, the teachers interviewed before Estadao He doesn’t seem afraid of introducing technology. They consider that more “content-oriented” and “just explain” disciplines will suffer the most, but point out that it is indeed possible to circumvent plagiarism with ChatGPT. In a more practical way, they point to the need for professors to become aware of the technology and its potential, and to think about assessments that require more than content, with an emphasis on problem-solving and analytical and critical comparison. On the other hand, they also assume the possibility of resuming oral assessment activities (directly in class or through video recording), in addition to essays.

Souza embodies an activity he usually imparts to students in law classes: the analysis of jurisprudence. “I ask them to find four judicial decisions on a topic and the job is to summarize the case and at the end compare the decisions,” he says. The professor decided to test how ChatGPT interacts with this task: he gave four decisions and asked to compare them.

“He does not make the comparison as if he were a graduate student, but what he does is interesting, because he always gives an introductory text on the subject with certain concepts, with certain definitions that are generally correct, which is a failure on the analytical part,” he says.

I joked on Twitter that Messages at Home has been turned off, but then it won’t necessarily be off. Those who would simply ask you to elaborate on a concept, to sum up a court decision, that’s over, Chat does. So you have to go further, you need to foster a critical, comparative, and analytical viewpoint in the student, preferably one that is not found on the Internet,” he summarizes.

Azevedo, of ESPM, says the directive he relays to the professors who handle the programming part is that they are “eager to change this job application bias.” For example, in computing, we can ask a student to develop a Fibonacci sequence. If you write there “notify a Fibonacci sequence”, ChatGPT will return you the final code. So this will not be an answer for us. Now, applying the sequence can be an answer for students to do them independently.”

Another discussion that arose after the popularization of ChatGPT was about the possibility of replacing it with the tutor. But Carlos Rafael has no such fear. “Both ChatGPT and every other authoring tool, it relies on you knowing what you want. But the student doesn’t always know what they want,” he says. “The tutor itself will not be replaced by ChatGPT. It can even be replaced with a pre-recorded video, pre-recorded 3D image for lessons.”

The report asked if ChatGPT thought it would be able to replace the teacher and its answer was negative. “ChatGPT is only a technological tool and cannot replace a teacher’s human interaction, empathy, and ability to teach. It can be a complementary aid, but it cannot replace the importance of human education.”