While ChatGPT is criticized for its misuse by students, OpenAI has developed a tool capable of spotting texts generated by artificial intelligence. Problem, this one is still very perfectible. It fails to detect every time the “paw” of the AI in a text.
Predictably, students flocked to ChatGPT to do their homework. What give sacred migraines to education professionals. Some institutions have decided to legislate quickly, such as Sciences Po which has simply banned the use of ChatGPT by its students under penalty of exclusion. But how will teachers be able to detect if a text has been generated by AI?
While several tools are being developed, like DetectGPT, OpenAI has just drawn its own. On his blog, the father of ChatGPT reveals that he designed a new tool trained to distinguish between text written by a human and text generated by a variety of AIs (not just ChatGPT). This could prove useful in spotting dishonest use of a chatbot. But OpenAI still has work to do before its detection tool is ready.
How to spot a text generated by ChatGPT?
The researchers admit that it “is not entirely reliable”. In fact, it correctly identified only 26% of the English texts written by the AI. And he was also wrong on 9% of the texts written by humans, wrongly assuring that they had been written by a chatbot. “The reliability of our classifier generally improves as the length of the input text increases. Compared to our previously released classifier, this new tool is significantly more reliable on text generated by newer AI systems”write the researchers.
In addition to its concerns of erroneous identification, the OpenAI tool lacks reliability on texts of less than 1000 characters. The researchers thus believe that its use must be confined for the moment to texts written in English. And for good reason, it works “significantly worse” in other languages; it is also not reliable for code verification.
Ultimately, “it should not be used as a primary decision-making tool, but rather as a complement to other methods for determining the source of a text fragment”, believes OpenAI. On the way, the tool, accessible here, should obviously improve.
