Anatoly Shalyto, a professor at ITMO University and Doctor of Technical Sciences, conducted an experiment with several language models and concluded that they cannot be trusted without verification. On July 23, 2026, he asked the free version of Perplexity a question about who his official opponents were for his doctoral dissertation. The model provided the names of two scientists who were actually opponents for his postgraduate student — 25 years after Shalyto's own defense.
The professor then turned to "Alisa," Yandex's model. It named different opponents, but they also turned out to be from his student's dissertation. On the second attempt, "Alisa" provided two names, only one of which was correct — the professor identified the second as his co-author, not an opponent.
Sber's model also made a mistake: it provided the same fictitious opponents, incorrectly stated the defense year (1998 instead of 1999), and named LETI as the leading organization, although Shalyto actually defended there, and the leading enterprise was the Institute of Control Sciences of the Russian Academy of Sciences.
"Without verification, the advice of these probabilistic and therefore irresponsible 'assistants' cannot be trusted, and if the information received from them must be thoroughly checked every time, then how can anything responsible be entrusted to them?
Shalyto noted that the paid ChatGPT did not answer the same question, but also did not make a mistake, accurately indicating the date and place of defense, and correctly noting that the opponents were listed on the second page of the abstract.
Комментарии