Seoul: Police recently filed for an arrest warrant against a YouTuber on charges of spreading false information after broadcasting AI-generated voice recordings and manipulated messenger conversations. Kim Se-ui, head of the YouTube channel Garo Sero Institute, released an audio recording purported to be of the late actor Kim Sae-ron in May last year. He claimed it had been provided by her bereaved family and that it was evidence she had been in a relationship with actor Kim Soo-hyun since her middle school years. However, police concluded that the recording was fake and generated using artificial intelligence.
According to Yonhap News Agency, the arrest warrant states that the screenshots of KakaoTalk conversations presented by the YouTube channel as messages exchanged between Kim Sae-ron and Kim Soo-hyun in 2016, when she was a middle school student, had likewise been manipulated: Although the other party's name originally appeared as "Unknown" in the messages, it had allegedly been changed to "Kim Soo-hyun."
Kim Soo-hyun suffered extensive damage after Kim Se-ui raised these allegations. It is reported that his financial losses amount to tens of billions of won, as multiple advertising contracts and drama productions were canceled, leading to penalty claims and lawsuits for damages. He was also forced to endure public moral criticism over allegations that he had dated a minor, and his acting career was, for all practical purposes, over.
Although a final court ruling has yet to be made, if the AI fabrication is confirmed through the trial, severe punishment will be necessary to serve as a warning about evidence manipulation using advanced technology. Because he is an actor whose public image is of critical importance, the harm he suffered was likely particularly severe. However, evidence manipulation using AI is a threat that could affect any member of the public.
In this case, police are said to have been able to determine whether the recordings were manipulated by analyzing the voice data in the audio files. This may have been possible because the material had been broadcast since March of last year. Had the evidence been manipulated using today's AI technology, it might not have been so easy to detect.
Evidence fabrication using AI had already appeared even before this case, creating confusion in both investigations and trials. Last year, a man in his 20s who committed investment fraud by forging documents with AI image-generation tools managed to avoid detention after submitting a fake bank balance certificate to a judge, but was later taken into custody once the certificate was found to be falsified.
AI-generated fake content about celebrities is often seized upon by "cyber wreckers" who spread sensational rumors to boost their subscriber counts, resulting in damage that rapidly snowballs for the celebrities concerned. As AI technology advances, people become more likely to accept fake content as real, and once someone is stigmatized, it is difficult to remove that label - no matter how strongly the victim asserts their innocence.
Furthermore, AI is making rapid progress not only in text and voice generation, but also in image and video production and processing. It is getting harder and harder to verify the authenticity of content, be it text, voice, images, or video, as AI manipulation becomes more sophisticated. AI is driving a paradigm shift across almost every sector of society. The criminal justice system is no exception.
Investigative authorities must ensure experts are trained to detect AI-based manipulation of evidence and the relevant supporting infrastructure. In addition, changes in trial procedures are needed to strengthen the verification of evidence reliability. Above all, AI-based manipulation of evidence should be regarded as a grave obstruction of justice, and those who manipulate or maliciously disseminate it must be held strictly accountable.