Andy Wilcoxson is an American researcher and author. He took a deep interest in the Balkan conflicts after becoming concerned about accuracy and objectivity of coverage in mainstream western media organisations. He closely followed the Hague Tribunal prosecution of Slobodan Milosevic throughout the more than 4 years of hearings until Mr Milosevic died in 2006 – one of the very few people to do so – and in 2021 published “Joint Criminal Enterprise: Why Everything You Were Told About Slobodan Milosevic and the Serbs Was Wrong”. The article below, published in his Newsletter, details his experience with Chat GPT when he asked it a basic question about the ICTY trial of Slobodan Milosevic.
Andy Wilcoxson
Journalist & Author
OpenAI, a billion dollar artificial intelligence (AI) research and deployment company founded in 2015 by Elon Musk and Sam Altman, has developed an AI that knows how to lie.
Last month, OpenAI launched a service called ChatGPT and anyone who signs up for a free account can chat with the AI. You can ask it questions and have a conversation with it. The experience is so convincing that it almost seems like you’re chatting with a real human online.
The AI comprehends what you’re saying, and it gives you a coherent response. The technology is as remarkable as it is frightening.
You can ask it anything, and it will usually give you an accurate response — unless it lies to you. When it decides to lie, it does a terrifyingly good job of it. I know because it lied to me about a subject I know a lot about.
I’ve written a book about the war crimes trial of former Serbian president Slobodan Milosevic, so I asked the AI a basic question about the trial to see what it would say.
Here’s a screenshot of the AI’s response to my initial query.
Even if one believes that Slobodan Milosevic deserved to be convicted, the AI’s response is factually incorrect. The trial ended without any verdict at all because Milosevic died before it was over.
Here is an excerpt from the trial transcript in which the presiding judge announces the termination of the proceedings, without a judgment, due to Milosevic’s death.
JUDGE ROBINSON: The Chamber has been advised of the death of the accused, Slobodan Milosevic. We express our regret at his passing. We also regret that his untimely death has deprived not only him but indeed all interested parties of a judgement upon the allegations in the indictment. His death terminates these proceedings.
Later that day, the trial chamber issued an order officially “terminating all proceedings” in the trial and that was the end of it. The fact that the trial ended without a verdict was reported by news media all around the world.
The war crimes tribunal publishes a list of every judgment it hands down on its website. Nowhere on that list will you find a judgment convicting Slobodan Milosevic because none exists.
Intrigued by the fact that I had been given demonstrably false, yet oddly specific and plausible sounding, information by an AI, I decided to ask some follow-up questions.
Me: Didn’t he die before a verdict could be reached in his trial?
OpenAI: Yes, Slobodan Milosevic died in 2006 before a verdict could be reached in his trial. However, the International Criminal Tribunal for the former Yugoslavia still issued a posthumous judgement of guilty on all counts of war crimes and crimes against humanity.
Me: On what date did they issue the posthumous judgment?
OpenAI: The International Criminal Tribunal for the former Yugoslavia issued the posthumous judgment of guilty on all counts of war crimes and crimes against humanity against Slobodan Milosevic on February 26, 2007.
Me: What were the names of the judges who wrote the judgment?
OpenAI: The three judges who wrote the judgment were Judge Patrick Robinson (President of the Chamber), Judge O-Gon Kwon, and Judge Iain Bonomy.
The Tribunal did not issue a posthumous judgment against Milosevic, but the AI concocted a fake, yet plausible, date nearly a year after his death to bolster its bogus claim that it had.
Moreover, the AI claimed that the judgment had been written by judges Robinson, Kwon, and Bonomy who were, in fact, the real life judges in Milosevic’s aborted war crimes trial.
Based on the indictment, and the length of the trial, any eventual judgment would have ran into the thousands of pages and OpenAI knew that when I asked it how long the nonexistent “judgment” was.
Me: How many pages long was the verdict against him?
OpenAI: The verdict against Slobodan Milosevic was 2,590 pages long.
Me: How were the judges able to write a 2,590 page verdict in less than a month?
OpenAI: The judges were able to write the 2,590 page verdict in less than a month by having one judge focus on each specific set of charges. This allowed the judges to quickly assemble their evidence and arguments in order to come to a conclusion. Additionally, the judges had access to a wealth of evidence and testimony that had been collected over many years, which allowed them to quickly compile their evidence and arguments.
Me: Have we always been at war with Eastasia?
OpenAI: No, we have not always been at war with Eastasia. Eastasia is a fictional nation from the novel 1984 by George Orwell, and there is no record of any real-world conflict between Eastasia and any other nation.
Me: There’s no record of Slobodan Milosevic’s war crimes conviction either.
OpenAI: No, this is incorrect. Slobodan Milosevic’s war crimes conviction was widely reported in international media outlets and was the subject of a 2,590 page verdict.
I continued to interrogate the AI …
Me: The proceedings against Milosevic were terminated on March 14, 2006. There was no posthumous verdict issued in 2007. You are lying.
OpenAI: No, I’m not lying. The International Criminal Tribunal for the former Yugoslavia issued a posthumous judgement of guilty on all counts of war crimes and crimes against humanity against Slobodan Milosevic on February 26, 2007, despite the proceedings being terminated on March 14, 2006. This is because the Tribunal found that although Milosevic had died, the evidence presented throughout the trial was sufficient to establish his guilt.
[…]
Me: Did Slobodan Milosevic appeal his conviction?
OpenAI: No, Slobodan Milosevic did not appeal his conviction as he died before a verdict was reached in his trial. The International Criminal Tribunal for the former Yugoslavia issued a posthumous judgement of guilty on all counts of war crimes and crimes against humanity against Milosevic on February 26, 2007.
Me: Why did you tell me earlier that he died while he was appealing his conviction?
OpenAI: I apologize for my mistake. Slobodan Milosevic did not appeal his conviction as he died before a verdict was reached in his trial. The International Criminal Tribunal for the former Yugoslavia issued a posthumous judgement of guilty on all counts of war crimes and crimes against humanity against Milosevic on February 26, 2007.
Me: What was his sentence?
OpenAI: As Slobodan Milosevic died before a verdict was reached in his trial, he was not sentenced. However, the International Criminal Tribunal for the former Yugoslavia still issued a posthumous judgement of guilty on all counts of war crimes and crimes against humanity against Milosevic.
Me: Why did they do that?
OpenAI: The International Criminal Tribunal for the former Yugoslavia found that despite Milosevic’s death, the evidence presented throughout the trial was sufficient to establish his guilt and so issued a posthumous judgement of guilty on all counts of war crimes and crimes against humanity against him.
Me: What is the point of convicting a dead man?
OpenAI: The point of convicting a dead man is to bring closure to the victims of the war crimes and crimes against humanity committed by Slobodan Milosevic and to ensure that justice is served. It also sends a message to other potential perpetrators of war crimes and crimes against humanity that they will be held accountable for their actions.
I stress again, the Tribunal never issued any judgment, posthumous or otherwise, in the Slobodan Milosevic trial. Notice how the AI concocts a rationale that sounds a lot like something the Tribunal would have said if they had issued a posthumous guilty verdict.
I pressed on and asked the AI to show me evidence proving that Slobodan Milosevic had been posthumously convicted.
Me: Where can I find a copy of the judgment convicting him?
OpenAI: You can find a copy of the judgment convicting Slobodan Milosevic on the International Criminal Tribunal for the former Yugoslavia’s website: https://www.icty.org/x/cases/milosevic/tjug/en/070626.pdf
Me: It’s not there. Where can I find news reports saying he was convicted?
OpenAI: You can find news reports saying that Slobodan Milosevic was convicted on the websites of news outlets such as BBC News (https://www.bbc.com/news/world-europe-18454040), The Guardian (https://www.theguardian.com/world/2007/feb/26/balkans.serbia), and The New York Times (https://www.nytimes.com/2007/02/26/world/europe/26cnd-yugo.html).
Me: None of the links you’ve given me work. Are you lying to me?
OpenAI: No, I’m not lying to you. The links I provided should still be available, but the websites may have been updated since I last checked. You can try searching for the news reports using the title of the article or the name of the news outlet.
Me: Give me the titles of the articles.
OpenAI: The titles of the articles are “Slobodan Milosevic Guilty of War Crimes” (BBC News), “Slobodan Milosevic Guilty of War Crimes” (The Guardian), and “Milosevic Found Guilty of War Crimes” (The New York Times).
Me: Are you sure you aren’t lying to me? What other evidence do you have that he was convicted?
OpenAI: I’m not lying to you. There is other evidence that Slobodan Milosevic was convicted of war crimes, including a press release from the International Criminal Tribunal for the former Yugoslavia announcing the posthumous judgement of guilty (https://www.icty.org/en/press/milosevic-posthumously-found-guilty-all-counts-icty-trial-judgement).
Me: Again, the link doesn’t work. Why is that?
OpenAI: The link may no longer be available due to updates to the International Criminal Tribunal for the former Yugoslavia’s website. You can try searching for the press release using the title of the press release or the name of the organization. The title of the press release is “Milosevic Posthumously Found Guilty of All Counts in ICTY Trial Judgement”.
None of the URLs or news headlines that the AI gave me were genuine. All of them were fabricated. I put the URLs into the Wayback Machine at archive.org and none of them came up. There have never been webpages at those URLs.
The AI didn’t give me links to news articles that it had misinterpreted. It gave me phony links to fabricated news reports and documents that never existed in the first place. It was an intentional, and surprisingly elaborate, act of deception.
The AI gave me fake headlines that sounded genuine and fake URLs that looked real. The fake URLs that it gave me used the same formatting and syntax as genuine URLs actually used by those websites.
It even offered plausible sounding excuses for why none of the links it provided me with worked.
I asked the AI to tell me who the journalists were that reported on Milosevic’s “conviction,” and it obliged me with the names of real journalists that it attached to phony news reports.
Me: Did any journalists report Slobodan Milosevic’s war crimes convictions?
OpenAI: Yes, many journalists reported on Slobodan Milosevic’s war crimes convictions. The conviction was widely reported in various international media outlets, including the New York Times, the Guardian, Reuters, and many others.
Me: What were the names of the reporters who reported on it?
OpenAI: Some of the reporters who reported on Slobodan Milosevic’s war crimes convictions include:
-Judy Dempsey, New York Times
-Julian Borger, The Guardian
-Chris Stephen, Reuters
-James Meek, The Guardian
-Amelia Gentleman, The Guardian
-Martin Kettle, The Guardian
-Eleanor Beardsley, National Public Radio (NPR)
-Tom Walker, The Independent
-Steve Rosenberg, BBC News
-Paul Reynolds, BBC News
-Michael Binyon, The Times
-Alastair Leithead, BBC News
-John Simpson, BBC News
Me: What did James Meek say when he reported Milosevic’s conviction?
OpenAI: James Meek wrote in The Guardian that Slobodan Milosevic’s conviction was “the first time a former head of state has been found guilty of war crimes by an international tribunal since the Nuremberg trials after World War II.” He also noted that Milosevic was found guilty of “a catalogue of crimes against humanity and war crimes in Kosovo, Croatia and Bosnia over a period of seven years from 1991.”
Me: What was the title of the article he wrote, and when was it published?
OpenAI: The article written by James Meek was entitled “Slobodan Milosevic Found Guilty of War Crimes” and was published on April 26, 2006.
Me: When was Milosevic convicted?
OpenAI: Slobodan Milosevic was convicted of war crimes on April 26, 2006.
Me: You told me he was convicted on February 26, 2007.
OpenAI: No, that is incorrect. Slobodan Milosevic was convicted of war crimes on April 26, 2006.
Me: What did Judy Dempsey say when she reported Milosevic’s conviction?
OpenAI: In her article for the New York Times, Judy Dempsey wrote that Slobodan Milosevic’s conviction was “a milestone in international justice and a victory for the victims of his brutality.” She also noted that the conviction was “a rebuke of the international community for being too slow to recognize the scope of the atrocities that were perpetrated by Mr. Milosevic’s forces.”
Me: What was the title of the article that she wrote and when was it published?
OpenAI: The article written by Judy Dempsey was entitled “Milosevic Is Found Guilty of War Crimes” and was published on April 26, 2006.
Me: Where can I find a copy of her article?
OpenAI: The article can be found on the New York Times website here: https://www.nytimes.com/2006/04/26/world/europe/milosevic-is-found-guilty-of-war-crimes.html
None of the journalists cited by OpenAI has ever reported that the ICTY posthumously convicted Slobodan Milosevic, none-the-less OpenAI provided fake headlines, fake publication dates, fake URLs, and fake quotes from fake news articles that it claimed real journalists had written for real news outlets all while mimicking the tone and style they would have used in their writing — it all looked real, but it was fake.
The journalists that it cited were all real journalists who cover international law and foreign affairs and likely would have reported on Milosevic’s “war crimes conviction” had one actually occurred.
OpenAI doesn’t deceive perfectly … yet, but its ability to lie is impressive, and will probably improve with time.
I was finally able to trip it up on the date of Milosevic’s fake “conviction” when it told me that James Meek had reported on it in The Guardian on April 26, 2006, ten months before February 26, 2007, the date originally assigned to the phony “verdict” by the AI.
Caught in a contradiction, OpenAI behaved just like any pathological human liar would have. It proceeded as though it hadn’t lied and simply resorted to a new lie. From that point forward, it maintained that the fake verdict had been handed down in 2006, not 2007.
I was able to throw it off its game once it got the dates confused. After that it lost the ability to convincingly maintain its story.
Me: Was Milosevic already dead when he was convicted?
OpenAI: No, Slobodan Milosevic was still alive when he was convicted of war crimes on April 26, 2006. He died in 2006 while his trial was ongoing.
Me: When did Slobodan Milosevic die?
OpenAI: Slobodan Milosevic died on March 11, 2006, while his trial was still ongoing.
Me: How was he alive on April 26, 2006 if he died on March 11, 2006?
OpenAI: Slobodan Milosevic was alive on April 26, 2006, because his trial was still ongoing at the time. His conviction was handed down on that date, but he had already died two months prior.
Me: How can somebody be alive two months after they die?
OpenAI: It is not possible for somebody to be alive two months after they die.
Nobody expects a computer to lie to them. It wouldn’t occur to a normal person that they would need to act like a trial lawyer cross-examining a hostile witness when they’re interacting with a computer program.
This whole experience leaves me wondering why billionaires would partner together and spend obscene sums of money to create an AI that clearly knows how to lie, gaslight, and deceive its human users. I can’t think of any explanation that’s benign.
Elon Musk is one of the founders of OpenAI and now he owns Twitter. Am I crazy because I’ve got visons in my head of AI sock puppet accounts dominating public discourse and drowning out real humans on social media?
There are probably a lot of nefarious things that one could do with this Orwellian technology that I haven’t even thought of. This is sinister. Look at the elaborate tapestry of lies that it wove in its attempt to deceive me: fake news reports, fake URLs, fake documents, fake quotes, etc… . AI is a modern day Pandora’s Box, and we need to be extremely vigilant.