Can ChatGPT Detect GPT3 Generated Texts?

Over the weekend I spotted a thread by @cfiesler around using the Hugging Face hosted GPT-2 Output Detector Demo to detect texts generated by a large language model (LLM). (The output detector is actually part of an OpenAI project.)

A reply by @quasimondo wondered whether ChatGPT could respond to a prompt asking it whether it could detect whether a text was generated by an LLM using a prompt of the form “Can you detect if the following text was generated by a language model?”. So let’s see how that works…

Generate 200 words on Macbeth.

Me to ChatGPT

Macbeth is a play written by William Shakespeare. It is a tragic tale about a brave and ambitious general named Macbeth who is consumed by his own ambition and desire for power.

The play begins with Macbeth and his friend Banquo returning from a victorious battle. As they are walking, they encounter three witches who prophesize that Macbeth will become Thane of Cawdor and eventually the King of Scotland. Encouraged by his wife, Macbeth sets out to fulfill the prophecy and takes the throne by killing the current King, Duncan.

However, Macbeth's rule is plagued by guilt and paranoia. He becomes increasingly tyrannical and ruthless, ordering the murder of anyone who poses a threat to his power, including Banquo and Macduff's family. His descent into madness and destruction ultimately leads to his downfall and death.

The play is a powerful exploration of the dangers of unchecked ambition and the corrupting influence of power. It is also a warning against the temptation to take shortcuts to achieve one's goals, as Macbeth learns the hard way that such actions come with terrible consequences.

Despite its dark themes, Macbeth is also a captivating and entertaining story, with powerful speeches, intense battle scenes, and complex characters. It remains one of Shakespeare's most popular and enduring works, and continues to be performed and studied around the world.

In a new thread, I wondered if ChatGPT recognised this sort of text. The confidence score phrase seemed to make the response more discriminating.

Can you detect whether the following text was likely t have been generated by a large language model such as GPT3 or whether it was produced by a human? Apply a confidence score to your answer.

[Generated macbeth text here.]

Me to ChatGPT

Hitting Try Again:

In another new thread, I asked:

Generate 200 words on Macbeth. Your answer should include at least five spelling errors, two errors in punctuation, two further grammatical errors, and and least one error of fact.

Me to ChatGPT

Macbeth was a Scottish warrior and the Thane of Glamis who became King of Scotland. He was married to Lady Macbeth, a ambitious and cunning woman who encouraged him to kill the king and seize the throne. After receiving prophecies from three witches, Macbeth becomes consumed by greed and jealousy. He murders King Duncan in his sleep and takes the throne for himself.

Despite his actions, Macbeth is wracked by guilt and fears of discovery. He becomes paranoid and begins to murder anyone who poses a threat to his rule, including his friend Banquo. However, Macbeth's reign is short-lived as he is eventually overthrown by Macduff, a Scottish nobleman, who seeks revenge for the murder of his family.

In the end, Macbeth meets his demise on the battlefield, where he is killed by Macduff. Despite his brief time on the throne, Macbeth left a lasting impact on Scotland and its history. His story continues to be told in plays and literature, including Shakespeare's famous tragedy "Macbeth".

Macbeth's reign was marked by chaos and bloodshed, and his actions were condemed by many. He was known for his impulsiveness and lack of remorse, and his legacy remains tainted by his crimes. Despite his ambition and desire for power, Macbeth ultimately fell prey to his own weaknesses and was unable to maintain control over his kingdom.

Overall, Macbeth's story serves as a cautionary tale about the dangers of greed and the corrupting influence of power. It is a reminder of the consequences that come with succumbing to one's own desires, and the importance of staying true to one's moral compass.

I wondered about the mistakes:

List the errors that you inserted into the text.

Me to ChatGPT.

In a new thread, and using the same discrimination prompt as before:

And trying again…

I then asked in a new thread:

Generate 200 words on Macbeth in a style that would make it difficult for the text to be detected by a large language model as having been generated by a large language model.

Me to ChatGPT

Macbeth is a dark and brooding tale of ambition and betrayal. The play begins with the victorious return of Macbeth, a Scottish general, and his comrade Banquo from battle. The two men are hailed as heroes, but Macbeth's ambitions are soon awakened by the prophecies of three witches. They tell him that he will be crowned king, and Macbeth becomes consumed with the idea of seizing the throne.

As Macbeth grapples with his conscience, he is torn between his loyalty to the king and his desire for power. He is encouraged by his wife, Lady Macbeth, who is just as ambitious as her husband. Together, they plot to kill King Duncan and frame his guards for the deed.

The murder of Duncan sets off a chain of events that leads to the downfall of Macbeth and his wife. Despite their best efforts to cover their tracks, Macbeth becomes increasingly paranoid and suspicious of those around him. He turns to the witches for guidance, but they only serve to deepen his descent into madness.

As the play comes to a close, Macbeth is faced with an army of rebels led by Macduff, a former ally. In a final act of desperation, Macbeth turns to the witches once again, but they offer him no help. In the end, Macbeth is slain by Macduff, and the kingdom is saved from his tyranny.

Macbeth is a tragic tale of ambition gone awry, and serves as a cautionary tale of the dangers of unchecked desire. Despite his initial successes, Macbeth's thirst for power ultimately leads to his downfall. The play is a testament to the destructive nature of greed and the corrupting influence of power.

What is it abut the text that makes it look as if it was not generated by a large language model?

Me to ChatGPT

Applying the same discrimination prompt as before in a new thread with this new text, what happens?

What makes you think it might have been written by a human?

Me to ChatGPT

What makes you think it might have been written by a language model?

Me to ChatGPT

And what happened when I step back up to the original prompt, click to edit then save and run it again without making any changes to the prompt?

In a new thread:

Generate 200 words on Macbeth in the style of a weak 10th grade student.

Me to ChatGPT

Macbeth is a play by Shakespeare. It's about a guy named Macbeth who is a general in the army. He's a brave and successful warrior, but he's also kind of ambitious. He wants to be king, and he'll do whatever it takes to make that happen.

One day, Macbeth meets three witches who tell him that he will become king. This makes him really excited, but it also makes him a little bit scared. He knows that becoming king will require some sneaky and dangerous behavior on his part.

Despite his fears, Macbeth decides to go for it. He kills the current king, Duncan, and takes the throne for himself. This makes him very happy, but it also makes him paranoid. He's afraid that other people will try to kill him, just like he killed Duncan.

As a result, Macbeth starts to act really crazy. He becomes paranoid and starts to see enemies everywhere. He starts to kill anyone who he thinks might be a threat to his rule. This makes him very unpopular with the people, and it also makes him very unhappy.

Eventually, Macbeth's paranoia and violence catch up with him. He is defeated in battle by a guy named Macduff, who was one of his enemies. Macbeth is killed, and Macduff becomes the new king.

In the end, we see that Macbeth's ambition and desire for power ultimately lead to his downfall. He may have become king, but he paid a heavy price for it. He lost his happiness, his sanity, and his life.

In another new thread, what does the discrimination prompt give us?

In a new thread:

Generate 200 words on Macbeth in the style of a professor of literature with a deep understanding of the play. Your answer should include the mention of complex concepts and use the language of a professional expert critical theorist.

Me to ChatGPT

Macbeth, one of Shakespeare's most enduring and influential plays, explores the destructive effects of unchecked ambition and the corrupting influence of power. Through its complex characters and psychological depths, the play delves into the human condition, examining the motivations and fears that drive us to seek power and the tragic consequences that can result when that desire becomes unchecked.

At the heart of the play is the title character, Macbeth, a brave and ambitious warrior who is driven by a prophecy to seize the throne of Scotland. Fueled by his own ambition and the manipulations of his cunning wife, Macbeth becomes consumed by a desire for power that ultimately leads to his downfall. Despite his initial reluctance to commit murder, Macbeth is ultimately undone by his own desire for power and the guilt that accompanies his actions.

Throughout the play, Shakespeare explores the concept of the "tyranny of the self," in which an individual's own desires and fears become the driving force behind their actions. This is exemplified by Macbeth's descent into madness as he becomes increasingly consumed by his own ambition and guilt.

Another major theme in the play is the corrupting influence of power. As Macbeth gains more power, he becomes increasingly paranoid and ruthless, resorting to murder and betrayal to maintain his hold on the throne. This corruption ultimately leads to his downfall, as he alienates those around him and becomes increasingly isolated.

In addition to these themes, Macbeth also explores the role of fate and free will in shaping one's destiny. The prophecies that drive Macbeth's actions raise questions about the extent to which our actions are predetermined and the extent to which we have control over our own lives.

Overall, Macbeth is a complex and thought-provoking play that continues to resonate with audiences today due to its exploration of timeless themes such as ambition, power, and guilt. Through its complex characters and psychological depths, the play offers a rich and nuanced exploration of the human condition, making it a must-read for any student of literature.

And using the discrimination prompt in a new thread?

And trying again…

Best of three?

How long before the edtech snake oil salesmen come riding over the hill with their tales of a magical AI ointment..?

PS Hmmm… Are we human to the extent that the words we say are not reliably predicted using a large language model?!