GPT-3 is particularly good at capturing the feeling of a genre or author, as I discovered when I used it to generate a hard boiled detective story about Harry Potter:
A small dingy office, early morning, furniture of the Salvation Army store variety. Sordid atmosphere. Harry Potter, in ratty tweed suit, unpressed shirt, and unshined shoes, sits behind the desk looking haggard, rumpled, and embittered.
The model generates plenty of non-sequiturs, but they’re easy to select out, and the quality of its writing given the right prompt is significantly higher than the average fan fiction you find online. In other words, GPT-3 can already write better fiction than most people in whatever style or fictional universe we want.
What’s more, the process is surprisingly fun – the experiments I’ve done feel similar to playing an old text adventure, except you never run out of canned text and at the end you’ve created a story that’s plausibly worth reading. I expect this process to be even more fun when done socially.
This is why I believe AI fan fiction is going to be a popular use of these models, both by individuals and mass collaborations – maybe with people voting on the best next sentence to the story.
To test this I wrote a script that lets me use GPT-3 to build a story a few lines at a time. I start with a story title and first sentence, then have GPT-3 generate twelve options for how to continue. If I don’t like any of the options I regenerate them. Every time I select an option it’s added to the text and fed back to GPT-3 to generate more options.
The process is about as fun as playing a video game or watching a good TV show. It feels a little dream-like in that you’re wondering what will happen next while simultaneously trying to guide the story. I think kids in particular will love it and imagine them doing things like inserting themselves into stories with their favorite characters.
The challenge is that the model better at making the next thing happen than it is at having an overarching plot. It can also be difficult to steer the story toward an ending. I didn’t have to generate additional options as much in the beginning, but toward the end I was probably averaging 5-10 regenerations (60-120 suggestions) per decision in order to drive the story toward a resolution. To be fair this is more because I wanted the story to go in a specific direction than it was because none of the options were good.
Another reason I did many regenerations was that I wanted to end the story before I went over the model’s word limit. In other experiments writing longer fiction I started deleting earlier parts of the story which makes GPT-3 lose context on what’s happened.
I have ideas for how to solve this limitation. I’ve had success using GPT-3 to compress large sections of a story into short summaries. By doing this repeatedly you can retain context while generating a longer story.
I also expect there’s a better way nudge the model toward resolving a story because it’s shown that it knows what an ending looks like: often at the end of a story it’ll generate blank lines, add ‘The End’, or start adding non-story text like reviews or comments.
There are parts of the story below that don’t quite make sense, but there are also parts that look like flashes of inspiration, like Death saying “THERE IS NO READINESS, ONLY PUNCTUALITY” and “HOW CAN WATER HAVE PNEUMONIA?”
Because I’m trying to demonstrate the power of the model I’ve adopted the artificial handicap of not being able to edit or add my own words. It’s easy to see the potential of advanced language models as writing tools because I’m already generating interesting content despite these handicaps. Without them it would be far easier: often GPT-3 will generate a continuation that’s almost, but not quite what I want, or I’ll see a promising direction I could point the model in by adding a single line.
(Again, as with my other GPT-3 posts, I picked the best options but other than the bolded prompt didn’t write a single word of the story.)
Here’s a short story by Terry Pratchett.
By Terry Pratchett
Death looked at the man and said ‘HELLO.’
The man statled. ‘Oh no.’
‘Can’t we talk about this?’ Death leaned forward and put his face inches from the man’s nose.
‘THIS IS ME TALKING TO YOU,’ he said.
The man quailed, and looked to the side.
‘I’m just not ready,’ he said.
‘THERE IS NO READINESS, ONLY PUNCTUALITY,’ said Death.
‘You’re supposed to give me time to put my affairs in order,’ the man said.
YOU HAVE EXACTLY THREE SECONDS.’
‘I… have got a cat…’
‘GOOD. AND NOW YOU HAVE TWO.’
‘And I haven’t finished my game of chess with the vicar…’
‘HOW SAD. AND NOW YOU HAVE ONE.’
‘Please,’ said the man, ‘please. I’ll do anything.’
Death sighed. He was very tired of doing this. ‘ALL RIGHT,’ he said. ‘TELL ME. WHAT IS IT YOU WANT MOST IN ALL THE WORLD?’
The man looked down at his hands, which were worn and callused. He looked up again. ‘Well,’ he said, ‘there is one thing.’
‘You can’t give it to me,’ said the man. ‘It’s impossible.’
Death shrugged. ‘THERE IS NO WANTING TO WHICH THERE IS NO GETTING,’ he said.
The man stood up, suddenly cheerful. ‘I want to live,’ he said.
‘YOU WANT TO LIVE?’ said Death. ‘AND YOU EXPECT ME TO GRANT THIS?’
‘I know it’s asking a lot,’ said the man. ‘But I’ll be really grateful.’
Death rubbed his chin with a skeletal finger. He thought about this. And then he said, ‘ALL RIGHT. TELL ME. HOW DID YOU DIE?’
‘What? Oh. I died of cold.’
‘COLD?’ Death stared at him. ‘COLD? IN THE SUMMERTIME?’
‘Yes. There was a great flood, and the cold got in everywhere. I had pneumonia.’
‘BUT THERE WAS NO FLOOD.’
‘There was a flash flood. It killed me.’
‘WHAT ABOUT THE PNEUMONIA?’
‘It was in the water.’
‘HOW CAN WATER HAVE PNEUMONIA?’
‘I don’t know,’ said the man. ‘But it did. It was horrible.’
Death frowned. He could sense that there was something wrong here, but he couldn’t quite put his finger on it.
‘YOU SURE YOU WANT TO COME BACK?’
Death sighed again. ‘ALL RIGHT,’ he said. ‘WELL, IF YOU ARE SURE…’
He clapped his hands together. There was a small puff of smoke, and then the man was gone.
Death got up and walked across the room. The door opened for him. He stood for a moment in the doorway, watching the rain fall.
THERE WAS NO FLOOD,’ he said.
‘I know,’ said a voice behind him. Death turned round.
‘Yes. It was me.’
‘YOU WERE IN THE FLOOD?’
‘No. I started the flood.’
‘Because it was the only way I could think of to meet you.’
Death sighed. He tried not to do it, but somehow he just couldn’t help himself.
‘OK,’ he said. ‘YOU’RE MEETING ME. NOW WHAT?’
The man pulled a small object out of his pocket. It was made of bone. He opened his hand, and a ball of light appeared above it. It spread out into a thousand fragments, which whirled around each other, gaining in speed and brightness.
‘NOW I AM YOU,’ he said.
‘I AM THE NEW DEATH. I HAVE TAKEN UP MY PLACE IN TIME. YOU ARE FINISHED. YOUR WORK IS DONE. YOU HAVE ENTERED HISTORY.’
‘But…’ Death looked down at himself. His robe was starting to fray at the edges. His scythe was losing its edge. His fingers were slightly yellow. He was suddenly aware of a twinge in his knees, and a strange feeling in the small of his back.
‘Yes,’ he said. ‘I suppose I am.’
‘THE TIME OF DEATH IS OVER.’
‘No more dying?’ said Death.
‘NO MORE DEATH,’ said the new Death.
Death shook his head. ‘This is going to take some getting used to,’ he said. ‘And I’m going to need a new name.’
‘YOU’RE NOT THE ONLY ONE,’ said the new Death.
‘What do you mean?’
‘I MEAN, I HAVE CHANGED MY NAME.’
‘Barry?’ said Death.
‘I THOUGHT IT SOUNDED FRIENDLY.’
‘What about “Mildew”?’ said Death.
‘That doesn’t sound friendly,’ said Barry.
‘No. No, you’re right there. All right, Barry it is.’
‘AND YOU NEED A NAME, TOO.’
‘Yes,’ said Death. ‘I’ve been thinking about that. I think I’d like to be called… Doug.’
‘ALL RIGHT. DOUG IT IS.’
‘Thank you,’ said Death.
They walked together through the rain.
‘You know,’ said Death, ‘I was getting rather tired of the whole business, to be honest. You know, the scythe, the robes, the whole nine yards.’
‘I THOUGHT YOU MIGHT BE.’
‘It’s been very rewarding, of course. I’ve met a lot of people.’
‘WHAT WAS THE BEST PART?’
‘When I got to kill Hitler.’
‘It was. I was quite proud of that one.’
They walked on for a while in silence.
‘What will you do now?’ said Death.
‘WELL,’ said Barry, ‘I’VE GOT SOME IDEAS. I’VE BEEN READING ABOUT THIS NEW THING CALLED THE INTERNET. PEOPLE SAY IT HAS A FUTURE.’
‘I shall miss you, Barry.’
‘YOU CAN STILL COME ALONG, IF YOU LIKE.’
‘WE COULD HAVE FUN TOGETHER.’
‘Don’t you think it’s a little… unprofessional?’
‘OH, WELL. THAT DOESN’T REALLY MATTER ANY MORE.’
‘Yes. I suppose you’re right.’
‘LET’S GO AND HAVE SOME FUN,’ said Barry. ‘THERE’S A LOT TO SEE AND DO.’
Death thought for a moment. ‘Yes,’ he said. ‘There is, isn’t there?’
And the two of them went off, in the rain.