It has been two years since the online premiere of a short film called ‘Sunspring.’ What’s the big deal about that? Well, the entire script was created by an AI. An AI system created the content for the short, while human actors took the strange semi-human dialogue in stride and gave earnest performances.
That production duo responsible for the film, director Oscar Sharp and AI researcher Ross Goodwin, are back with another AI project, though this next installment does not inspire much confidence. ‘Zone Out,’ is the film created as an entry in the Sci-Fi-London 48-Hour Challenge. The project had to be produced in 48 hours and adhere to certain specific prompts. These are the same criteria under which ‘Sunspring’ was made. This time, Sharp and Goodwin let their AI system, lovingly referred to as Benjamin, handle the film’s entire production from top to bottom.
In order to have Benjamin “write, direct, perform and score” this short film without any human intervention, Sharp and Goodwin developed a workflow, Sharp said in a recent interview. This workflow had Benjamin taking care of the following: putting together footage from public domain films, face-swapping the database of human actors into that footage, inserting voices to read Benjamin’s script, and scoring the film. What a busy robot!
AIs like Benjamin rely on a LSTM (long short-term memory) recurrent neural network, which Annalee Newitz from ARS Technica explained:
“To train Benjamin, Goodwin fed the AI with a corpus of dozens of sci-fi screenplays he found online—mostly movies from the 1980s and ’90s. Benjamin dissected them down to the letter, learning to predict which letters tended to follow each other and from there which words and phrases tended to occur together. The advantage of an LSTM algorithm over a Markov chain is that it can sample much longer strings of letters, so it’s better at predicting whole paragraphs rather than just a few words. It’s also good at generating original sentences rather than cutting and pasting sentences together from its corpus. Over time, Benjamin learned to imitate the structure of a screenplay, producing stage directions and well-formatted character lines.”
The new film, ‘Zone Out’ has a script that, like ‘Sunspring’s,’ is odd and not-quite-human, using strange and choppy sentences that normal humans would not say. The weirdness of the dialogue is magnified by having so many other film-production tasks automated by AI (Benjamin), including face-swapped actors.
During production of ‘Zone Out,’ Sharp and Goodwin struggled to find enough public-domain film footage that they could use. The issue wasn’t just about copyright, the footage also had to contain enough shots with one actor facing directly toward the camera that Benjamin could easily insert it into whatever it composed. With the help of their lawyer, Goodwin and Sharp settled on two films to use for the project: ‘The Last Man on Earth’ and ‘The Brain That Wouldn’t Die.’
The impact of the time limitation is obvious in the final product, and Sharp himself admits that limits from the AI technology hindered their vision of a decent-looking and sounding finished product. Issues with the face-swap and face-puppeteering systems in place also didn’t do the film many favors. Sharp explained:
“We eventually had to accept that the film would just look ‘badly dubbed.”
However, the biggest failure in the process came from an attempt to have a different AI system, a convolutional neural network, automate the process of selecting footage from the public domain films to be edited by Benjamin. Sharp said:
“There were neither sufficient object descriptors in the screenplay nor sufficient numbers of unique objects in the shots.”
This meant the auto-editing system didn’t have enough data to use. Sharp and Goodwin were careful to let the AI makes decisions as a “director” and pick film scenes, shot lengths, and casting assignments according Benjamin’s vision. Sharp said:
“Editor Jono Chanin and I worked under the presumption that this was the story Benjamin was trying to tell and edited accordingly, while also keeping verbatim to Benjamin’s screenplay. So here, some human interpretation finally broke in, despite my hope to purge it completely from this iteration of the Benjamin saga. That remains a goal for next time.”
Sharp isn’t giving up on the project and thinks that improved computational efficiency and refined data-parsing tools may make this kind of computer-generated filmmaking a real possibility in the future. He explained:
“In the days since this experiment, Ross has already uncovered some new technologies he thinks could lead us to fully automated editing—and something else we’re nicknaming ‘whole-movie puppetry. Exciting stuff.”
Check out ‘Zone Out’ below!
