Cognitive Computing

IBM Research Takes Watson to Hollywood with the First “Cognitive Movie Trailer”

Share this post:

How do you create a movie trailer about an artificially enhanced human?

You turn to the real thing – artificial intelligence.

20th Century Fox has partnered with IBM Research to develop the first-ever “cognitive movie trailer” for its upcoming suspense/horror film, “Morgan”. Fox wanted to explore using artificial intelligence (AI) to create a horror movie trailer that would keep audiences on the edge of their seats.

Movies, especially horror movies, are incredibly subjective. Think about the scariest movie you know (for me, it’s the 1976 movie, “The Omen”). I can almost guarantee that if you ask the person next to you, they’ll have a different answer. There are patterns and types of emotions in horror movies that resonate differently with each viewer, and the intricacies and interrelation of these are what an AI system would have to identify and understand in order to create a compelling movie trailer. Our team was faced with the challenge of not only teaching a system to understand, “what is scary”, but then to create a trailer that would be considered “frightening and suspenseful” by a majority of viewers.

As with any AI system, the first step was training it to understand a subject area. Using machine learning techniques and experimental Watson APIs, our Research team trained a system on the trailers of 100 horror movies by segmenting out each scene from the trailers. Once each trailer was segmented into “moments”, the system completed the following;

1)   A visual analysis and identification of the people, objects and scenery. Each scene was tagged with an emotion from a broad bank of 24 different emotions and labels from across 22,000 scene categories, such as eerie, frightening and loving;

2)   An audio analysis of the ambient sounds (such as the character’s tone of voice and the musical score), to understand the sentiments associated with each of those scenes;

3)   An analysis of each scene’s composition (such the location of the shot, the image framing and the lighting), to categorize the types of locations and shots that traditionally make up suspense/horror movie trailers.

The analysis was performed on each area separately and in combination with each other using statistical approaches. The system now “understands” the types of scenes that categorically fit into the structure of a suspense/horror movie trailer.

Then, it was time for the real test. We fed the system the full-length feature film, “Morgan”. After the system “watched” the movie, it identified 10 moments that would be the best candidates for a trailer. In this case, these happened to reflect tender or suspenseful moments. If we were working with a different movie, perhaps “The Omen”, it might have selected different types of scenes. If we were working with a comedy, it would have a different set of parameters to select different types of moments.

It’s important to note that there is no “ground truth” with creative projects like this one. Neither our team, or the Fox team, knew exactly what we were looking for before we started the process. Based on our training and testing of the system, we knew that tender and suspenseful scenes would be short-listed, but we didn’t know which ones the system would pick to create a complete trailer. As most creative projects go, we thought, “we’ll know it when we see it.”

Our system could select the moments, but it’s not an editor. We partnered with a resident IBM filmmaker to arrange and edit each of the moments together into a comprehensive trailer. You’ll see his expertise in the addition of black title cards, the musical overlay and the order of moments in the trailer.

Not surprisingly, our system chose some moments in the movie that were not included in other “Morgan” trailers. The system allowed us to look at moments in the movie in different ways –moments that might not have traditionally made the cut, were now short-listed as candidates. On the other hand, when we reviewed all the scenes that our system selected, one didn’t seem to fit with the bigger story we were trying to tell –so we decided not to use it. Even Watson sometimes ends up with footage on the cutting room floor!

Traditionally, creating a movie trailer is a labor-intensive, completely manual process. Teams have to sort through hours of footage and manually select each and every potential candidate moment. This process is expensive and time consuming –taking anywhere between 10 and 30 days to complete.

From a 90-minute movie, our system provided our filmmaker a total of six minutes of footage. From the moment our system watched “Morgan” for the first time, to the moment our filmmaker finished the final editing, the entire process took about 24 hours.

Reducing the time of a process from weeks to hours –that is the true power of AI.

The combination of machine intelligence and human expertise is a powerful one. This research investigation is simply the first of many into what we hope will be a promising area of machine and human creativity. We don’t have the only solution for this challenge, but we’re excited about pushing the possibilities of how AI can augment the expertise and creativity of individuals.

AI is being put to work across a variety of industries; helping scientists discover promising treatment pathways to fight diseases or helping law experts discover connections between cases. Film making is just one more example of how cognitive computing systems can help people make new discoveries.

Add Comment
60 Comments

Leave a Reply

Your email address will not be published.Required fields are marked *

[…] better way to ignore that rule than to have IBM task an AI to create the first-ever “cognitive movie […]

Reply

Sebastian

¿Who was the filmmaker?

Reply

[…] make the film, an IBM blog post explains, Watson analyzed the trailers of over 100 horror and thriller film trailers to see understand what […]

Reply

[…] make the film, an IBM blog post explains, Watson analyzed the trailers of over 100 horror and thriller film trailers to understand what […]

Reply

[…] make a film, an IBM blog post explains, Watson analyzed a trailers of over 100 fear and thriller film trailers to know what sounds, […]

Reply

[…] make the film, an IBM blog post explains, Watson analyzed the trailers of over 100 horror and thriller film trailers to see understand what […]

Reply

[…] make the film, an IBM blog post explains, Watson analyzed the trailers of over 100 horror and thriller film trailers to understand what […]

Reply

[…] make the film, an IBM blog post explains, Watson analyzed the trailers of over 100 horror and thriller film trailers to understand what […]

Reply

[…] Morgan opens September 2. For more on the IBM team’s process check out their PR blog post here. […]

Reply

Lifeboat News: The Blog

[…] beyond their control. Fitting, then, that they’d employ the help of America’s AI sweetheart IBM Watson to build the film’s […]

Reply

[…] трейлер фільму жахів "Морган". Як повідомляється в блозі IBM, студія попросила створити трейлер, який "буде […]

Reply

[…] трейлер для фильма ужасов "Морган". Как сообщается в блоге IBM, студия попросила создать трейлер, который "будет […]

Reply

[…] The good news it that it doesn’t look like computers will be replacing filmmakers or editors anytime soon, but if you want to know more about the process of how this was accomplished, check out a full write-up over here. […]

Reply

[…] to make movie trailers. 20th Century Fox has tapped into the supercomputer’s powers to create the first AI-made trailer for its upcoming thriller film Morgan. It’s a fitting start for […]

Reply

[…] The good news it that it doesn’t look like computers will be replacing filmmakers or editors anytime soon, but if you want to know more about the process of how this was accomplished, check out a full write-up over here. […]

Reply

Maria Capello

I can’t wait to see this movie. I love Thrillers. I wonder if IBM will do a movie premiere? or even get discount tickets to the movies?

Reply

[…] do filme Morgan. O filme fala sobre um super ser humano com modificações cibernéticas; por isso, a equipe de pesquisa da IBM usou a plataforma de inteligência artificial Watson para editar o “trailer perfeito” para o filme. Ele pode ser visto no final desta […]

Reply

Luca Marchi

What are the Watson services/APIs that were used in this project?

Reply

[…] son blog, John Smith, qui a piloté le projet pour IBM, explique comment Watson, l’intelligence […]

Reply

[…] с научно-исследовательским подразделением IBM Research приказали когнитивной системе IBW Watson изготовить трейлер с для […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, […]

Reply

[…] с научно-исследовательским подразделением IBM Research приказали когнитивной системе IBW Watson изготовить трейлер с для […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, ‘what is […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, ‘what is […]

Reply

[…] с научно-исследовательским подразделением IBM Research приказали когнитивной системе IBW Watson изготовить трейлер с для […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, ‘what is […]

Reply

[…] 20th Century Fox has partnered with IBM Research to develop the first-ever “cognitive movie trailer” for its upcoming suspense/horror film, “Morgan”. Fox wanted to explore using artificial intelligence (AI) to create a horror movie trailer that would keep audiences on the edge of their seats.” (Lesen mehr unter IBM.com) […]

Reply

[…] September 2; check out the trailer below for a hint of what the movie is all about (and even check out an AI-created movie trailer!) Discuss further over in the 3D Printed Morgan forum at […]

Reply

[…] pas pu tout faire toute seule. John Smith, l’ingénieur d’IBM en charge du projet indique donc qu’il a fallu ensuite procéder à un montage et à des finitions plus abouties, bel et bien […]

Reply

[…] Fox approached IBM and asked them if they’d be willing to let Watson have a go at creating a trailer for the movie. The movie was, after all, about a sentient machine. IBM agreed. […]

Reply

[…] Fox approached IBM and asked them if they’d be willing to let Watson have a go at creating a trailer for the movie. The movie was, after all, about a sentient machine. IBM agreed. […]

Reply

[…] Fox approached IBM and asked them if they’d be willing to let Watson have a go at creating a trailer for the movie. The movie was, after all, about a sentient machine. IBM agreed. […]

Reply

[…] down the film to just six minutes of footage. Of course, not everything it picked was suitable, but according to IBM, it offered a different way of evaluating […]

Reply

[…] Watson was able to make the trailer by first analyzing 100 horror movie trailers, studying each scene, and looking for common ground. Next, it was fed the 90-minute movie, and chose 10 scenes that it thought would fit in  the trailer. An editor then pieced together the scenes. […]

Reply

[…] Watson was able to make the trailer by first analyzing 100 horror movie trailers, studying each scene, and looking for common ground. Next, it was fed the 90-minute movie, and chose 10 […]

Reply

Laleph : Actualite alternative

[…] de composer une bande-annonce –ce qu’elle est parvenue à faire–, explique John Smith, sur son blog, qui a piloté le projet. Pour autant, Watson n’est pas parvenue à «monter» ces extraits […]

Reply

[…] Source: IBM […]

Reply

[…] and understand in order to create a compelling movie trailer,” wrote IBM’s John R. Smith, in a blog post. […]

Reply

[…] différemment en fonction de chaque spectateur »,  explique John Smith sur le site d’IBM. « Il fallait donc que Watson apprenne à reconnaître ce qui était […]

Reply

[…] de composer une bande-annonce –ce qu’elle est parvenue à faire–, explique John Smith, sur son blog, qui a piloté le […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, ‘what is […]

Reply

[…] movie trailer,” John R. Smith, IBM Fellow and manager of multimedia and vision said in a post announcing the project. “Our team was faced with the challenge of not only teaching a system to understand, ‘what is […]

Reply

Buttercup

This is Amazing, a new step in AI world. good job IBM Watson project.

Reply

[…] По материалам FastCoCreate, Asmarterplanet. […]

Reply

Chris Briggs

The question isn’t whether AI systems can create. The question is whether they would want to.

Reply

[…] O resultado você confere no vídeo abaixo, que também acompanha falas dos profissionais da IBM no projeto e do próprio diretor, Luke Scott(em inglês). Caso queria dar uma olhada no post original no blog da IBM, clique aqui. […]

Reply

[…] Volgens IBM was dit vooral een experiment om te kijken wat de uitkomst zou zijn. Uiteindelijk heeft het team één moment dat door Watson was geselecteerd niet gebruikt, omdat het niet bij de rest van de trailer zou passen. Het team beweert verder dat het maken van de trailer hierdoor in 24 uur voltooid kon worden. […]

Reply

[…] IBM-Blog auf asmarterplanet.com erläutert Projektleiter John R. Smith: „Unser System konnte die Momente auswählen, aber es […]

Reply

[…] nécessité une triple analyse : visuelle, sonore et de composition. C’est ce qu’explique John R. Smith, un cadre d’IBM chargé de la vision par ordinateur et du […]

Reply

[…] John R. Smith erklärt in einem Blogbeitrag wie genau das […]

Reply