The AI Story Behind the Wimbledon Poster

By David Provan

The AI Story Behind the Wimbledon Poster

This article was written by Stephen Hammer, IBM’s Sports CTO with IBM iX.

In 2018, the Official Championships Poster celebrates the 150th anniversary of The All England Lawn Tennis Club. The Club was founded in 1868 with the first Championships being held in 1877. Posters have featured tennis and The Club for many years with the earliest poster in the Museum’s collection dating from 1893 – a railway poster advertising how to get to The Championships. Over the years since the Official Championships Poster has featured a range of styles, designers and world-famous artists.

With such a rich heritage and a significant milestone to commemorate the selection of an image for the 2018 poster proved to be an interesting challenge. Which of many thousands of images of The Club over the years should be chosen? An intriguing answer surfaced: why not use all of them?

The image for the poster features the Club House during The Championships and on a quick glance looks like a water colour image. On closer inspection you can see that it is in fact made up of a mosaic of images. Over 300,000 separate images were reviewed from the entire historical archive of The Club before nearly 9,000 images were selected to make up the final design. While mosaic images are not uncommon the approach used to create this image is a unique combination of art and artificial intelligence. Working with IBM and Watson their industry leading artificial intelligence capability, AI became the artist.

Below is the source image chosen for us to work from:

Club House

The task of looking through such a large photography archive would have required significant time from a dedicated designer.  A typical mosaic approach would also match tiles based on colors present in the source image. The unique approach taken using Watson for this poster is that images from the archive have been matched to specific parts of the original image: photography of ivy to match the ivy on the club house; ball boys matched to ball boys; grass courts to grass courts. The mosaic is actually a number of different themed mosaic images merged together.

To create the poster in this way, Watson was taught to recognise various elements within the original image. Once taught Watson was able to search the entire archive search for the required images to make the mosaic, finding the right context for the picture to recreate the original. The result is 150 years of archive photography creating a single beautiful image.

Watson Visual Recognition API

Classifying large numbers of photos is a very large task. It’s a lot of work to visually look at every photo and make content decisions for that photo. This is where AI comes in. The IBM Watson Visual Recognition APIprovides several different ways for developers to classify and obtain metadata from photos. The following key features were used in this project:

  • Generic classification: Out of the box, Watson’s Visual Recognition Pre-trained models can identify the presence of many things in an image. Try the online demo here.

  • Face Detection: Detects faces and dimensions in photos.

  • Custom Models: Visual Recognition can also be trained for custom images and objects.

To use Watson Visual Recognition, you will need an API key. You can find instructions on how to start here.

Watson Visual Recognition these 3 features as broad strokes to help categorize these 300,000 images into the following themes. Watson Visual Recognition was further used to help refine and prioritize images within these themes.

  • Courts

  • Fans

  • Nets

  • Scoreboards

  • Chair Umpires

  • Ballkids

  • Backstop

  • Clubhouse

  • Clubhouse Roof

  • Clubhouse Clock

  • Clubhouse Covered Areas

  • Uncategorized Regions

  • Sky

  • Roof

The categorized photos were cropped into tiles and applied to the theme-appropriate area in the source image. The following image shows how these themes were applied with each border outlining a different theme region:

categorized photos

Courts

To begin categorizing images, a custom classifier was created to identify court images from non-court images. We were specifically looking for the perspective above the baseline looking down on the court for a uniform look when the image was tiled.

To create a custom classifier with Watson Visual Recognition, images that are positive examples of what we are looking for are needed. Negative examples (what we are not looking for) can be optionally provided. For this classifier, we supplied 100 positive photos and 100 negative photos. Each were put in separate zip files and uploaded into the Watson Visual Recognition interface. Here’s are two examples:

Positive: Overhead shot of the court

Positive: Overhead shot of the court

Negative

Negative

curl -X POST -F “images_file=@00002382.png” –max-time 30 “https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key=<<YOUR-API-KEY-HERE>>&version=2018-03-19&classifier_ids=<<YOUR-CLASSIFIER-NAME-HERE>>&threshold=0.3

A positive JSON response payload with a 0.526 confidence level looks like this:

{

“images”: [

{

“classifiers”: [

{

“classifier_id”: “<<YOUR-CLASSIFIER-ID-HERE>>”,

“name”: “<<YOUR-CLASSIFIER-NAME-HERE>>”,

“classes”: [

{

“class”: “Wimbledon Tennis Court”,

“score”: 0.525552

}

]

}

],

“image”: “00002382.png”

}

],

“images_processed”: 1,

“custom_classes”: 1

}

We found 30,000 court images this way. However, many of these images needed cropping. As you can see in the image from above, the backstop is visible at the top of the image:

backstop

The image is valid but when placed in the mosaic along with other images, stripes appeared in the finished mosaic where the backstop is. This is not look good in the finished product and looked like stripes. It needed to be cropped out and removed.

be cropped out and removed

To crop these images to remove the stripe, each image was examined using another classifier. If the top or the bottom of the image contained this dark stripe then the image was quarantined and cropped. The resulting mosaic did not contain this.

resulting mosaic

Fans

To detect fans in images, we used the detect_faces API. This allowed us to count the number of faces in images and the size of those faces. Images with a large number of faces in them were safely assumed to be fan images. Note: These images were pre-screened with the pre-trained models as containing faces (see Umbrella example below). Also, the pre-trained models contain a model called “crowd” which was also used.

Fans

curl -X POST -F “images_file=@3544151120001.jpg” –max-time 30 “https://gateway-a.watsonplatform.net/visual-recognition/api/v3/detect_faces?api_key=<<YOUR-API-KEY-HERE>>&version=2018-03-19&threshold=0.3”

And the JSON response:

{

“images”: [

{

“faces”: [

{

“age”: {

“min”: 26,

“max”: 31,

“score”: 0.5460434

},

“face_location”: {

“height”: 47,

“width”: 45,

“left”: 215,

“top”: 87

},

“gender”: {

“gender”: “FEMALE”,

“score”: 0.9997433

}

},

{

“age”: {

“min”: 26,

“max”: 34,

“score”: 0.30170313

},

“face_location”: {

“height”: 61,

“width”: 43,

“left”: 385,

“top”: 2

},

“gender”: {

“gender”: “MALE”,

“score”: 0.6910262

}

},

{

“age”: {

“min”: 24,

“max”: 28,

“score”: 0.76478297

},

“face_location”: {

“height”: 40,

“width”: 31,

“left”: 91,

“top”: 93

},

“gender”: {

“gender”: “MALE”,

“score”: 0.72577006

}

},

{

“age”: {

“min”: 28,

“max”: 32,

“score”: 0.6017702

},

“face_location”: {

“height”: 60,

“width”: 35,

“left”: 0,

“top”: 96

},

“gender”: {

“gender”: “MALE”,

“score”: 0.99572265

}

},

{

“age”: {

“min”: 25,

“max”: 28,

“score”: 0.97392553

},

“face_location”: {

“height”: 35,

“width”: 34,

“left”: 263,

“top”: 179

},

“gender”: {

“gender”: “FEMALE”,

“score”: 0.69005907

}

}

],

“image”: “3544151120001.jpg”

}

],

“images_processed”: 1

}

Umbrellas

We didn’t need a lot of umbrella photos but they were easily found using the Watson Visual Recognition pre-trained models.

Umbrellas

curl -X POST -F “images_file=@5002074328001.jpg” –max-time 30 “https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key=<<YOUR-API-KEY-HERE>>&version=2018-03-19&threshold=0.6”

And the JSON response that contains a 0.63 confidence level for an Umbrella. Note the threshold was set to 0.6 to reduce noise in the response.

{

“images”: [

{

“classifiers”: [

{

“classifier_id”: “default”,

“name”: “default”,

“classes”: [

{

“class”: “umbrella”,

“score”: 0.634,

“type_hierarchy”: “/shelter/canopy/umbrella”

},

{

“class”: “canopy”,

“score”: 0.634

},

{

“class”: “shelter”,

“score”: 0.634

},

{

“class”: “disco”,

“score”: 0.604,

“type_hierarchy”: “/indoors/disco”

},

{

“class”: “indoors”,

“score”: 0.662

},

{

“class”: “alizarine red color”,

“score”: 0.973

}

]

}

],

“image”: “5002074328001.jpg”

}

],

“images_processed”: 1,

“custom_classes”: 0

}

Sky

Some of the photos were classified by the Watson Visual Recognition pre-trained models as containing the sky. However, this did not result in nearly enough images to complete this portion of the mosaic. There simply aren’t a lot of images in the library of just the sky. What images there are of the sky are part of photos taken where the sky is the background or the sky is a large portion of the photo such as above a building. If these images were used as tiles the sky would not look like a sky.

Therefore, we batch-cropped many of the images to remove the bottom half of the image. Images such as this:

Sky

And the top half cropped by 50% resulted in an image like this which Watson was able to classify as the sky using the pre-trained models.

Sky

curl -X POST -F “images_file=@_CR15472C.JPG-cropped.jpg” –max-time 30 “https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key=<<YOUR-API-KEY-HERE>>&version=2018-03-19&threshold=0.3”

And the JSON Response:


{

“images”: [

{

“classifiers”: [

{

“classifier_id”: “default”,

“name”: “default”,

“classes”: [

{

“class”: “sky”,

“score”: 0.907

},

{

“class”: “troposphere”,

“score”: 0.677

},

{

“class”: “nature”,

“score”: 0.798

},

{

“class”: “azure color”,

“score”: 1

}

]

}

],

“image”: “_CR15472C.JPG-cropped.jpg”

}

],

“images_processed”: 1,

“custom_classes”: 0

}

Other Themes:

Building the custom classifiers was aided by the fact that some of the images contained EXIF data. A perl script was written to scan this EXIF data and categorize these images based on the key words that we were looking for (or not looking for!). This allowed us to quickly find enough images to build the custom classifier.

#!/usr/bin/perl

use Image::ExifTool;

my $dir = $ARGV[0];

my $dest = $ARGV[1];

my $keyword = $ARGV[2];

unless($keyword) {

die(“usage: perl exif.pl INDIR OUTDIR KEYWORD”);

}

@files = `find $dir -name “*.jpg”`;

 

for my $file (@files) {

chomp($file);

print $file . “\n”;

# Get Exif information from image file

my $exifTool = new Image::ExifTool;

my $info = $exifTool->ImageInfo($file);

 

# Print all key/value pairs

my $desc = $info->{‘Description’};

 

if ($desc =~ m%${keyword}%gsio) {

system(“mv $dir/$file $dest/$file”);

}

}

Like the Sky, the white roof and the green roof below were also a problem area with just too few images to tile it based on a theme. Therefore, black and white images were used for the white roof and color historical photos were used for the green roof bordering the top of Centre Court.

Tile Selections:

With such a large pool of imagery to work from, many different photos could work in each tile location. However, once tiled some tile contents would look better than others if the objects in those tiles were larger. Therefore, each tile needed to be prioritized.

For the fan images, ballkids, chair umpires, historical, roof and unclassified themes larger faces were preferred because larger faces would show up better in the finished mosaic. For the backstop images, smaller faces were preferred because this would help the backstop appear more as a solid color. For the sky, clubhouse and other Watson-classified images the Watson confidence level in relation to the theme determined priority.

 

Conclusion:

In this project, 300,000 photos of varying quality were categorized and ranked. These photos were then used as themed input into the mosaic. This would be a monumental manual effort to otherwise do.

The Watson Visual Recognition API is an incredible tool to classify and categorize large numbers of photos. The flexibility of the tool, the pre-trained models and the ability to train your own provides a tremendous resource in quickly and accurately grouping photos.

Be the first to hear about news, product updates, and innovation from IBM Cloud