September 8, 2016 | Written by: David Provan
Categorized: Events | Watson
Share this post:
If you come to the US Open this year, things are different. Not a little different a lot different. The United States Tennis Association (USTA) has moved the southern courts to allow better walkways between courts, built a brand new Grandstand Stadium and placed a new retractable roof on Arthur Ashe Stadium.
“The USTA has always strived to move forward as an organization, to innovate and to think boldly, and the transformation of the USTA Billie Jean Tennis Center falls directly in line with that goal,” said USTA Chairman of the Board and President, Katrina Adams.
Innovation at the US Open is by no means limited to the physical grounds. The event has also seen a great deal of technology innovation as well. IBM has partnered with the USTA since 1990: the first official website was launched in 1995 and IBM delivered the first iPhone app in 2009, with augmented reality the following year. Our relationship with USTA supports the organization’s goals to, as stated above, “innovate and think boldly.”
New for 2016, IBM has introduced several cognitive solutions using Watson services on the IBM Bluemix platform. For this blog, we will discuss two of the solutions we built using Watson.
Watson Visual Recognition “auto detects” players
Each day the US Open photographic editorial team process approximately 700 images, the IBM system allows them to “auto detect” players by using the Watson Visual Recognition API. The photo is processed by API and results are received. Our Liberty application, connected to Cloudant, then checks the result and runs a fuzzy search against our player list to map the name of a given players ATP or WTA tour identifier. The API then returns the name along with the tour identifier if it’s a recognized player. For people we identify who are not tennis players, we simply return their name to assist the content team in captioning and titling photos.
For the visual recognition application, our development cycle used a IBM GitHub repository connected to the IBM Bluemix Pipeline part of DevOps. It is set up to listen on the master branch only, and then build and deploy the application into our Liberty container on Bluemix. Local developers work on a develop branch and when it’s time to release the product, merge develop into master. We set our Bluemix project up using the getting started guide which will connect your GitHub project into the Bluemix DevOps services. The pipeline configuration is all click-to-configure and operates expediently. We often commit, and by the time we open the pipeline page, the build is halfway done in deploying to Liberty.
Our RESTful interface is developed using JAX-RS and using the standard annotations in support of that.
Watson Speech to Text for subtitled videos on demand
For 2016, the USTA asked IBM to enable subtitles to all video on demand content produced during the US Open. With over 600 videos produced during the tournament, we work with USTA to define a way to achieve subtitles that was not labor intensive.
The designed solution utilized the IBM Watson Speech to Text API endpoint. As each video is processed through the IBM Cloud Video Platform, an associated audio file is produced. Our Content Management System ingests this content and requests out to the Speech to Text Service to provide automated subtitles. We then store these returned captions and allow the user to make edits for readability, punctuation and times when audio quality may have prevented strong subtitle confidence.
Once reviewed, the captions are pushed out to the web infrastructure and delivered to all digital platforms for consumption.
Helping the model learn
The final piece of the solution allowed us to train the speech-to-text model to improve accuracy. This was achieved with a Python run time in Bluemix passing data through several Watson and external services.
A process of domain adaptation for tennis allowed us to train several speech to text language models to improve Word Recognition Rate (WRR) while reducing Word Error Rate (WER). Five Docker containers were deployed in Bluemix to run Python applications that generated different versions of tennis corpora for language model training. Several Watson services, machine learning and Natural Language Processing (NLP) packages were combined together to produce highly accurate video on demand subtitles.
Many different types of content such as news, interviews, dictionaries and corrected subtitle files are crawled, processed and stored into DashDB on Bluemix. The Watson Language Translation Service filtered any content that was not delivered in English. All of the English content was spellchecked to autocorrect word errors. Next, tennis glossaries and dictionaries were ingested into the Watson Retrieve and Rank through the Watson Document Conversion service to measure the relevancy of nouns and verbs that were expanded to include hyponyms and hypernyms. The relevancy score and other NLP measures from the Python Natural Language Toolkit was input into a custom trained machine learning model, Logistic Regression, to help improve the precision of the language model. The words that were relevant for tennis were included within the expanded corpora. Finally, the Watson Speech to Text Service API trains a new language model with the expanded corpora.
As a result of this work, Watson is now trained for tennis. It is now familiar with terms associated with the US Open, e.g. Arthur Ashe is correct instead of “arthur ash.”
The online language model learning has led to a 5% improvement in Waton’s confidence for speech to text capability. It’s exciting—as a new model is trained everyday, subtitle accuracy is improving.
A major sporting event is a complex environment in which to operate. Changes are based on human elements on court and we need to be able to react and respond to events in near real-time. The DevOps pipeline in Bluemix is enabling us to release real time updates to our application in a repeatable and safe manner; the API end points for Watson are helping us improve our language capabilities in real-time.
These innovations have created production efficiencies for the client which allows them to put more focus on strategic areas. What’s next? Bluemix Services will continue to address and provide production efficiencies and innovation.
Put simply, Bluemix allowed us to react at the pace we need to support the US Open—to be bold and to innovate.
(*) Our thanks to Brian Adams from IBM iX and Aaron Baughman (@BaughmanAaron) who were integral to the build.