A high-tech solution for a very human need

The COVID-19 pandemic has been hard on everyone, but for school-age kids whose social skills are still developing, and whose need for peer interaction ranks close to their need for food and oxygen, it’s been particularly challenging.

For Sarah Hamilton, a nine-year-old student at Wallscourt Farm Academy in Bristol, UK, one of the hardest parts of remote learning was that it meant no choir. Her school boasts a 400-member student choir, and singing with that group is a highlight of Sarah’s school day. Members of the choir even wrote a song they planned to sing for the school’s Harvest Festival, which normally takes place in September.

Unfortunately, COVID-19 meant that children were learning remotely from home, then later in “bubbles,” or small clusters at school, not able to gather in choir-sized groups.

Fortunately for Sarah and her classmates, her father is Matt Hamilton, a cloud developer advocate for IBM in London.

Matt Hamilton portrait

Matt Hamilton, co-creator of Choirless

Spurred by Call for Code

“The school choir has a fantastic song they wrote themselves, a quite powerful song about resilience,” said Hamilton, adding that he felt bad they weren’t able to perform together, and wondered if there was something he could do to help. “It was just one of those random ideas that comes out of nowhere” — and it happened to coincide with the 2020 Call for Code competition.

“The IBM Hursley Labs ran a one-day event for people to brainstorm ideas for Call for Code. I came up with the idea about a week before that event, and literally the day before the event put out a call on Slack saying ‘Who’s interested in this?’”

Glynn Bird and Sean Tracey were intrigued. “One of the enduring threads of my career has been creating technology that lets people do things in the real world,” said Tracey. “So when Matt came up with this idea — for something that lets people use technology to better their lives, to get out of the malaise and melancholy of the COVID situation, I loved it. It struck me as something that was so not what they were expecting for Call for Code, but that should absolutely be done, because it definitely fills a need people have.”

Also a cloud developer advocate, Tracey had met Hamilton a couple of times. Neither have met Bird in person, but the three clicked immediately online and got right to work.

IBMers gather to sing "coast to coast"
Play Icon

IBMaple, a virtual band comprised of IBM Canada employees

Music for mental health

“Because I’m a member of a community choir in real life, it kind of piqued my interest,” said Bird. “The demographic of our choir is relatively old, between 50 and 80. It’s the high-risk group for COVID. And there have been reports in the press that singing is a real super-spreader thing. You know, people belting out and expelling air into the faces of other people in an indoor setting for two or three hours is extremely high risk.

“The last time our choir met was in March, because of the risk, but for some of these people it’s the high point of their week. There are 150 people in our choir; it’s a big social meeting place. It’s not important to the economy or anything, but it’s great for people’s health, it’s important for well-being.

“And it turned out to be a winner, or a near-winner, anyway,” he added. Choirless took second place in the internal IBM Call for Code competition, out of entries submitted by 6,500 IBM employees.

IBMers sing "Yellow Submarine"
Play Icon

Choirless Call for Code IBM Submission Video

The right mix of skills

By a great stroke of luck, each member of the trio brought a unique set of skills to the project. “I’ve been involved mainly in the algorithm side,” explained Hamilton. “Glynn’s worked a lot on the database side and he’s a very skilled musician. He plays piano and bass guitar and sings, so he’s done a lot of the testing. Sean has done all of the front-end user interface that you see and interact with.”

“It just turned out that we happened to be three legs on a stool, as it were,” said Bird. “We came up with the right combination of skills, by accident really. We didn’t recruit or interview for it, it just happened.”

Choirless, the product of the team’s effort, is an AI-fueled app that lets people once again enjoy singing as part of a group.

Why not just get everybody on Zoom or WebEx and let them sing? It’s not that easy, thanks to something called latency. Latency is that tiny, or sometimes not-so-tiny, time lag that happens when people in different locations participate in a group meeting. It’s a minor annoyance in conversation, but a complete disaster when people are trying to sing together. “Even a few hundred milliseconds of latency can completely throw things off,” said Hamilton.

Professional musicians who have been recording from home during the pandemic have the benefit of expensive technology and professionals who know how to operate it — advantages that are out of reach for most amateurs.

Choirless overcomes the latency problem, resulting in a nearly flawless synchronization of voices and instruments. Using Watson Studio, Hamilton built a synchronization algorithm that uses machine learning to track fluctuations in the recordings being merged.

“It looks at a number of different features in the recordings," he explained. “It looks at the spectral flux, which is the rate of change of frequency over time; it looks at the crest factor, which is the rate of change of volume over time; and the chroma, which is actually the notes being played, and it tests all the different possible combinations. It splits the song into five chunks and synchronizes those five chunks independently. It tries to find the point at which they synchronize the best.

If a recording is slightly out of time with the lead, Choirless cuts a fraction of a second off the front end, ensuring synchronization without speed shifting or auto-tuning.

“This technology was originally developed by NASA, and currently controls most of the intercontinental ballistic missiles in the world. So this is, actually, rocket science.”

Though technologically complex, Choirless is easy to use. The group leader logs into the app and records the reference track, then invites other group members to participate. Each member logs into the app on their own time and sings or plays along with the master track. “You effectively do a duet with that one piece,” said Hamilton. When a new recording is uploaded, Choirless integrates it seamlessly into the original version, and within six minutes renders a new version of the recording, with the new addition included.

A group of performers work together using chiorless
Play Icon

Amazing Grace, performed by members of City Church Bradley Stoke

Choirless feeds the need to create

The Choirless team actually went well above and beyond the requirements of Call for Code. “I think you’re supposed to come up with some words on what you’re going to do, and maybe a proof of concept,” mused Bird. “We just kind of kept going until it was a fully formed thing. We couldn’t stop ourselves, really.”

“Like Glynn said, we just have a need to keep making things, so we kept making Choirless,” laughed Tracey.

The team is also finding Choirless an effective tool in their day jobs. “My role as an advocate is to help people understand and build things with IBM technologies and open source technologies,” said Tracey. “I’ve been trying to push for a project-focused way of doing advocacy, showing people what can be done with a combination of these things. Choirless ties into this quite nicely.”

Hamilton creates a weekly Twitch video for the developer community. “On the Twitch today, I was talking about a particular tool we’re using in Choirless for optimizing machine learning modules. I’m actually able to use Choirless as material for the teaching stuff I do on Twitch.”

To date, the app has been used by musicians of all kinds and abilities. “Once people get it — and it doesn’t take long to get it — they adore it and they keep coming back,” said Tracey. “So we’ll see the same people popping up performing new videos, but this time with their dog, or their kid or their friends, which is wonderful to see.”

When queries began coming in from large bands wanting to use the app, Tracey took it upon himself to test the limits of the system with his own version of a stress test. “I play the bagpipes and mandolin, much to the chagrin of my neighbors,” he said. “My father insisted that we learn the ways of Scotland, being a Scot himself, and so I can make a mean haggis, I can play a mean bagpipe, and I can drink whisky like it’s water.”

Tracey recorded six sessions of himself playing the bagpipes. Hamilton duplicated those sessions with slight random delays, creating 300 unique recordings to test the algorithm and rendering pipeline. Choirless synced them all up and successfully produced one only slightly daunting recording.

 

300 recordings of bagpipes
Play Icon

The sound of 300 synchronized bagpipes

A big role for open source technology

Choirless runs on the IBM Cloud, and wouldn’t exist without access to open source technology. “We stand on the shoulders of giants,” said Tracey. “The technology that Choirless is built on is enabled by these massive, massive projects, from the Cloud Foundry project, which allows us to spin off apps very quickly, to open source frameworks that we use to build application servers and API interfaces.”

“We couldn’t have built Choirless without open source software,” agreed Hamilton. “The main guts of it, the part that does the compositing of the videos, is all open source. We couldn’t have built that in the time we had.”

All three team members are proponents of open source, seeing it as critical for compulsive creators like themselves. Glynn said that regarding Choirless, “we were determined to put every line of code out there, so everyone could use it themselves as a learning opportunity and hopefully contribute stuff themselves. We’re very happy to have contributions from the public if they find a better way to do things.”

“The algorithm I’ve written does pretty well, but it doesn’t always get it right,” said Hamilton. “We’re looking at maybe running a competition and putting it out to the public to see if someone else can come up with a better algorithm. I’m not a musician. Somebody who’s written a paper on recognizing music using AI, for example, might approach it in a completely different way than I have.”

The team is already seeing Choirless used in ways that go beyond their initial goals. “We started off with choirs being the main use case, because of COVID,” said Bird, “but it really is, in general terms, a music collaboration platform. I use it as a recording device. Obviously, I can’t play piano and guitar and whatever at the same time, but I can record them separately and layer them on top of each other.”

“We’ve had people who wanted to sing Happy Birthday to their grandmother on the other side of the world, and they couldn’t all get together, so they sang Happy Birthday to her on Choirless,” said Hamilton.

“There’s no culture in the world that doesn’t have a musical tradition,” added Tracey. “This is an app that enables people to share that, not just with people they know, but with people they don’t know. Beyond COVID, it’s very possible that a choir could exist that has members in Ottawa, Canada, and Sydney, Australia, who meet on a regular basis because this is how you do things now. It’s a whole new platform.”

Stephen Howe and Matt Hamilton speak over web video
Play Icon

IBM Radio interview with Matt Hamilton