Your personal assistant on the Web

Share this post:

Having a newfound dependency on the mobile web, internet users are increasingly looking for shortcuts and the easiest way to get things done. For example, our GPS devices automatically know where we are – providing us with driving directions instantly or the location of the nearest hamburger joint quickly – without having to think much about it. And now, IBM Research has taken web automation technologies one step further, with IBM CoScripter Conversational Interface: an Intelligent Assistant for interacting with the Web, nicknamed CoCo for this blog entry.

This new tool originated from CoScripter, a system for recording and automating repetitive activities on the web, such as paying a monthly credit card bill, requesting a vacation hold for postal mail, or checking flight arrival times. With CoCo, you can do all of this from your mobile device.

CoCo brings together web automation technologies – CoScripter and Highlight (http://www.almaden.ibm.com/cs/disciplines/user/#highlight) – making the web available to the user’s command through any simple text messaging interface. With this, users can, for example, send an SMS to CoCo and ask it to do something on the web, without actually being in front of a computer.

How does it work?

From a very high-level, the flow as seen in the picture is the following:

  • The user sends a message to CoCo via SMS, via e-mail or any other text messaging system
  • CoCo takes the message and looks through this user’s CoScripter scripts for a match
  • When CoCo finds the script, it adds the necessary parameters from the user’s command and then ships it for execution in a headless browser, running on Highlight
  • Highlight executes the user’s CoScript and returns the relevant portions of the Web site navigated
  • CoCo sends the result back to the user via SMS, e-mail, etc.

Whenever there is a need for clarification or confirmation, CoCo can dialogue with the user to ask for additional information. In this way, CoCo becomes a communicative assistant with which you can talk to get computing tasks done.

Watch the video demonstration

IBM researchers Tessa Lau and Julian Ariel Cerruti presented this paper, “A Conversational Interface to Web Automation,” at the 23rd ACM UIST Symposium in New York City on October 5. Co-authors are Guillermo Manzato, Mateo Bengualid, Jeffrey Bigham, and Jeffrey Nichols.

More stories

A new supercomputing-powered weather model may ready us for Exascale

In the U.S. alone, extreme weather caused some 297 deaths and $53.5 billion in economic damage in 2016. Globally, natural disasters caused $175 billion in damage. It’s essential for governments, business and people to receive advance warning of wild weather in order to minimize its impact, yet today the information we get is limited. Current […]

Continue reading

DREAM Challenge results: Can machine learning help improve accuracy in breast cancer screening?

        Breast Cancer is the most common cancer in women. It is estimated that one out of eight women will be diagnosed with breast cancer in their lifetime. The good news is that 99 percent of women whose breast cancer was detected early (stage 1 or 0) survive beyond five years after […]

Continue reading

Computational Neuroscience

New Issue of the IBM Journal of Research and Development   Understanding the brain’s dynamics is of central importance to neuroscience. Our ability to observe, model, and infer from neuroscientific data the principles and mechanisms of brain dynamics determines our ability to understand the brain’s unusual cognitive and behavioral capabilities. Our guest editors, James Kozloski, […]

Continue reading