A conversation between Jerry Cuomo and Don Scott, Director of Engineering at Submergence Group and the mastermind behind the Mayflower Autonomous Ship, on automation at sea.

Covered in this chapter

• The coolest job in tech
• The “AI Captain”
• Rules and regulations at sea
• From the ocean to the enterprise

The Art of Automation: Table of Contents

Full steam ahead

This chapter of the Art of Automation is a reduced transcript of a conversation between Jerry Cuomo and Don Scott, Director of Engineering at Submergence Group and the mastermind behind the Mayflower Autonomous Ship. In this special discussion on perhaps the most famous IBM engagement since Jeopardy!, Don and Jerry discuss the role AI-powered Automation is playing in the Mayflower Autonomous Ship project, which is attempting to become the first entirely automated vessel to traverse the Atlantic Ocean. They elaborate on the “AI Captain,” safety, trust and why AI explainability is so important. Jerry and Don close by exploring how lessons learned from automation at sea can be applied back to an enterprise setting.

Jerry and Don

CUOMO: Welcome to the Art of Automation, a podcast that explores the application of automation in the enterprise. Today, we have an exciting topic that, on its surface, may seem a bit out of the boundaries of classic enterprise. This episode will explore a unique application of automation, which is to traverse the ocean in a quest for data and discovery. As you hear the story unfold, I’m sure you will identify with as many similarities as differences between automation at sea and automating your enterprise. Trust me, you’ll see this.

And yes, I’m talking about the Mayflower Autonomous Ship (or, MAS for short). MAS is a grassroots initiative led by marine research nonprofit ProMare, with support from IBM. Their mission is to provide a flexible, cost-effective and safe option for gathering critical data about the ocean. MAS can spend long durations at sea carrying scientific equipment and making its own decisions about how to optimize its routes and mission. With no human captain on board, MAS uses the power of, yes, AI-powered Automation to traverse the ocean in a quest for data and discovery.

So, the ship’s AI captain performs similar roles to a human captain — assimilating data from a number of sources, it’s constantly assessing its route, status and mission, and it makes decisions about what to do next. Cameras and computer vision system scan the horizon for hazards and streams of meteorological data reveal potentially dangerous storms. Machine learning and automation software ensure that decisions are safe and in line with collisions, regulations.

So, for today’s episode, we have the good fortune to have Don Scott with us. Don is the director of engineering at Submergence Group, and we’ll discuss automation in the context of this masterpiece of modern automation technology, the AI captain. Welcome, Don, to the Art of Automation.

SCOTT: It’s great to see you again, Jerry. Happy to be here.

The coolest job in tech

CUOMO: All right, let’s just jump into the first question, if you don’t mind. Don, as I alluded to in the beginning, you have a pretty unique job. Can you broadly share what you do at ProMare and the Submergence Group?

SCOTT: Oh, sure. I’ve worked my entire career as an ocean engineer. My focus has been in the navigation and control systems for manned and unmanned submarines for the U.S. and the U.K. navies. But for the last 15 years, I and my team have been developing marine systems with different levels of autonomy — so, being able to operate unmanned.

CUOMO: So, Don, first of all, I think everyone out there is saying: “how cool is that?”  

Is this software, hardware, all of the above? That plus more? Can you tell us a little bit about the tools of your trade?

SCOTT: So, ProMare and Marine AI and Submergence Group, we kind of do everything. My focus is on the software aspect. It’s on the sort of intelligent decision-making about where the boat should go and how it should operate — things like that.

CUOMO: Could you share a little bit of your favorites from a software tool perspective? Are you a C++, Java? What’s your forte?

SCOTT: I started writing in ASSEMBLY, you know. That’s certainly not my favorite, I’ll tell you that. We’re actually a big sort of C# house, so we’re kind of focused on in terms of our tools. You know, the younger guys sort of favor Python — not my thing — but that seems to be where everything’s going.

The “AI Captain”

CUOMO: Let’s dive into that. You said autonomous operations, and that’s kind of like the highest level of achievement by an automated system. And your AI captain uses technology like cameras, computer vision, AI and rules and edge computing all as a means to safely navigate around other ships, buoys and other ocean hazards during this transatlantic voyage. So, tell us a little bit, if you can, about how you put these technologies into play to achieve this autonomous automation; and, what did you learn while applying this technology?

SCOTT: So, the traditional role of the sea captain is the safe and efficient operation of a vessel at sea, and that’s exactly the capability that we need to implement for true autonomous capability. Mayflower, or the AI captain on Mayflower, you know, needs to evaluate the current situation and recommend a safe course and speed.

To do this, we implemented — call it a “hybrid AI solution.” You know, it’s a classic design architecture for autonomous systems. We’re not reinventing the wheel here. You have sensing, computing and then acting.  And what we’ve learned is the sensing is the real key, right? Like changing visual and physical information into something a computer algorithm can understand.

You know, we’re looking at video streams from six different cameras using computer vision capabilities there; looking at AIS, which is a standard system for identifying ships at sea; radar, for looking around you; a fathometer for knowing water depth; and, weather reporting; and importantly, chart information, too. We use all this information — we put this into what’s called like a data fusion layer where we, basically, pull all this information together to create what we call a hazard map. So, it just sort of defines the safe and unsafe regions at sea.

CUOMO: Very interesting, Don. And is this flexible? Does it learn over time? For example, I read something almost humorous a few weeks ago before the launch of the Mayflower that there were some fears about, uh‑oh, what happens if a seagull swoops in? Would the introduction of a seagull just kind of popping in and then popping out affect the algorithm, is that something that you’re planning for or you can add on the fly later?

SCOTT: Yes. I’m not sure where the seagull thing came in, but we’re not too focused on identifying different types of birds. We were more focused on identifying different types of ships and what we call LOMOs — so, Low Observable Marine Objects — so lobster pots and basically flotsam that could be a hazard to navigation.

So, in terms of adaptability and learning — when we are doing the transit, we have a fixed set of inference models for the computer vision, but we’ve always retained the ability to push new models out to the ship as required.

We’re not doing any retraining at sea. We don’t we don’t want to be introducing potential new behavior while during a transit, so we sort of a fixed set of inference models when we’re operating at sea.

Rules and regulations at sea

CUOMO: That makes sense, Don. And you had once asked me if I’ve ever talked to a marine regulator. So, I guess when you get into safety and what you said at the top about safe navigation, and when you asked me that, I don’t even think I answered because…that sort of question that I would never expect a person to ask me.  No, I’ve never talked to a marine regulator.

SCOTT: I thought everybody talked to marine regulators…

CUOMO: [CHUCKLING] No, no, I’m sorry. Sorry, sorry, not this person here.

But then you went on in a very interesting way to explain about international regulations for preventing collisions at sea and how our rules, language and AI explainability, in general, establish a level of understanding, trust, increased confidence from these regulators. Can you share a little bit more detail for our audience here about how that works?

SCOTT: Yes. It’s a pretty complex issue, to be honest. We need to start by understanding the environment that we’re working in. Since we began sailing in the oceans, you know, back when the Egyptians first had the first sailing boats with reeds, there were an understanding of how these ships would interact when they came across each other.

There’s a fairly large organization called the IMO — the International Maritime Organization — which is a big international regulatory agency that’s responsible for overseeing quite a bit about the ocean. And one of the things they look after is like the safety of navigation at sea.

And they’ve created these conventions and rules and collision regulations, or COLREGs, that we like to call them. And like I said, these are the COLREGs, they dictate how ships interact at sea. It becomes a very complex situation, because COLREGs only address single ship interactions — how two ships deal with each other —  but anyone who’s been to sea will know there’s usually lots of ships around, like especially in harbors and things like that. So, it becomes a bit of a dance, right, on how ships interact.

CUOMO: And is it fair to say that marine regulators can’t read code — Python or C#.

SCOTT: Yes, absolutely. I’ll certainly address that about how is it that we make these rules understandable to the regulator, and because the COLREGs, they can be somewhat ambiguous, right?

Like there’s my favorite term is something called “passed and clear,” where you know, when you’re overtaking another vessel, you’re not supposed to go back to your original course until you’re, quote, “passed and clear of the previous vessel.” What’s “passed and clear”? It’s when the other captain feels safe, right? So, when is that?

So, this brings up this whole issue of trust the decision that an AI captain’s making. And it’s a pretty significant hurdle, right, because they want to be able to not just understand the decision that’s been made but also why the decision’s been made, like the rationale behind it as well.

And that’s the strength of using, for our solution, it’s a deterministic rules set. We use something called ODM — Operational Decision Manager.

CUOMO: Yes, I know it well.

SCOTT: You know it well, right? And you know, because of its pedigree in the financial services industry, it provides a completely transparent and immutable record of the decision-making process. And it’s this transparency and explainability that is required to develop trust. This is a trust that’s required for the regulatory boards to understand why we’re making these decisions.

CUOMO: Do you literally show the regulators the rules, as written, actually makes sense to the regulators?

SCOTT: Well, we haven’t had to yet, but we certainly will. We certainly are able to. You know, that’s kind of the power of this approach.

CUOMO: Wonderful. Yes, very interesting, very cool.

From the ocean to the enterprise

CUOMO: Don, can you help us jump the tracks on how lessons learned from your experience building the AI captain to autonomous operations in an enterprise setting?

SCOTT: I guess first off, it probably would have been easier in an enterprise setting itself.

CUOMO: Yes.

[ LAUGHTER ]

SCOTT: But we decided to do this at sea.  So, that has its own sort of difficulties or its own challenges, let’s say, than working sort of just on a bench or whatever.

I guess one of the big things we did is we needed to containerize a lot of these different apps that we’re running that make up the AI captain. We kind of have an expectation that systems are going to fail, and you can’t have a critical system like COLREGs manager — which is what we call it — on a single point of failure, you need to be able to move it around.

And you know, we needed to do that on the first transit attempt when we were in the recovery mode. You know, we had a system failure, but because of the system architecture and the flexibility we were able to move these critical systems on to working systems and keep the ship operational and the way it worked. This idea is that, you know, you can also push hot fixes up to the boat even while it’s operational, which is a key element as well.

CUOMO: Yes, so resiliency in your enterprise, can learn a lot from resilience while making a transatlantic voyage. Very interesting.

SCOTT: One thing I would mention, though, is I think one of the real things that we learned, and I think this applies to not just the marine domain but also your enterprise domain, is that we needed to be able to manage the division between the expert knowledge of our master mariners and the programming of the rule set itself. You touched on that a little bit earlier.

We needed to be able to really effectively translate what at times seemed really ambiguous rules coming from the master mariner; and once they’ve been written into the rule set, be able to sort of mirror them back to the master mariner in a way they can understand, so they can say, yes, that’s exactly what I meant, and understand that that’s actually the rule that was going to get applied.

CUOMO: Well, Don, I have to say not only is this unique factor very high, its cool factor is off the charts, but also its applicability.  If you squint your eye and tilt your head as you’ve been describing, it’s quite applicable to a very broad set of industries.  And I’m sure you and your team are quite proud of this accomplishment, and so are we.  It’s so impressive and inspiring. Thank you very much, Don, for joining us today.

SCOTT: Thanks, Jerry. I really enjoyed talking with you.

Learn more

Make sure you check out The Art of Automation podcast, especially Episode 18, from which this chapter came.

The Art of Automation: Landing Page

• Foreword: The Business of Automation
• Chapter 1: Introduction to the Art of Automation
• Chapter 2: Automation with Robotic Process Automation (RPA)
• Chapter 3: Automation with Intelligent Document Processing
• Chapter 4: Automation and APIs
• Chapter 5: Automation with AIOps
• Chapter 6: Automation in Healthcare
• Chapter 7: Automation in Insurance
• Chapter 8: Automation and the Weather
• Chapter 9: Automation at Sea
• Chapter 10: Automation in Retail
• Chapter 11: Automation and Process Mining
• Chapter 12: Automation of Yesterday, Today and Tomorrow
• Chapter 13: Automation and Observability
• Chapter 14: Automation and Digital Employees
• Chapter 15: Automation in Financial Services
• Prependix: The Art Behind the Art of Automation