Who needs connectors when we have Web APIs
gcuomo 060000X9CG Visits (12373)
Who needs connectors when we have Web APIs
by Robert Vila and Jerry Cuomo - November 2014
USB and Web APIs have something in common
If you’re like me, you remember having a drawer filled with different types of hardware connectors and adapters. Your keyboard, mouse, network adapter, phone and dozens of other peripherals had their own proprietary type of connectors. Each device was designed for a unique type of socket and worked with only one protocol. It was a mess. And then came USB. Suddenly, that drawer became much cleaner. By embracing a more modern method of connecting devices, we opened up a new world of possibilities. One socket could support any type of device. Devices could communicate with a host and with each other. A common protocol allowed asynchronous, chatty communication for a better user experience. It’s now common for homes to have wall outlets that contain not only traditional power outlets, but USB connectors as well. We would never have gotten this far if we didn’t take advantage of the capabilities that modern connectors provide.
In the world of software, we’ve struggled with the equivalent of a drawer of outdated cables and adapters. Proprietary data formats, protocols, and connectivity methods kept our data and services in silos.
The software industry is finally taking the next step. Web APIs have become the software equivalent of the USB connector. We now use REST and HTTP as a simple communication protocol and JSON to efficiently exchange data. These are the foundation for accessing data in a modern way that is free from the mess imposed by limitations of past decades of computing. By embracing this style of connectivity, we can now package, share, and mash up data in ways we could never envision in a world of silos.
Connectors Built for a Bygone Era
These assumptions are no longer true, but most software connectors still act as if these limitations exist today.
A good example is the SWIFT data format. SWIFT is used by banks to send payment orders between institutions in large batches of files. Members use the proprietary SWIFTNet network to send only valid, well-formed messages. In fact, banks are charged a penalty for every incorrectly formatted document.
To parse a SWIFT document, a developer typically reads SWIFT data from a file, uses a data model to validate the document, creates the entire object in memory, and then operates on the data. The first SWIFT message was sent in 1977 and the format was well suited for the era in which it was created. However, if you were designing a method to exchange bank data today would you create a format that is serialized to a binary file? Would you expect bank transactions to occur overnight on a batch or in real-time? Would you build your own proprietary network or take advantage of ubiquity of the Internet?
Ghosts of Connectors Past
It’s easy to pick on a format that was created more than 30 years ago. But are more recent connectors still showing these limitations? SalesForce was built to bring CRM to the cloud, so connectors to SalesForce should be designed for a more modern interaction style, right?
Most software connectors to even modern sources like SalesForce are built to implicitly satisfy the assumptions of the pre-Web era. The graphic on the right represents the SalesForce data model. Many connectors to SalesForce provide access to the objects such as Account, Partner and Opportunity. We can now access portions of the data, but still need to understand the SalesForce data model, which involves reading an 800 page manual. A pre-defined schemas in tools such as Cast Iron can help, but it’s still challenging to define repeatable subsets of the data model and avoid fetching much more data than you need. If you need to transform SalesForce data into another format, there is relatively little that mapping tools do to accelerate the process. In general, a user still needs to understand the SalesForce data model well to work with the data.
An API-Driven Connector Model
We can now use Web APIs to unshackle ourselves from the assumptions that held connectors back in the past. Here’s a more modern set of assumptions:
This is an entirely different philosophy of connecting to a data source. Previous connectors were built on a model of retrieving every piece of data we could possibly need in a highly structured format because we might not be able to reach that system again. Today’s connectors don’t need to be lectured. They can have a conversation.
We don’t need to preemptively gather all the data we might ever need. Instead, we ask for what we need when we need it. If we need more data, it’s just a REST call away.
Rigidly defined data formats still have a place in this world. Systems of Record that operate on financial transactions need to rely on secure exchange of information that can be validated and audited. But not all systems need this level of rigidity. There’s an explosion of unstructured data that we can now leverage. JSON gives us a common way to exchange data and intuit the structure as we process it.
A well-defined Web API can both openly and flexibility provide the capability of most software connectors of yesterday. A good API lets you access only the data you need without exposing the complexity underneath. For example, if we want to access a customer’s name, we shouldn’t have to retrieve a large data object that carries extra baggage such their address, previous orders, and campaigns that have worked on them. We shouldn’t need to know the intricacies of how that data is stored in a back-end system. If we need more data, it should be easy to use our context to get the additional information.
Exploring A Modern Financial API
Let’s explore how financial transactions are handled in an environment created for a more modern era. The PayPal API allows a developer to send and receive payments using simple Web APIs. Here’s an example of how to create a payment request using the PayPal API:
With a single API call, we’ve initiated a real-time financial transaction that is dramatically simpler than exchanging and validating a SWIFT document. The response object from this transaction gives us information such as the transaction ID that we might need for later usage.
How about if we need to make an adjustment to the payment we’ve created to add a shipping address? In the old model of connectors, we would have to receive an entire Payment object, parse it, create a transformation map to add in the new information, then update the source with an entirely new object. In the world of Web APIs, this is all we need:
With a little JSON and a single Web API call, we’ve accomplished what would have taken dramatically longer in the old world. Today’s software connectors are built on these philosophies and take advantage of the conversational and asynchronous properties of modern Web-based systems.
Do We Still Need Connectors At All?
If Web APIs solve so many problems, then do we still need a software connector at all? Perhaps. But in the web world, connectors now become an intermediary that we call an API proxy.
An API proxy is an abstraction layer between the application and a service end-point. It provides the ability to target multiple end-points, improved reusability, asynchronous capabilities for mobile, and support enterprise transactionality.
For developers, an API proxy can simplify the process of using an API. We demonstrated how simple it is to use the PayPal API to add payment capabilities to our app, but what if our customers now demand that we add support for ApplePay, Google Wallet and more? We would need to learn each of those vendor APIs and manage multiple security keys and transformation maps. However, if our API Proxy provides a simple and generic API for payments that abstracts away the details of the specific provider API, we don’t need to learn new APIs and manage their keys. An API proxy provides the same abstraction layer that is the foundation of well-written object oriented or functional code.
Here is a simple example of an API proxy created in Node-Red, which provides a simple Payment API abstraction providing a single payment interface to Google, Apple and PayPal. The flow detects the desired payment network, adjusts the JSON payload to the correct fields and layout for the corresponding network, then makes a REST call to either Google, Apple or Paypal.
On mobile devices, connectivity is not always reliable and bandwidth is at a premium. An API proxy can add support for asynchronous Web API calls to provide a great user experience even when the network conditions are poor. Firebase.com is a great example of this capability. Firebase is a web-based data store that supports WebSockets for asynchronous interactions. Operations are performed via API calls that operate on a live server when connectivity is suitable or on a local database when it is not available. The database is transparently synchronized with backend systems when connectivity improves. The end user is oblivious of this – as they should be. A tightly bound connector or basic Web API call will not provide this level of flexibility in mobile environments.
High-end software connectors provide support for transactions. While I would not recommend implementing advanced transaction properties atop RESTful APIs, there are many examples of support for simple transaction capability on top of REST APIs. For example, consider a RESTful shopping basket scenario. The shopping basket is conceptually your transaction basket. In the same way that you can add multiple items to a shopping basket and then submit that basket to process the order, you can add entries to the transaction basket. When all the pieces are in place you can POST/PUT the transaction basket with all the entries.
Web APIs defined in REST and/or WebSockets can provide assured transactionality even when built on these simple protocols. As an added bonus, adding this capability does not over-complicate our interface, which is important because the vast majority of use cases don’t need that level of robustness.
Finally, implementing an API proxy can be even more valuable for users that don’t write much code but need easy access to data. Using an API-driven method of connecting to data sources allows us to expose valuable information to users who are not comfortable writing a SQL query or parsing JSON. Imagine a palette of data made available from back-end systems. This data can be mashed up from a variety of sources and wired to nodes that allow for deeper analysis and orchestration. We can offer a marketplace that contains not only Web APIs but useful pieces of functionality (script code) that is built on the combination of multiple data sources.
Node-red, mentioned above, is open source tool for wiring together hardware devices, APIs and online services in new and interesting ways. Node-RED provides a browser-based flow editor that makes it easy to wire together flows using the wide range nodes in the palette. Flows can be then deployed to the run-time in a single-click. A built-in library allows you to save useful functions, templates or flows for re-use.
In 2014, we’ve advanced our infrastructure to the point where connectivity between systems uses common protocols and flexible data formats. We’ve finally broken free from the rigidity imposed on us by unreliable networks, proprietary protocols and overly structured data. Let’s celebrate this and use it to improve how we integrate systems. Application, Data and B2B integration come together. Built on a fabric of rich, API-driven conversations between systems, we can reach more varied data sources than ever before. We can operate on that data in a modern, composable style. And much like it did with USB, standardizing on REST and Web APIs will unlock our data and services to create new use cases we could never envision alone.