Don’t Build Your Own REST API
The engineering team at DreamFactory designed and built some of the very first applications that use web services. Over the years, we made many mistakes trying to create the perfect mobile backend for these applications.
Talking to customers, we’ve learned that many companies face the very same challenges. This paper lays out some of the traps and pitfalls that companies often experience when they decide to build their own REST APIs.
Part one discusses the danger of growing REST API complexity over time. Part two looks at some of the ineffective solutions companies attempt to resolve the problem. Part three talks about the advantages of building a reusable services platform. And the last part covers some of the additional capabilities that will be required for a really effective REST API strategy.
Part One) The REST API Complexity Problem
When a company decides to start a new mobile project, they first develop the business requirements, and then start building the actual software. Usually there is a client-side team that designs the application and a server-side team that builds the backend infrastructure. These two teams must work together to develop a REST API for the project.
Meanwhile, the server team needs to identify the backend data sources and external web services that the application will need for end-user collaboration and data access. They must develop single sign-on for user management, roles and access control permissions to backend data, and a REST API that can securely expose the backend assets to end users of the application.
This methodology creates an iterative development cycle as the client application is developed. The client team specifies various new REST API services that they need, and then the server team has to build the services, after which the integration must be tested for performance and functionality. This back and forth interface negotiation consumes lots of time and money as the two teams converge on an acceptable REST API.
The Tower of Babel
Building your own REST API is not just expensive and time consuming. In the example discussed above, the client team and the server team do eventually manage to get their application working as designed. But the real trouble starts when the next mobile project enters the picture.
New mobile projects typically have new requirements that were not anticipated by the existing REST API services. You could go back and try to expand the scope of the old services, but they are already in production, and so in many situations new REST API services are created for each new project.
The modern enterprise may need hundreds or even thousands of mobile applications. And so the API building process continues over a few years with different developers, consultants, and contractors. Some of the teams are great at writing REST APIs, others not so great. Teams use different security mechanisms, credential strategies, user management systems, and API parameter names. Each service is hard-wired to specific data sources located in various clouds and data centers.
The result, as you can imagine, is a giant mess. Data access rights are confused, user management is complex, and application deployment is cumbersome. The system is difficult to manage, impossible to scale, and probably full of security holes.
The mistake here is that development activity started with business requirements and application design and then worked back to server-side data sources and software development. This is the wrong direction. It’s better to first identify the data sources that need to be mobilized and then create a comprehensive and reusable REST API platform that supports general-purpose application development.
Many of these problems disappear as soon as a company adopts a REST API platform strategy. There is no need to keep building server-side software for each new project. RESTful services are no longer tied to specific pieces of infrastructure. Applications can be moved between clouds or from testing to production with ease. Client-side application design is decoupled from security and administration. Development expenses and time to market are dramatically reduced.
Part 2: Band Aids for REST API Complexity
Increasing REST API complexity is a major problem for the modern enterprise. In short, as a company builds more and more web services, the server-side becomes increasingly complex, and this reduces portability, scalability, efficiency, and security. Building all of these APIs also slows down the development process and increases cost. In this section, I discuss some of the solutions that companies use to address this problem, and assess their effectiveness.
Better, Faster, Spaghetti
It starts innocently enough: someone needs to build a new enterprise application that requires web services. The development team has to think carefully about the services they need and then figure out how to build them and test them. In some cases, they use a REST API creation tool or adopt a specialized developer framework. Other internal and external teams may also use these tools, or perhaps something else, and over time they build lots of REST APIs for lots of new projects using different tools.
All of these services typically have different user-management systems, different security protocols, different parameter styles, and different requests and responses. They will be hardwired to various data sources, and designed to run on different pieces of physical infrastructure, in the cloud or on premises.
Using highly efficient REST API creation tools has an unintended consequence. Ironically, companies that use these tools hit the complexity wall harder and faster than they otherwise would have. The real problem is not how to build services quickly, but rather how to build the right services that can be reused for general-purpose application development across multiple projects.
Unfortunately, architecting a general-purpose REST API platform is a much more difficult problem than simply creating a bunch of web services. Only after building many applications will the proper design patterns become apparent, but most companies are unable to start over at that point.
One Proxy To Rule Them All
Some companies have tried to address the complexity problem by using API Management software. These systems allow existing REST APIs to be hooked up to a proxy server that can expose them in a unified way. Additional features include usage reporting, API throttling, and ecommerce gateways.
API Management can provide a unified interface for client application development, but it doesn’t address the core problem of backend complexity. Instead, API Management actually adds an additional arbitration layer on top of the existing mess, which increases the overall complexity of the system.
API Management solutions also require an additional proxy server gateway, and this limits the runtime performance needed for mobile applications and high-volume transactions. This gateway becomes yet another system that further reduces the performance, scalability, and portability of the backend infrastructure.
Fixing the Wrong End of the Stack
Another way to tackle complexity and data security is to implement Mobile Device Management (MDM). This approach attempts to sidestep the root cause of complexity by ignoring the server-side spaghetti and instead controlling data access from the client side. But as new mobile devices are deployed and employees demand flexible access to enterprise data sources, this strategy becomes more difficult to maintain.
To really address the root cause of complexity, companies should control access to the data source, not the device. Then, developers can build any type of client application that they want, given the existing service architecture. This “decouples” client-side software development from server-side software development, and dramatically speeds the development cycle.
With a comprehensive REST API service platform, users come in through single sign-on and are assigned a role that governs data access. If they leave their tablet on a plane, an administrator can deactivate their session. Controlling data access from the server side "decouples" security concerns from any particular client device.
In summary, backend complexity limits portability, scalability, efficiency, and security. Powerful REST API creation tools compound the problem. API Management and Mobile Device Management can mask some of the difficulties, but they don’t address the root of the problem.
Part Three) Building Reusable REST API Services
Luckily, there is a strategy that avoids these problems. This section discusses how to build a general-purpose and reusable REST API platform that functions as a universal data-access layer. This approach dramatically reduces complexity and provides a better, faster, and less costly way to build modern and secure enterprise mobile and web applications.
The Possible Dream
Wait up for a second: how do we know it’s even possible to build a reusable REST API platform in the first place? Our company has always built new services for each new project. That’s how we have to do it, right?
Think for a moment about ODBC and JDBC. These are the programmer interfaces that connect PHP and Java to SQL. They are used widely and are almost never modified for a particular project. This is a good example of a general-purpose interface for a complex backend database. What if someone wrote a comprehensive REST API for any backend data source or external service?
Consider the platform libraries on Microsoft Windows and Apple OS X that are designed for application development. These are APIs that support a huge number of development scenarios. There might be some application that can’t be written because of a missing API, but this is a rare situation compared to the long tail of applications that are written every day for Mac and Windows.
SOA is Not a Mobile Backend
Now, the old-timers in the audience might think this is related to Service-Oriented Architecture. But there’s a difference: SOA is primarily used for server-side integration and usually requires a big software development effort. Ask modern developers what they think about SOA, and they will tell you, “I love that show!” In other words, SOA is not a mobile backend. We need SOA capabilities that can be used from the client-side to address the needs of modern developers.
Comprehensive and Reusable
Reusable interfaces need to be flexible by design. For example, filter string parameters should enable a developer to dynamically query any SQL database or NoSQL document store. The service should support arrays of objects and related objects returned in the same transaction. This supports a huge number of access patterns “out of the box” without the need for further customization.
The services also need to cover many different usage scenarios in a comprehensive manner. Just considering the SQL interface, we need support for record-level access control, rollback and commit, data pagination, download limits, array sorting, metadata operations, schema discovery, and SQL views. Other features like date and time formats and stored procedures fall outside of the ANSI Standard and need to be handled in a consistent manner across different types of SQL databases.
Special cases need to be supported in an organized fashion. A server-side scripting and customization engine should modify requests and responses to handle formula fields, workflow triggers, special access controls, field validation, and custom usage limitations. The scripting engine can be used to implement custom services when necessary. Access to external web services is another way to extend the functionality of a reusable REST API services platform.
Here Comes the Sun
Everything changes when a company adopts a strategy based on comprehensive and reusable interfaces. System administrators set up the backend infrastructure and specify user access rights, and then application designers can safely build anything they want on top of that platform. This “decouples” client-side application development from server-side security, user management, and administration.
Client-side developers can use the same REST API for each new project. They can now tap into a universal data access layer to power their applications. The backend work is already finished. They might use different parameters or data objects, but the programming style is the same. A well-designed API can also minimize the differences between SQL, NoSQL, and file storage. Add filters to file storage and you get NoSQL. Add object enforcement and metadata to NoSQL and you get SQL. Consistent and well-designed services provide a much easier way to learn the API and write applications.
The applications written on a service platform also benefit from the virtualization provided by the service layer. Services are no longer tied to specific pieces of backend infrastructure. Moving an application from development to testing and on to production is easy. The service platform can be installed anywhere, so applications can be moved between clouds and data centers with ease.
Using a comprehensive REST API platform is much better than developing your own web services for each new project. The advantages are profound for server-side administrators, client-side developers, the applications that are written, and the company as a whole.
Part 4) More Than a REST API
We should consider some of the additional issues that companies will need to handle if they want to build a complete backend platform for enterprise mobile, web, and IoT applications. It’s going to take a lot more than just a REST API endpoint.
How Do I Work This?
Each API needs to be documented so that client developers can understand how to use the service. There will be different URLs for each application in development, testing, and production. Each URL will have identifiers for different types of resources. There will be required or optional URL parameters that take various arguments. Each service will have HTTP verbs that identify the allowed operations on the resource.
Most services return a JSON or XML document. Some services also require a document when they are called; for example, if you are creating database records. Client developers will need to understand the format of all of these requests and responses. This is a difficult problem to solve, because the format will be different based on various parameters and the backend database involved.
Sitting down with a technical writer and documenting a sophisticated and dynamic API is bound to be error-prone. A self-documenting API system is a better solution. If the documentation is online and interactive, then developers can quickly try out requests and responses, and see the different patterns of API usage.
Going further, your REST API should have a Client SDK that simplifies access for native mobile and HTML5 developers. A Client SDK can enable developers to leverage the REST API definitions and quickly write software in their native language for any target device or framework.
Custom-built REST API platforms are often administered with server logs and command-line operations. Someone might need to manage users, change permissions, configure services, upload applications, or create server-side business logic. These activities require highly technical personnel to maintain the system.
A better answer is to write an administrative console that enables less technical developers and designers to quickly change the backend platform. The admin console can also allow partners and end users to make some changes when appropriate. Third-party developers can use the development environment without access to master database credentials.
But creating an admin console for your REST API is serious work. Many of the capabilities of an admin console should also be available as RESTful services. So one secure way to proceed is to write a suite of administrative services for your backend and build the admin console on that platform. This will require additional services for administration and an additional application as well.
The Complete Package
Building a REST API is one thing, but often companies will need some degree of portability for the system as well. If they decide to change cloud vendors or move an application from the data center to the cloud, then they also need to move all of that backend code. Custom-built REST APIs tend to have hard-wired connections to various databases and pieces of backend infrastructure that can make this difficult.
Ideally your REST API backend can be installed almost anywhere, including IaaS clouds like Amazon, Azure, or Google, PaaS destinations like Heroku and OpenShift, and even on desktop computers like OS X, Windows, and Linux. This enables extreme horizontal portability across development, test, and production environments: developers can build applications on a desktop computer, system administrators can run the platform anywhere, and partners or customers can run and manage their own installations when necessary.
Security, Scalability, and Reliability
Anyone building their own REST API will need to address the issues surrounding security, scalability, and reliability. There is a real potential for nasty surprises as your implementation is rolled out to hundreds, thousands, or millions of users.
For security, the user role and permission system needs to be rock solid. Can you handle record-level access control? Are you preventing SQL injection attacks? Does your platform handle SSL and CORS correctly? Is the single sign-on system airtight? What about credential management? Can the URL routing engine be exploited? Is the server-side scripting system secure?
Your REST API will need to scale both horizontally and vertically. Additional servers behind a load balancer should result in more transactions per second. Increases in processor speed or available memory should result in faster transaction times. The process threading architecture needs to support this goal. This is why companies should be a little wary of using Node.js for building backend APIs; it isn't multi-threaded, which limits scalability.
Lastly, system reliability is very important for transactional hosting. Do you handle exponential backoff during a DOS attack? Can you place limits on service transactions to govern usage from a particular client? Can you monitor the backend platform for service outages? Can you handle a server-side script that takes too long to run? Can you paginate database transactions that are too big? Can you report on services that are failing, being called too much, or transacting too much data?
In many ways, the API endpoint is just the tip of the iceberg. Much of what is needed to build a secure and scalable RESTful services backend is below the surface.
If you’ve been following along up to now, the line of reasoning we’ve explored should paint a clear, cautionary, and ultimately hopeful picture:
• Building your own REST APIs can snowball into a giant mess that adds to backend complexity and slows mobile and web development projects.
• API development tools, and API management tools don’t solve the root complexity problem, they just add an additional arbitration layer to the mess.
• Building comprehensive, reusable REST API services for groups of applications is possible with a carefully designed RESTful backend.
• A REST API endpoint it just the tip of the iceberg. You’re going to need more than that for a secure, reliable, and scalable RESTful backend.
I hope this discussion will help people make better decisions and understand some of the hidden risks of “rolling your own” RESTful services. There are some gigantic benefits of using a pre-built backend as a platform for reusable and scalable RESTful services.
My company DreamFactory Software has been working on these problems for over a decade. We’ve struggled to solve these issues ourselves, and this inspired us to build the first open source mobile backend: the DreamFactory Services Platform. The DreamFactory Services Platform enables you to transform any data source into a comprehensive and well-documented palette of secure, customizable REST API services. We’ve built in deep support for a wide variety of backend data sources, including SQL and NoSQL, file storage systems, and external services.
To simplify front-end development, the DreamFactory Services Platform also generates client SDK code for every major development environment. And security is delivered by working with your own user authentication schemes with built-in services for user management, application hosting, record-level access control, and server-side scripting. As a free, open source software package, DreamFactory can be installed on any server, in the cloud, or on premises. And in addition to great community support, we offer professional product support services.
So when it comes to your REST API strategy for mobile and web application integration, you now have a choice. You can continue trying to build and manage your own APIs (with all of the associated problems) or you can use a free, open source backend that provides a comprehensive palette of secure, scalable, customizable RESTful services for any data source in minutes.