Developing mobile apps with Node.js and MongoDB, Part 1: A team's methods and results

Speeding time to value of systems of engagement

Explore the advantages of using Node.js (server-side JavaScript) to develop systems of engagement. In this article, see the results achieved by an IBM Extreme Blue team who used Node and MongoDB with IBM Passes, an application that demonstrates the notion of systems of engagement. Their experience with Node and MongoDB provides interesting insight into how rapid application development can be achieved with these solutions that are starting to appear in the enterprise. If you decide to proceed with Node and MongoDB, please refer to Part 2 of this series, which provides how-to details and best practices for Node production.

Update on 26 November 2013: Additional clarity was added on the use of Node.js in the enterprise, alongside Java, as a enterprise application platform.

Update on 05 September 2013: See more background on the authors' technology choices and performance test approach, in response to reader feedback.

Zach Cross, IBM Extreme Blue intern, IBM

Author photo - Zach CrossZach Cross holds a B.S. in Computer Science from the University of North Carolina (UNC) at Chapel Hill. He is currently pursuing a graduate degree in Computer Science at UNC. Zach's expertise lies in software engineering and systems, especially network protocols. He also has a strong interest in distributed systems and GPGPU programming. He enjoys biking, reading and cooking in his spare time. He completed an IBM Extreme Blue internship in August 2013.



Aga Pochec, IBM Extreme Blue intern, IBM

Author photo - Aga PochecAga Pochec has dual Masters Degrees in International Business from the Warsaw School of Economics and the prestigious Community of European Management Schools and International Companies (CEMS) Program in Europe. She is currently completing her Masters in Business Administration at Harvard Business School in Boston and is an Extreme Blue intern at IBM. Aga is an experienced project manager with over five years of business intelligence and consulting experience spanning Europe and Asia Pacific. Her expertise lies in marketing, strategy, and business development. She loves traveling and exploring new cultures and cuisines in her free time. She completed an IBM Extreme Blue internship in August 2013.



Daniel Santiago, IBM Extreme Blue intern, IBM

Author photo - Daniel SantiagoDaniel Santiago is currently a computer engineering student at the University of Puerto Rico, Mayagüez. He is passionate about programming and interested in mobile, backend, and web development. Daniel likes working with emerging technologies such as Node.js and MongoDB. In his free time, he enjoys cooking, running and assembling gaming computers. He completed an IBM Extreme Blue internship in August 2013.



Divit Singh, IBM Extreme Blue intern, IBM

Author photo - Divit SinghDivit Singh is currently an undergraduate student at Virginia Tech, pursuing a B.S. in Computer Science. Divit's expertise lies in mobile technologies as he works with PhoneGap and Sencha Touch to create cross-platform applications. He enjoys developing applications and is a published developer on the Google Play Store as well as GitHub. He also likes working with emerging web technologies such as Node.js. He spends his spare time playing sports, music, and video games. He completed an IBM Extreme Blue internship in August 2013.



26 November 2013 (First published 19 August 2013)

Also available in Portuguese

Mobile and cloud development have dramatically changed the way companies operate and connect with customers. Enterprises increasingly seek back-end solutions that are easy to scale, ready for cloud and mobile deployments, and rapidly produced.

While Java is still the standard for building enterprise applications and the cloud, there is a strong push in the industry to reduce the amount of context switching when building applications. With concepts of MobileFirst design leading the way, developers are starting with an "outside-in" approach to development. Because "outside-in" design starts with the user, the user interface is prominently out front. One of the linchpins to developing UIs is JavaScript. By being able to start with JavaScript on the client and then being able to reuse these same JavaScript skills on the server (both Node.js and MongoDB are JavaScript based), there is a significant reduction in the amount of context switching between the various programming languages that a developer is required to understand. With industry heavyweights like Walmart, General Motors, and LinkedIn openly adopting and advocating Node, Node.js appears to going full force into the enterprise, alongside Java, as a enterprise application platform.

About the IBM Extreme Blue team

The Extreme Blue program is IBM's internship program for talented students pursuing software development and MBA degrees. Learn more at ibm.com/extremeblue.

The Extreme Blue team from the IBM lab in RTP, NC, was challenged to develop an entire backend in Node.js for IBM Passes. The team successfully built it in 40% less time than required by an alternative Java solution, while offering the same functionality. The team also performed comprehensive performance tests that demonstrated easier scalability and better hardware utilization of the Node.js backend (compared to Java).

This article introduces the key features and advantages of Node.js used with MongoDB. You will learn about the types of solutions best suited for Node.js production as well as the advantages and disadvantages of Node production.

Developers who are familiar with the basic concepts of Node.js, JavaScript, JSON, and RESTful web services will get the most out of this article.

 

IBM Passes

IBM Passes is a solution built around Apple Passbook, which was released by Apple in Fall 2012. Apple Passbook is an application that allows users to conveniently store in one location boarding passes, movie tickets, retail coupons, loyalty cards, and other materials for customer engagement (referred to as passes). Apple released the application together with specifications for passes. However, the company did not provide a way for businesses to create and manage these passes. IBM Passes addresses this problem. It features a RESTful web service that provides an API that other applications can consume. This service interacts with Apple's Push Notification Service via IBM PushWorks and handles analytics using IBM WebSphere® Analytics Platform.

IBM produced a frontend to this service, which enables users to create, update, and distribute passes. This frontend is referred to as Pass builder, shown in Figure 1. Pass builder is an HTML/JavaScript solution that relies on the RESTful interface of the Passes backend. For more on Pass builder, see the comprehensive introduction or brief introduction and video.

Figure 1. IBM Pass builder interface

The current implementation of the IBM Passes backend uses Java servlets to access DB2. With this solution, JSON documents are stored as strings in DB2's relational database. This requires serialization and deserialization for all transactions that involve these JSON documents, and leads to additional overhead. Furthermore, querying properties of these documents become more complex as nesting grows deeper.


An alternative to Java: a Node.js and MongoDB technology stack

A note on our technology choices

The team responsible for the original IBM Passes application originally considered using Node.js and MongoDB for this application. However, for a variety of reasons outside the scope of this article, they stuck with the more traditional enterprise development platforms (Java EE and a SQL database) for the backend.

Our project was focused on an alternative implementation of the RESTful backend to IBM Passes solution. We used a completely different technology stack: Node.js and MongoDB. This enabled us to replace the Java implementation while still supporting the frontend client. Our implementation provides the same set of APIs as the Java implementation. As a result, it is interchangeable as a backend to the Pass builder client.

In addition to recreating the functionality of the Java version, we conducted extensive performance testing on the two applications. Furthermore, the technology stack we used allowed us to implement the core functionalities in three weeks' development time as compared to the Java team's five weeks (40% reduction in time to value). We chose to use this technology stack for several reasons:

  • Apple Passbook specification (strictly JSON)
  • RESTful web service with JSON being primary data exchange format
  • JavaScript Everywhere (frontend, backend, data store)

Why we chose Node.js

Node advantages

Node.js (Node) is a scalable, event-driven I/O environment built on top of Google Chrome's JavaScript runtime—essentially, a server-side implementation of JavaScript. Google V8 actually compiles JavaScript into native machine code prior to execution, resulting in extremely fast runtime performance—something not typically associated with JavaScript. As such, Node enables you to rapidly build network apps that are lightning fast and highly concurrent.

Node is a JavaScript web server runtime. Because it uses JavaScript, frontend developers can work in the same language as backend developers. This concept is referred to as JavaScript Everywhere. In practice, this concept unifies development efforts by reducing the number of different concepts that developers must understand. Furthermore, the most used data exchange format, JSON, can be natively parsed by both ends. This is another way processing overhead can be reduced through simplification of serialization. In addition to the benefit of using a script language, Node's lightweight runtime enables rapid development and deployment. As a result, Node is a powerful solution for agile development, where iteration typically involves working in vertical slices.

Table 1. Key technology concepts used for IBM Passes
Pass builder frontendPasses backend
JavaScriptJava
Node.jsServlet container
MongoDBDB2
JSONSQL
JSON

Node contains features that support the workloads of the modern web: small and frequent structured data exchanges. This makes Node perfect for systems of engagement. In other words, it is an ideal candidate for applications that involve exchanges of information rather than computational processes. Technically speaking, this means choosing Node for I/O-bound applications rather than CPU-bound ones. IBM Passes is a perfect example of such an application. The only computationally expensive operations are cryptographic signing and compression (zipping). The rest of the application revolves around the exchange of JSON data and image resources.

Node differs from most web application runtimes in the manner that it handles concurrency. Rather than using threading to accomplish concurrency, Node relies on an event-driven loop running in a single process. Node supports an asynchronous (non-blocking) model, whereas technologies such as Java support a synchronous (blocking) model. To clarify the key differences between these two concepts, consider the following restaurant metaphor:

Asynchronous model

Think of a web application as a restaurant and its incoming requests as customers making orders. An asynchronous application would allow a single waiter to serve multiple customers at once. The waiter would serve the customers as orders are completed. In the down time, the same waiter would handle new orders from customers.

Synchronous model

A synchronous "restaurant" would dedicate a single waiter to a single customer from the start of its order to completion. As a result, the restaurant needs as many waiters as customers. Furthermore, there would be times when waiters would wait while other work could be done.

The difference illustrated by this metaphor is simply the mechanism used to achieve concurrency. Java accomplishes it with threads, while Node uses an event loop. A result is that Java must suffer additional overhead in context switching between threads. In other words, more threads means more time spent switching contexts and less time doing work on incoming requests. This makes scaling a Java application more expensive.

Node supports asynchronous I/O, based on events such as the completion of a file read operation or a database query. These events are handled with callback functions, which enable the application to proceed while I/O is being performed. The event loop handles these events. For more on the event loop, refer to "Node.js for Java developers," a developerWorks article by Andrew Glover.


Why we chose MongoDB

MongoDB is a document-oriented database designed for ease of development and scaling. MongoDB's ability to store JavaScript objects natively saves time and processing power. Instead of a domain-specific language like SQL, MongoDB utilizes a simple JavaScript interface for querying. Looking up a document is as simple as passing a JavaScript object that partially describes the search target.

MongoDB was a great solution for our problem domain because, as mentioned, Apple's Passbook specification is highly reliant on JSON. In fact, the passes themselves are described as JSON objects. As a result, storing them as-is with MongoDB is a better approach than reducing the semi-complex JSON structure to multiple relations in a relational database (such as DB2).


Benchmarking

Our approach to performance testing

The team (IBM Mobile Cloud Development team and Extreme Blue Team) decided to do a bake-off instead of trying to blur the lines to truly evaluate the complete stack. We found that for Java-based technologies, it isn't easy to find adapters to talk to MongoDB, similar to the lack of adapters for Node.js to talk to DB2. So we thought the time would be better spent maintaining the separation between old-school and new-school. What we also found is that the technologies would typically be combined old + old, or new + new, so we chose to focus primarily on these pairings.

Due to time constraints, there were two things we did not have a chance to do. First, some of our team members are long-time web container guys, so we would have liked to see performance numbers at a call stack level to indicate which methods were the most expensive (both shallow and deep) to get both a fine-grained and cumulative view on time spent in each method.

We wanted to compare our Node implementation of the IBM Passes to its Java version. We decided to benchmark the two applications' end to end performance, with the metric of interest ultimately being the response times of HTTP requests made to the various API endpoints. The main purpose for this approach was to capture the performance of the entire stack behind each solution. By accounting for variables such as operating system and network, we achieved objective comparison of the two applications. Moreover, we believed that the end-to-end response times were a better reflection of an actual user experience than more granular analysis such as the timing of database queries or timing individual functions.

Methodology

For the purpose of benchmarking both applications, we wrote a Node wrapper around Apache Benchmark, an HTTP benchmarking command line tool. From there, we created a benchmarking framework that ran against generic test definitions and generated results. Apache Benchmark gave us the mean response time, its standard deviation, the number of requests completed per second, the number of unsuccessful (non-200) responses, the number of server errors, and the number of timeouts. The parameters were concurrency level, number of requests, and request configuration. We added test definitions and setup functions into this framework and proceeded to benchmark our application.

We addressed the tooling side of our benchmarking process. The test environment is of equal, if not greater, importance due to the intended scientific approach to this process. We decided to use OpenStack to mimic a cloud environment. We configured identical instances for the Node application, the Java application, and the load generation application. Each instance was given a 2.4 GHz quad core (virtual) processor, 8 GB (virtual) RAM, and 80GB of storage. Figure 2 shows the topology of the OpenStack deployment used.

Figure 2. OpenStack deployment topology

These instances were configured with the same base image (Ubuntu Server 12.04 64-bit). Furthermore, only software necessary for running and monitoring the applications was installed. When it comes to runtime configuration, we used the default configuration for Node (the same that Chrome V8 Engine relies on).

We consulted the Java team regarding runtime configuration of the JRE and used default configurations for the servlet container and DB2. We acknowledge that further configuration could potentially improve the performance of the back end. However, that is also true for the Node runtime and misses the point. Our purpose is to prove that it meets performance criteria and can be scaled horizontally rather than vertically, in addition to facilitating agile development.

Benchmarking jobs were queued manually via the load generator instance's web interface, or as a result of changes to the code base. Benchmarking proceeded within the OpenStack environment. We defined a benchmarking job as the execution of all test suites (such as groups of API endpoint runs) given the parameters of branch name, concurrency level, and total requests to make. Within a benchmark, a fixed set of API endpoints were tested by the benchmarking tool. These tests were run sequentially (that is, without overlapping traffic). Furthermore, benchmarks (groups of tests) were run sequentially for the same reason.

Results

Our test case demonstrated that the Node and MongoDB implementation had faster response times when the concurrency level exceeded 50. We can attribute this to the I/O-bound nature of the Passes application. Below the concurrency level of 50, the Java application was faster because the application degraded from I/O to CPU-bound under the reduced load. With higher concurrency levels, the single Node instance clearly outperformed the Java/DB2 instance for every API call.

Figure 3. Performance outcomes (counting “faster” endpoints per application)

To compare hardware utilization of Node vs. Java implementation, we needed to profile the instances that the two applications were running on. For this purpose, we used two popular statistic collection applications: CollectD and StatsD. Installing CollectD on our instances introduced negligible overhead and made CPU, memory, and disk usage transparent. We used StatsD on the monitoring instance to collect these statistics. Graphite was used on the same instance for viewing time series data as graphs presented in a dashboard format.

We were able to correlate each benchmark with time series hardware utilization data. While we have average usage values for memory and CPU usage during the benchmarking of each endpoint, we found it more valuable to show general trends. Figure 4 shows these metrics over the time period of a long-running benchmark. The Node application used a lower percentage of the CPU and less memory on average. The low CPU usage might be interpreted as a result of the I/O-bound nature of the application.

Figure 4. Performance results of hardware utilization benchmarking

Conclusions

While Node might not be a panacea for all the challenges of the modern web, it is perfect for demands of systems of engagement. Node was designed specifically for I/O-bound applications and frequent exchanges of information. Its lightweight runtime enables agile development and immediate iteration. As demonstrated in our test case of IBM Passes, Node led to 40% reduction in time to value while allowing us to double the traffic served with half the servers (in comparison to Java implementation). Thus, Node was able to deliver the same functionalities as Java, and it outperformed Java in terms of rapid development and better hardware utilization.


Acknowledgments

Special thanks to our mentors, whose wisdom and experience guided us throughout our internship: Joshua A. Alger, Andy Dingsor, Curtis M. Gearhart, Christopher Hambridge, and Todd Kaplinger. A big thank you to Ross Grady, RTP Lab Manager for IBM Extreme Blue, whose patience and push for constant improvement ensured our success. We would also like to express our gratitude to Jeff Jagoda for his Node.js experience and invaluable feedback.

Resources

Learn

Get products and technologies

  • Download IBM product trials: Get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

Discuss

  • Get involved in the developerWorks community: Connect with peers and experts as you explore the developer-driven blogs, forums, groups, and wikis.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Mobile development on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Mobile development, Java technology
ArticleID=954701
ArticleTitle=Developing mobile apps with Node.js and MongoDB, Part 1: A team's methods and results
publish-date=11262013