Worklight and JSONStore

IBM Worklight is a framework that enables developers to build hybrid and native mobile applications for many prevalent environments, such as Google's Android and Apple's iOS mobile operating systems. Part of the value of using this framework is having powerful APIs for solving key problems across the board.

For example, one common problem many developers face while crafting their mobile experience is securely managing data for offline use between their clients and the server. One of the APIs provided by this powerful framework to aid in such a scenario is called JSONStore. It provides an easy to use API, industry standard AES 256 bit data encryption, indexing of key fields for fast look up times, works with JSON formatted data and provides support for hybrid (Android, iOS, Windows Phone 8 and Windows 8) and native (Android and iOS) environments.

There is a great deal of information available to get started with these technologies. Some areas worth looking into before you continue reading:

• Knowledge Center
• Getting Started Modules
• JSONStore Examples
• JSONStore Working with External Data

Cloudant

Going back a couple of years you would see many companies investing heavily in their on premise infrastructure. These entities had to buy many servers based on estimates of how much traffic they expected to receive. They also had to contract system administrators to manage these servers, while ensuring an acceptable level of performance, security and scalability. Furthermore, typically you would need expensive software on top of that.

Fast forward to today and many companies are taking an entirely different approach to their infrastructure. These entities are preferring to pay others to manage key parts of their infrastructure (e.g. databases) so they can spend most of their capital on their core business (e.g. making their product better).

IBM has been investing heavily in simplifying tasks so companies can focus on what's really important to their bottom line, while at the same time reducing costs. It's easy to spot this trend when looking at recent acquisitions.

Cloudant is one of those companies recently brought into the IBM family. It's a NoSQL managed database that scales vertically across many servers located in various locations across the globe. It's powered by the open source database called Apache CouchDB. Many of the core Cloudant developers are active contributors to the open source project.

There's a plethora of information available to help you get started with this revolutionary Cloud as a Service solution:

• Cloudant Documentation
• Cloudant API Reference Documentation
• Cloudant Examples
• Cloudant Webinars

Goal and scope

The goal of this article is to show how one could work with both Cloudant on the backend and Worklight's JSONStore API on the client. There are many ways of keeping data on the backend (e.g. MySQL, MongoDB). There are also many ways of keeping data on the client (e.g. HTML5 LocalStorage, SQLite). The article presumes the reader has evaluated the various alternatives and has picked Cloudant and JSONStore. There are many reasons why developers would pick those two technologies, for example, a key feature of Cloudant is scalability and an important feature of JSONStore is security.

It's also imperative to mention that every application is different. You may have to tweak some of what you learn here in order to meet your requirements. Most of the content aims to be very generic and many not apply to all situations.

 

Data

Let's imagine we're creating an application for a list of contacts. The data would look like the following in table form:

Name	Age
carlos	10
mike	20

and like the following in JSON form:

[{"name" : "carlos", "age" : 10}, {"name" : "mike", "age" : 20}]

 

Cloudant Setup

You must go to cloudant.com and create an account. After you have done that, login to your account and go to the control panel (username.cloudant.com).

Next you need to add a database, call it people. This is where all your data will live on the backend.

If you don't want to bother with security while learning, you may go into the people database and modify the permissions to allow read and write access to Everybody Else. However, this is likely something you would not want in a production environment.

You may also add some data via the New > Document menu inside the people database. After you have finished adding data, you can see what you have added via the following URL: https://username.cloudant.com/people/_all_docs.

JSONStore Setup

If you don't have one already, you must create a Worklight Project and a Hybrid Application. After you've done that you may add environments such as: Android, iOS, Windows Phone 8 and Windows 8. Since JSONStore is an optional feature make sure you enable it as shown in the documentation.

You will be using the JSONStore initialization API to create two collections. You can think of collections as different tables inside a database.

The first collection will store the data we care about, let's call it the people collection. It should look like this once it has data inside:

name	age	id	_rev
carlos	10	2sze...	34d...
mike	20	42sa...	45e...

The values for name and age are part of our data. However, the values for id (which is mapped as _id when you get data from the backend) and _rev are generated by the backend database. The former is a unique identification number for a record and the latter is the revision number. We must map _id to something else (e.g. id) because JSONStore has its own _id which is different from Cloudant's _id value.

The second collection will keep track of the last time we have successfully communicated with the backed, let's call it lastUpdate. It should look like this once it has data inside:

collectionName	lastSeq
people	196-g1AAAAEjeJzL...

The value for last_seq is used to get changes from the backend, since other clients may have written to the database. It's part of the _changes API documented here. From the documentation: Returns a sorted list of changes made to documents in the database.

Transport Layer

You may want to take advantage of the Adapters as listed in the Server-side portion of the IBM Worklight Getting Started Modules in order to communicate with the backend. Alternatively, you could contact the Cloudant API directly using an XMLHttpRequest API like dojo.xhr and jQuery.ajax. There are many advantages to using Adapters which are mentioned in the documentation. One key advantage is security; for example, you can keep authentication data and tokens inside the context of the adapter in the server, and protect the adapter with security tests.

Pull

You need a way of applying changes made on the backend. We call this operation pull, but there's no single API for this. You use a couple of different APIs to achieve this with maximum flexibility. The first part of the algorithm is to ensure that the collections affected by the operation have been initialized.

After the collections have been properly initialized we can begin to load changes. One approach is to contact the _changes API on the people collection, passing include_docs as true to make sure the API call also returns the documents. We also need to pass since with the last_seq value we have stored in the lastUpdate collection. If no value for last_seq exists, we don't pass it and get back all changes. This is the case when we first try to pull data with an empty people collection on the client.

After we get the changes, need to either perform an update using JSONStore's change API which will update documents or add new documents. If the document was removed on the backend, we need to use JSONStore's remove API to ensure the document is removed from the client. In both cases the markDirty flag should be false so the change is not tracked as a local-only change.

After all the changes have been handled, the last_seq value used in the _change API call should be saved to the lastUpdate collection. That way the next time we ask for changes, we only get the changes since the last time we asked for changes, instead of all the changes since the database was created.

It's worth mentioning that you may want to follow a slightly different approach here based on your application's requirements. For example, you may want to use views to get only a portion of the data. Also, you may not want to get the documents directly from the _changes API if your documents contain a substantial amount of data.

It is assumed that all local-only changes have been sent to the server before the pull algorithm explained above is executed. An algorithm to send all local-only changes to the backend is described below.

Push

Documents with local only changes are called dirty documents when using the JSONStore API. When you perform operations such as add, replace and remove you may pass {markDirty: true} via the options object to note that change has not been sent to the server.

The first thing we must do is make sure that the collections being used are initialized. After that we can call JSONStore's getAllDirty API to get all documents with local only changes. If that API call returns no documents, we are done. Results from that API should look like this when the array returned is not empty:

[{json: {name: 'carlos', age: 10, id: '2sze...', _rev: '34d...'}, _operation: 'add', ...}]

Note that we have all the data for the document in the value for the json key. We also have the last operation performed on the document on the _operation field.

Based on the value of the _operation field we have to decide whether the document must be added, replaced or removed on the backend.

If the document was added on the client, we simply call the Cloudant API to add the document on the server too. The API uses the POST HTTP verb and takes the data representing the document in the body of the request. We get back an _id which is mapped locally as id to avoid conflicts and an updated _rev value. Both of these values should be used to update the local data using the JSONStore replace API and then the previously dirty document should be passed to the JSONStore markClean API to ensure that change is recognized as being up to date with the server.

If the document was removed on the client, we simply call the Cloudant API to add the document on the server too if the document exists on the server (that is, it has an identification number (id) and revision number (_rev) from the server).

The dirty document is then passed to JSONStore's markClean and then removed using JSONStore's remove API with the markDirty flag set to false.

If the document was replaced locally but does not contain an identification number (id) and revision number (_rev) we treat it as it was a new document added only on the client. Otherwise we call Cloudant's API to add the document, this API used the PUT HTTP verb and takes the document's id as part of the path. It also takes the data for the document in the request's body. The response should return an _id value mapped locally as id and a revision number (_rev) which should be used to update the local data using JSONStore's replace API. Followed by passing the dirty document to the markClean API the change is recognized as being up to date with the server.

Conclusion

During the course of this post we have talked about Worklight and pointed to valuable resources for learning the framework. We also introduced Cloudant, shared learning resources and created a new database using their web based control panel. We also talked through possible algorithms developers can use to keep data on the client, while receiving and sending changes to a Cloudant database.

Tags:  jsonstore worklight couchdb cloudant