Work with external data

Learn about the different concepts that are required to work with external data.

For the actual API examples, see JSONStore examples.

Pull

Many systems use the term pull to refer to getting data from an external source.

There are three important pieces:

External Data Source: This source can be a database, a REST or SOAP API, or many others. The only requirement is that it must be accessible from either the MobileFirst Server or directly from the client application. Ideally, you want this source to return data in JSON format.
Transport Layer: This source is how you get data from the external source into your internal source, a JSONStore collection inside the store. One alternative is a MobileFirst adapter.
Internal Data Source API: This source is the JSONStore APIs that you can use to add JSON data to a collection.

Note: You can populate the internal store with data that is read from a file, an input field, or hardcoded data in a variable. It does not have to come exclusively from an external source that requires network communication.

Example pull scenario

All of the following code examples are written in pseudocode that looks similar to JavaScript.

Note: Use MobileFirst adapters for the Transport Layer. Some of the advantages of using MobileFirst adapters are XML to JSON, security, filtering, and decoupling of server-side code and client-side code.

External Data Source: Backend REST endpoint

Imagine that you have a REST endpoint that read data from a database and returns it as an array of JSON objects.

app.get('/people', function (req, res) {

  var people = database.getAll('people');

  res.json(people);
});

The data that is returned can look like the following example:

[{id: 0, name: 'carlos', ssn: '111-22-3333'},
 {id: 1, name: 'mike', ssn: '111-44-3333'},
 {id: 2, name: 'dgonz' ssn: '111-55-3333')]

Transport Layer: MobileFirst adapter

Imagine that you created an adapter that is called people and you defined a procedure that is called getPeople. The procedure calls the REST endpoint and returns the array of JSON objects to the client. You might want to do more work here, for example, return only a subset of the data to the client.

function getPeople () {

  var input = {
    method : 'get',
    path : '/people'
  };

  return MFP.Server.invokeHttp(input);
}

On the client, you can use the WLResourceRequest API to get the data. Additionally, you might want to pass some parameters from the client to the MobileFirst adapter. One example is a date with the last time that the client got new data from the external source through the MobileFirst adapter.

var adapter = 'people';
var procedure = 'getPeople';
 
var resource = new WLResourceRequest('/adapters' + '/' + adapter + '/' + procedure, WLResourceRequest.GET);
resource.send()
.then(function (responseFromAdapter) {
  // ...
});

Note: You might want to take advantage of the compressResponse, timeout, and other parameters that can be passed to the WLResourceRequest API.

Alternatively, you can skip the MobileFirst adapter and use something like jQuery.ajax to directly contact the REST endpoint with the data that you want to store.

$.ajax({
  type: 'GET',
  url: 'http://example.org/people',
})
.then(function (responseFromEndpoint) {
  // ...
});

Internal Data Source API: JSONStore

After you have the response from the backend, you can work with that data by using JSONStore.

JSONStore provides a way to track local changes. It enables some APIs to mark documents as dirty. The API records the last operation that was performed on the document, and when the document was marked as dirty. You can then use this information to implement features like data synchronization.

The change API takes the data and some options:

replaceCriteria

These search fields are part of the input data. They are used to locate documents that are already inside a collection. For example, if you select:

['id', 'ssn']

as the replace criteria, pass the following array as the input data:

[{id: 1, ssn: '111-22-3333', name: 'Carlos'}]

and the people collection already contains the following document:

{_id: 1,json: {id: 1, ssn: '111-22-3333', name: 'Carlitos'}}

The change operation locates a document that matches exactly the following query:

{id: 1, ssn: '111-22-3333'}

Then the change operation performs a replacement with the input data and the collection contains:

{_id: 1, json: {id:1, ssn: '111-22-3333', name: 'Carlos'}}

The name was changed from Carlitos to Carlos. If more than one document matches the replace criteria, then all documents that match are replaced with the respective input data.

addNew

When no documents match the replace criteria, the change API looks at the value of this flag. If the flag is set to true, the change API creates a new document and adds it to the store. Otherwise, no further action is taken.

markDirty

Determines whether the change API marks documents that are replaced or added as dirty.

An array of data is returned from the MobileFirst adapter:

.then(function (responseFromAdapter) {

  var accessor = WL.JSONStore.get('people');

  var data = responseFromAdapter.responseJSON;

  var changeOptions = {
    replaceCriteria : ['id', 'ssn'],
    addNew : true,
    markDirty : false
  };

  return accessor.change(data, changeOptions);
})

.then(function() {
  // ...
})

You can use other APIs to track changes to the local documents that are stored. Always get an accessor to the collection that you perform operations on.

var accessor = WL.JSONStore.get('people')

Then, you can add data (array of JSON objects) and decide whether you want it to be marked dirty or not. Typically, you want to set the markDirty flag to false when you get changes from the external source. Then, set the flag to true when you add data locally.

accessor.add(data, {markDirty: true})

You can also replace a document, and opt to mark the document with the replacements as dirty or not.

accessor.replace(doc, {markDirty: true})

Similarly, you can remove a document, and opt to mark the removal as dirty or not. Documents that are removed and marked dirty do not show up when you use the find API. However, they are still inside the collection until you use the markClean API, which physically removes the documents from the collection. If the document is not marked as dirty, it is physically removed from the collection.

accessor.remove(doc, {markDirty: true})

Push

Many systems use the term push to refer to sending data to an external source.

There are three important pieces:

Internal Data Source API: This source is the JSONStore API that returns documents with local-only changes (dirty).
Transport Layer: This source is how you want to contact the external data source to send the changes.
External Data Source: This source is typically a database, REST or SOAP endpoint, among others, that receives the updates that the client made to the data.

Example push scenario

All of the following code examples are written in pseudocode that looks similar to JavaScript.

Internal Data Source API: JSONStore

After you have an accessor to the collection, you can call the getAllDirty API to get all documents that are marked as dirty. These documents have local-only changes that you want to send to the external data source through a transport layer.

var accessor = WL.JSONStore.get('people');

accessor.getAllDirty()

.then(function (dirtyDocs) {
  // ...
});

The dirtyDocs argument looks like the following example:

[{_id: 1,
  json: {id: 1, ssn: '111-22-3333', name: 'Carlos'},
  _operation: 'add',
  _dirty: '1395774961,12902'}]

The fields are:

_id: Internal field that JSONStore uses. Every document is assigned a unique one.
json: The data that was stored.
_operation: The last operation that was performed on the document. Possible values are add, store, replace, and remove.
_dirty: A time stamp that is stored as a number to represent when the document was marked dirty.

Transport Layer: MobileFirst adapter

You can choose to send dirty documents to a MobileFirst adapter. Assume that you have a people adapter that is defined with an updatePeople procedure.

.then(function (dirtyDocs) {
  var adapter = 'people',
  procedure = 'updatePeople';
 
  var resource = new WLResourceRequest('/adapters/' + adapter + '/' + procedure, WLResourceRequest.GET)
  resource.setQueryParameter('params', [dirtyDocs]);
  return resource.send();
})
 
.then(function (responseFromAdapter) {
  // ...
})

Note: You might want to take advantage of the compressResponse, timeout, and other parameters that can be passed to the WLResourceRequest API.

On the MobileFirst Server, the adapter has the updatePeople procedure, which might look like the following example:

function updatePeople (dirtyDocs) {

  var input = {
    method : 'post',
    path : '/people',
    body: {
      contentType : 'application/json',
      content : JSON.stringify(dirtyDocs)
    }
  };

  return MFP.Server.invokeHttp(input);
}

Instead of relaying the output from the getAllDirty API on the client, you might have to update the payload to match a format that is expected by the backend. You might have to split the replacements, removals, and inclusions into separate backend API calls.

Alternatively, you can iterate over the dirtyDocs array and check the _operation field. Then, send replacements to one procedure, removals to another procedure, and inclusions to another procedure. The previous example sends all dirty documents in bulk to the MobileFirst adapter.

var len = dirtyDocs.length;
var arrayOfPromises = [];
var adapter = 'people';
var procedure = 'addPerson';
var resource;
 
while (len--) {
 
  var currentDirtyDoc = dirtyDocs[len];
 
  switch (currentDirtyDoc._operation) {
 
    case 'add':
    case 'store':
 
    resource = new WLResourceRequest('/adapters/people/addPerson', WLResourceRequest.GET);
    resource.setQueryParameter('params', [currentDirtyDoc]);
 
      arrayOfPromises.push(resource.send());
 
    break;
 
    case 'replace':
    case 'refresh':
 
    resource = new WLResourceRequest('/adapters/people/replacePerson', WLResourceRequest.GET);
    resource.setQueryParameter('params', [currentDirtyDoc]);
 
 
      arrayOfPromises.push(resource.send());
 
    break;
 
    case 'remove':
    case 'erase':
 
    resource = new WLResourceRequest('/adapters/people/removePerson', WLResourceRequest.GET);
    resource.setQueryParameter('params', [currentDirtyDoc]);
 
      arrayOfPromises.push(resource.send());
  }
}
 
$.when.apply(this, arrayOfPromises)
.then(function () {
  var len = arguments.length;
 
  while (len--) {
    // Look at the responses in arguments[len]
  }
});

Alternatively, you can skip the MobileFirst adapter and contact the REST endpoint directly.

.then(function (dirtyDocs) {

  return $.ajax({
    type: 'POST',
    url: 'http://example.org/updatePeople',
    data: dirtyDocs
  });
})

.then(function (responseFromEndpoint) {
  // ...
});

External Data Source: Backend REST endpoint

The backend accepts or rejects changes, and then relays a response back to the client. After the client looks at the response, it can pass documents that were updated to the markClean API.

.then(function (responseFromAdapter) {

  if (responseFromAdapter is successful) {
    WL.JSONStore.get('people').markClean(dirtyDocs);
  }
})

.then(function () {
  // ...
})

After documents are marked as clean, they do not show up in the output from the getAllDirty API.