Accelerate the performance of Dojo TreeGrid

Load large data in less time

Dojo v1.4 TreeGrid is a useful widget to present hierarchical data on a web page. However, TreeGrid is extremely slow when working with large data sets. In this article, learn how you can alleviate this problem by customizing TreeGrid and QueryStore.

Sheng Hai Xu (xshai@cn.ibm.com), Software Engineer, IBM

Photo of Sheng Hai XuSheng Hai Xu is a software engineer with the Globalization Development group at the IBM China Development Lab in Shanghai. He has experience developing web-based tooling with Dojo and Web 2.0 technologies.



Yang Liu (elainel@cn.ibm.com), Software Engineer, IBM

author photoYang Liu is a software engineer at IBM China Software Development Lab in Shanghai, China. She works in the Shanghai Globalization Lab, and focuses on RFID technology, Web services and globalization technologies. You can reach her at elainel@cn.ibm.com.



Jie Hu (hujiesh@cn.ibm.com), Software Engineer, IBM  

Photo of Jie HuJie Hu is a technical leader in the Globalization Development area at the IBM China Development Lab in Shanghai. He has six years experience in J2EE development.



14 December 2010

Also available in Chinese Japanese

Introduction

The Dojo v1.4 TreeGrid is a useful widget when you want to present hierarchical data on a web page. But if you have a large data set, TreeGrid's performance is extremely slow. In this article, learn how to work around this problem by customizing TreeGrid and QueryStore. The article describes problems you might encounter when using the two widgets, explains the reasons for the errors, and then helps you create a solution. You can also download the sample code used in this article.


Dojo grid and large data sets

Of course, you can use the grid to effectively display relatively small data sets in the browser. You get the niceties of sorting, column resizing, and so on. However, there's a practical limit to the number of records you can deal with at any given time, which eventually leads to the issue of paginating results.

You can forget about pagination—those days are finally over. Dojo's grid uses lazy Document Object Model (DOM) rendering as the grid scrolls. For a relatively small data set with less than a few hundred records, lazy DOM rendering builds out the DOM for a preloaded data set when you scroll. For example, if you had 100 records but can only view 20 at a time, you wouldn't need to build the nodes for any of the records in 21 through 100 until you scrolled to that particular section. Sorting by a column and related tasks works in memory as expected for small data sets; you can effectively get things done in JavaScript.

For very large data sets it quickly becomes impractical, or even impossible, to use JavaScript for tasks such as sorting by a column. If the data set is gargantuan, it's not practical to try to maintain it in the browser with an ItemFileReadStore, which loads all data into a browser.

Dojo's approach to dealing with large data sets is simple, elegant, and boils down to two things:

  • A dojo.data implementation that can request data from the server in arbitrary page sizes. This article uses QueryReadStore.
  • A grid that's capable of scrolling such that it requests and loads a particular page on demand.

The rest of this article shows you how to build a lazy-loaded tree grid with QueryReadStore.


QueryReadStore, TreeGrid, and TreeModel

This section provides a very basic introduction to QueryReadStore, TreeGrid, and TreeModel.

QueryReadStore

As stated in the dojocampus.org doc (see Resources), QueryReadStore is very similar to ItemReadStore. They both use JSON as their exchange format. The difference is in the way they query data. ItemReadStore makes one fetch from the server and handles all sorting and filtering in the client. That's fine for hundreds of records. But for hundreds of thousands of records, or slow Internet connections, ItemReadStore is less feasible.

QueryReadStore makes a request to the server for each sorting or query, making it ideal for large data sets with small windows of data, as in dojox.grid.DataGrid. Listing 1 shows how to create a QueryReadStore.

Listing 1. Create a QueryReadStore
var url = "./servlet/QueryStoreServlet";
var store = new dojox.data.QueryReadStore({
    url: url,
    requestMethod: "get"
});

TreeGrid and TreeModel

A tree widget, such as Tree and TreeGrid, presents a view of hierarchical data. The TreeGrid widget itself is merely a view of the data. The real power comes with TreeModel, which represents the actual hierarchical data that the Tree widget will display.

Typically, data ultimately comes from a data store, but the Tree widget interfaces with a dijit.tree.Model (an object matching a certain API of methods that the tree needs). Therefore, the Tree widget can access data in various formats, such as with a data store where items reference their parents.

TreeModel is in charge of certain tasks, such as connecting to the data source, lazy loading, and querying from the Tree widget about items and the hierarchy of items. For example, a task might be getting children of an item for the Tree widget. TreeModel is also in charge of notifying the Tree of changes to the data.

To get started, you need to define the query of the ForestStoreModel (an implementation of TreeModel) to return the multiple top-level items for the TreeGrid. You will also create the model adapter for the tree to access the store. You need the parameters in Listing 2 to construct a TreeModel.

Listing 2. Code to create TreeModel
var query = { 
    type: "PARENT" 
}; 

var treeModel = new dijit.tree.ForestStoreModel({ 
    store: store, // the data store that this model connects to 
    query: query, // filter multiple top level items 
    rootId: "$root$", 
    rootLabel: "ROOT", 
    childrenAttrs: ["children"], // children attributes used in data store. 
    /* 
      For efficiency reasons, Tree doesn't want to query for the children 
      of an item until it needs to display them. It doesn't want to query 
      for children just to see if it should draw an expando (+) icon or not. 
      So we set "deferItemLoadingUntilExpand" to true. 
    */ 
    deferItemLoadingUntilExpand: true 
});

Relationship between QueryReadStore, TreeGrid, and TreeModel

To understand how to use a TreeGrid, keep in mind the following tree components that feed each other:

  • QueryReadStore is in charge of fetching data from the server.
  • ForestTreeModel is in charge of querying the hierarchy of items from QueryReadStore and notifying the TreeGrid to update.
  • TreeGrid is in charge of displaying the data and handling user events only.

Figure 1 shows the relationship between the three. Listing 3 shows how to create a tree.

Figure 1. Relationship between QueryReadStore, TreeGrid, and TreeModel
Flowcart showing steps from server to user event, with queryreadstore, foresttreemodel, and treegrid as steps.
Listing 3. Create a Tree
// define the column layout for the tree grid. 
var layout = [ 
    { name: "Name", field: "name", width: "20%" }, 
    { name: "Age", field: "age", width: "auto" }, 
    { name: "Position", field: "position", width: "auto" }, 
    { name: "Telephone", field: "telephone", width: "auto" } 
]; 

// tree grid 
var treegrid = new dojox.grid.TreeGrid({ 
    treeModel: treeModel, 
    structure: layout, // define columns layout 
    /* 
      A 0-based index of the cell in which to place the actual expando (+) 
      icon. Here we define the "Name" column as the expando column. 
    */ 
    expandoCell: 0, 
    defaultOpen: false, 
    columnReordering: true, 
    rowsPerPage: 20, 
    sortChildItems: true, 
    canSort: function(sortInfo) { 
        return true; 
    } 
}, "treegrid"); 

treegrid.startup();

How QueryReadStore works with TreeGrid

This section explores rendering the data in the TreeGrid, sorting with QueryReadStore, and expanding parent nodes.

Rendering

The QueryReadStore will send a request to the server for the top-level nodes. The first request the store will make to the server will be a request for the top-level nodes (the children of the root). This dojo.data request follows a specific JSON format.

The tree grid will make the initial request using the query provided to the model, and the store will combine that with the target. Listing 4 shows the request format, the REST URL pattern, and the server response to the request. In the example, the request is made to a REST pattern.

Listing 4. Query request and server response
// query 
{ 
  query: {type: "PARENT"}, 
  start: 0, 
  count: 20 //rowsPerPage attribute of the tree grid 
} 

// request sample 
url?type=PARENT&start=0&count=20 

// server response 
{ 
  "identifier": "id", 
  "label": "name", 
  "items": [ 
    { "id": "id_0", 
      "name": "Edith Barney", 
      "age": 39, 
      "position": "Marketing Manager", 
      "telephone": 69000044 
      "children": true, 
      "type": "PARENT" 
    }, 
    { "id": "id_1", 
      "name": "Herbert Jeames", 
      "age": 43, 
      "position": "Brand Executive Manager", 
      "telephone": 69000077, 
      "type": "Child" 
    }, 
    ... 
  ], 
  "numRows": 10000 // total records in the grid 
}

The item "Edith Barney" is of type "PARENT", and the "Herbert Jeames" item type is "Child". The query in the tree model is {type: "PARENT"}, so the tree grid will only display the "Edith Barney" record initially.

You don't need to actually include the children; the presence of a children property will indicate to the Tree that the node is expandable and an expansion icon will be included. The children attribute of item "Edith Barney" is not a false value, so the tree model will treat it as a node with children. There will be a plus (+) sign in the Edith Barney row in the tree grid. Figure 2 shows the result.

Figure 2. Rendering result of the created TreeGrid
Columns showing names, ages, job title, position, and telephone number, with a + by several names.

Now that you have the data rendered in the TreeGrid, you'll want to verify whether the TreeGrid features, such as sorting and expanding parent nodes, work correctly with the QueryReadStore.

Sorting

To exercise the sorting function, click on the header of the Name column and sort by name ascending. It is the same for sorting the other columns in ascending or descending mode.

Listing 5. Sorting
// the tree model will pass a request to the query store like this: 
{ 
  query: {type: "PARENT"}, 
  start: 0, 
  count: 20, 
  sort: { 
    attribute: "name", 
    descending: false 
  } 
} 

// The query store will request server like this: 
url?type=PARENT&start=0&count=20&sort=name

Figure 3 shows the sorted result.

Figure 3. Sorted result
Columns showing names, ages, job title, position, and telephone number, with a + by several names, sorted alphabetically.

Expanding parent nodes

For large tree data sets you probably want to only load the necessary data for the visible nodes of the tree. When a user expands a node, it starts to load the children of that node. Ideally, you only want to make one HTTP request per expansion for optimal performance.

When a user clicks on one of the nodes in the example, the tree will ask the store to load the item and the store will request the resource. But strange things may happen, such as in our case when the plus icon disappeared and the sub-rows didn't show.

Figure 4. Expanding does not work correctly
Columns showing names, ages, job title, position, and telephone number. One row does not have a + sign and wouldn't expand.

So, why doesn't it work?

Look through the source code of TreeGrid and TreeModel in Listing 6 to see how expanding sub-rows work.

Listing 6. How TreeGrid and TreeModel expand sub-rows
//TreeGrid.setOpen
setOpen: function(open){
    ...

    treeModel.getChildren(itm, function(items){
        d._loadedChildren = true;
        d._setOpen(open);
    });

    ...
} 
//TreeModel.getChilren
getChildren: function(/*dojo.data.Item*/ parentItem, /*function(items)*/ onComplete,
/*function*/ onError){
    // summary:
    // Calls onComplete() with array of child items of given parent item, all loaded.

    var store = this.store;
    if(!store.isItemLoaded(parentItem)){
    // The parent is not loaded yet, we must be in deferItemLoadingUntilExpand mode,
    // so we will load it and just return the children (without loading each child item) 
        var getChildren = dojo.hitch(this, arguments.callee);
        store.loadItem({
            item: parentItem,
            onItem: function(parentItem){
                getChildren(parentItem, onComplete, onError);
            },
            onError: onError
        });
        return; 
    }
    // get children of specified item
    var childItems = [];
    for(var i=0; i<this.childrenAttrs.length; i++){
        var vals = store.getValues(parentItem, this.childrenAttrs[i]);
        childItems = childItems.concat(vals);
    }
}

The tree model will ask the store to load the parent item first. The parent item is not fully loaded because its children attribute is not an array that lists all the children of this item.

Somehow, however, this case will pass the store.isItemLoaded(parentItem) check logic and you have childItems=[true]. Because true is not a valid item, the tree grid will skip this invalid child item, which results in nothing happening. Look at the code for the store.isItemLoaded method in Listing 7 to find out what's going on.

Listing 7. store.isItemLoaded
isItemLoaded: function(/* anything */ something){
    // Currently we don't have any state that tells if an item is loaded or not
    // if the item exists it's also loaded.
    // This might change when we start working with refs inside items ...
    return this.isItem(something);
}

The comments in this method provide the explanation. QueryReadStore does not support partially loading items at this time.


Customize QueryReadStore to support partial loading

In this section, you'll customize QueryReadStore and TreeGrid to implement partial loading and fix regression bugs.

You can download all the customized code. It's a web project, run in Eclipse, that's easily configured.

Create CustomQueryStore

To make the expand function of tree grid work with QueryReadStore, you must customize QueryReadStore to make it work with a partially loaded item. First, you must extend the class to add the missing methods. As shown in Listing 8, simply subclass QueryReadStore to CustomQueryStore.

Listing 8. Subclass QueryReadStore
/* treegrid-demo\WebContent\script\dojo-1.4.3\demo\data\CustomQueryStore.js */ 

dojo.provide("demo.data.CustomQueryStore"); 
dojo.require("dojox.data.QueryReadStore"); 

dojo.declare("demo.data.CustomQueryStore", dojox.data.QueryReadStore, { 
    /* @Override */ 
    isItemLoaded: function(/* anything */ something) { 
        // TODO 
    } 
});

The next step is to change the rule for checking if an item has been loaded, as shown in Listing 9.

Listing 9. isItemLoaded method
/* treegrid-demo\WebContent\script\dojo-1.4.3\demo\grid\CustomTreeGrid.js */ 

/* @Override isItemLoaded method */ 
isItemLoaded: function(/* anything */ something) { 
    // Currently we have item["children"] as a state that tells if an item is 
    // loaded or not. 
    // if item["children"] === true, means the item is not loaded. 
    var isLoaded = false; 

    if (this.isItem(something)) { 
        var children = this.getValue(something, "children"); 
        if (children === true) { 
            // need to lazy loading children 
            isLoaded = false; 
        } else { 
            isLoaded = true; 
        } 
    } 

    return isLoaded; 
}

QueryReadStore does not have a loadItem method, so you will create it. This method will request a server for an array that lists all the children of the item.

Listing 10. Override loadItem, getValues, and setValues
/* treegrid-demo\WebContent\script\dojo-1.4.3\demo\grid\CustomTreeGrid.js */ 

/* @Override loadItem method */ 
loadItem: function(/* object */ args) { 
    if (this.isItemLoaded(args.item)) { 
        return; 
    } 

    var item = args.item; 
    var scope = args.scope || dojo.global; 
    var sort = args.sort || null; 
    var onItem = args.onItem; 
    var onError = args.onError; 

    if (dojo.isArray(item)) { 
        item = item[0]; 
    } 

    // load children 
    var children = this.getValue(item, "children"); 

    // load children 
    if (children === true) { 
        var serverQuery = {}; 

        // "parent" param 
        var itemId = this.getValue(item, "id"); 
        serverQuery["parent"] = itemId; 

        // "sort" param 
        if (sort) { 
            var attribute = sort.attribute; 
            var descending = sort.descending; 
            serverQuery["sort"] = (descending ? "-" : "") + attribute; 
        } 

        // ajax request 
        var _self = this; 

        var xhrData = { 
            url: this.url, 
            handleAs: "json", 
            content: serverQuery 
        }; 

        var xhrFunc = (this.requestMethod.toLowerCase() === "post") ? 
                       dojo.xhrPost : dojo.xhrGet; 
        var deferred = xhrFunc(xhrData); 

        // onError callback 
             deferred.addErrback(function(error) { 
            if (args.onError) { 
                args.onError.call(scope, error); 
            } 
        }); 

        // onLoad callback 
        deferred.addCallback(function(data) { 
            if (!data) { 
                return; 
            } 

            if (dojo.isArray(data)) { 
                var children = data; 

                var parentItemId = itemId; 
                var childItems = []; 

                dojo.forEach(children, function(childData) { 
                    // build child item 
                    var childItem = {}; 
                    childItem.i = childData; 
                    childItem.r = this; 

                    childItems.push(childItem); 
                }, _self); 

                _self.setValue(item, "children", childItems); 
            } 

            if (args.onItem) { 
                args.onItem.call(scope, item); 
            } 
        }); 
    } 
} 

/* @Override geValues method */ 
getValues: function(item, attribute) { 
    //  summary: 
    //      See dojo.data.api.Read.getValues() 

    this._assertIsItem(item); 
    if (this.hasAttribute(item, attribute)) { 
        return item.i[attribute] || []; 
    } 

    return []; // Array 
} 

/* @Override seValue method */ 
setValue: function(/* item */ item, /* attribute-name-string */ attribute, 
                   /* almost anything */ value) { 
    // summary: See dojo.data.api.Write.set() 

    // Check for valid arguments 
    this._assertIsItem(item); 
    this._assert(dojo.isString(attribute)); 
    this._assert(typeof value !== "undefined"); 

    var success = false; 
    var _item = item.i; 
    _item[attribute] = value; 
    success = true; 

    return success; // boolean 
}

Now the example has a new CustomQueryStore. Replace the QueryReadStore in the JavaScript, as shown in Listing 11.

Listing 11. Use CustomQueryStore to send a request
var url = "./servlet/QueryStoreServlet";
var store = new demo.data.CustomReadStore({
    url: url,
    requestMethod: "get"
});

The code returns the result shown in Figure 5.

Figure 5. Returned result by customQueryStore
Columns showing names, ages, job title, position, and telephone number, employees shown in subrows below the manager.

Customize TreeGrid to fix regression bugs

It's time to check on other features to make sure there are no regression bugs after applying the CustomQueryStore.

Expand a row first, then click on the Name header to perform the sorting. As shown in Figure 6, an error occurs.

Figure 6. Error occurs when sorting CustomQueryStore
Screenshot with 'sorry an error occurred' over the first row.

After investigating, you find that the tree grid would always hold the states of expando functions. Before the sorting action, the fourth row is expanded and the tree grid remembered it. After the sorting action, the items were different, but the tree grid would still try to open the expando function of the fourth row. An error occurred because now the item for the fourth row does not have any children. You need to clear the expando states when the tree grid performs a sorting query.

Just as you customized QueryReadStore, you need to customize the TreeGrid. Then, using the CustomTreeGrid to replace TreeGrid should fix the sorting defect.

Listing 12. Customize TreeGrid
dojo.provide("demo.grid.CustomTreeGrid");
dojo.require("dojox.grid.TreeGrid");
dojo.declare("demo.grid.CustomTreeGrid", dojox.grid.TreeGrid, {
    /* @Override */
    sort: function() {
        this.closeExpando();

        this.inherited(arguments);
    },

    closeExpando: function(identities) {
        if (identities) {
            if (dojo.isArray(identities)) {
                // close multiple expando
                dojo.forEach(identities, function(identity) {
                    this._closeExpando(identity);
                }, this);
            } else {
                // close single expando
                var identity = identities;
                this._closeExpando(identity);
            }
        } else {
            // close all expando
            var expandoCell = this.getCell(this.expandoCell);
            for (var identity in expandoCell.openStates) {
                this._closeExpando(identity);
            }
        }
    },

    _closeExpando: function(identity) {
        var expandoCell = this.getCell(this.expandoCell);

        if (expandoCell.openStates.hasOwnProperty(identity) === true) {
            var open = expandoCell.openStates[identity] || false;
            if (open === true) {
                // clean up expando cache
                this._cleanupExpandoCache(null, identity, null);
            }
        }
    }

});

Together, the TreeGrid and QueryReadStore are a powerful combination for lazy loading data. Large, extensive hierarchical data can be displayed without large, upfront data transfers. You can leverage QueryReadStore's partial loading support to perform lazy loading with a single request per expansion.


Performance comparison

For this article, we compared the TreeGrid performance using ItemFileReadStore versus QueryReadStore. Figure 7 and Table 1 show that the customized QueryStore and grid uses only about 1/30 of the time used by ItemFileReadStore and basic TreeGrid.

Figure 7. Performance comparison between ItemFileReadStore and CustomQueryStore
Chart showing bar graph with itemfilereadstore reading 1400 and queryreadstore showing less than 100
Table 1. Performance comparison
Used data store Server data load times (s) Grid rending time (s) Total
ItemFileReadStore 1.45 12.65 14.1
CustomQueryStore 0.14 0.37 0.51

Summary

In this article, you learned how to create a customized solution to accelerate the performance for Dojo TreeGrid to load large data. You explored how QueryReadStore, TreeModel, and TreeGrid work together to fetch and render data. Customizing QueryStore and TreeGrid can solve the problem of Dojo currently not supporting large data very well. Results in the article showed that the performance of the solution is about 1/30th of the time that ItemFileReadStore and TreeGrid take.


Download

DescriptionNameSize
Sample code for this articletreegrid-demo.zip31KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Web development on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Web development
ArticleID=594756
ArticleTitle=Accelerate the performance of Dojo TreeGrid
publish-date=12142010