Grails is a Web application framework that leverages the dynamic syntax of Groovy and the back-end muscle of the Java platform. memcached is a general-purpose distributed memory caching system that is used in some of the most heavily trafficked sites on the Web. Using the two together makes it possible to rapidly build extremely responsive and scalable Web applications.

In this second half of my introduction to memcached and Grails, I'll walk you through the process of integrating memcached into an existing Grails application. I'll begin by introducing you to the Grails application and getting it set up in your development environment. I'll then introduce the memcached client API and walk you through creating a Grails service to wrap it.

With all the components in place to use memcached in your Grails application, you might be wondering how you can best leverage it. I'll discuss the most effective areas where you could inject caching into an existing application and also demonstrate some techniques used to store and retrieve values from the cache. Finally, you'll walk through a hands-on exercise implementing caching in one of the application's Grails controllers, testing the results as you go along.

The Grails application

In order to see memcached in action, you'll need a sample Grails application to work from. The application need not be very complex, so you'll just build a simple contact-management app that allows a user to manage a collection of contacts and their associated information. This mini-CRUD application's sole function will be to let users store and retrieve Contact objects using memcached. To get started, execute the following command in your development environment:

grails create-app

Enter the application name. For this example, use the name contactmanager. Once Grails has created the application, cd to the contactmanager directory and start up the application:

cd contactmanager
grails run-app

Open your browser to http://localhost:8080/contactmanager. If everything went okay, you should see the Grails welcome page.

Creating the domain

Next, you need to define the Contact domain class. We'll keep it pretty simple by only adding three properties to the domain: firstName, lastName, and email. After adding these properties, your Contact domain should look like Listing 1:

class Contact {

    def String firstName
    def String lastName
    def String email

    static constraints = {
        firstName(maxLength: 50, blank: false)
        lastName(maxLength: 50, blank: false)
        email(blank: false, nullable: false, email:true)
    }
}

Go ahead and create this Groovy class in your own contactmanager/grails-app/domain directory.

Creating the Controller

Next, you'll create the application Controller that will serve up Contacts. Start by executing the following command:

grails create-controller
Contact

Now open up your newly created ContactController and change it to look like Listing 2:

class ContactController {

    def scaffold = Contact
}

This will give you the default Grails scaffolding. Now you want to add some content. So, open up BootStrap.groovy (used to initialize Grails applications on startup) and modify it as shown in Listing 3. Database initialization, dynamic configuration, and injecting meta-changes into Groovy classes are all common tasks performed by BootStrap.groovy. I also frequently use it to configure my development database to a default state.

class BootStrap {

    def init = {servletContext ->

        (1..10000).each {i ->

            def Contact contact =

                new Contact(firstName: "Bob",

                lastName: "Johnson${i}",

                email: "bob.johnson${i}@email.com").save()
        }
    }

    def destroy = {  

    }
}

The purpose of the code in Listing 3 is to create 10,000 contacts in the Grails application's default HSQLDB database. Feel free to restart the application and poke around a bit; you'll get an idea of what you're working with that way.

Adding logic to ContactController

The final step in creating the Grails sample app is to add simple logic to the ContactController to print query runtimes. Later on, you will use this information to see how much caching has improved the application's performance. Before adding the logic to the controller, you first need to tell Grails to generate all the scaffolding code for Contact. Do this by executing the grails generate-all command:

grails generate-all

Now open the file contactmanager/grails-app/controllers/ContactController.groovy. You should see all of the closures needed to support your Contact domain. Find the list closure and add timing statements to it, as shown in Listing 4:

def list = {

    params.max = Math.min( params.max ? params.max.toInteger() : 10,  100)

    def startTime = new Date().getTime()
    def contactInstanceList = Contact.list(params)
    def contactInstanceTotal = Contact.count()
    def endTime = new Date().getTime()

    println "Transaction time was ${(endTime - startTime) / 1000} seconds."
 
    [contactInstanceList: contactInstanceList, contactInstanceTotal: 
        contactInstanceTotal]
}

The code in Listing 4 adds a startTime that is set to the current system time in milliseconds. It then runs the two queries used to get a list of Contacts and the total number of Contacts. After running these two queries, you get the current time in milliseconds again and set them in the endTime variable. Finally, you print out the difference divided by 1,000 to see, in seconds, how long the queries took to run.

Restart the application using grails run-app and watch the console as you page through the contact list results.

Injecting memcached into the Grails app

The first thing you need to do to add the memcached client to your Grails application is download the appropriate jar file and copy it to your contactmanager/lib directory. For this example, I'm using Spymemcached, a Java client for memcached. Go ahead and download the JAR; the latest release as of this writing is 2.3.1.

Once you have the jar file in your contactmanager/lib, your next step is to create a Groovy class that will expose the API. I have chosen to use a Grails service for this implementation, for two very good reasons: First, all Grails services are managed as Spring beans and can therefore be automatically injected into your Controllers. Second, being a Spring bean, the Grails service gives you access to the org.springframework.beans.factory.InitializingBean interface, which lets you initialize the service after all other properties have been set.

So, let's create the service. Open whatever IDE or editor you use and create the Groovy class from Listing 5 in the contactmanager/grails-app/services directory:

import net.spy.memcached.AddrUtil
import net.spy.memcached.MemcachedClient
import org.springframework.beans.factory.InitializingBean

class MemcachedService implements InitializingBean {

    static final Object NULL = "NULL"
    def MemcachedClient memcachedClient

    def void afterPropertiesSet() {
        memcachedClient = new MemcachedClient(AddrUtil.getAddresses("localhost:11211"))
    }

    def get(String key) {
        return memcachedClient.get(key)
    }

    def set(String key, Object value) {
        memcachedClient.set(key, 600, value)
    }

    def delete(String key) {
        memcachedClient.delete(key)
    }

    def clear() {
        memcachedClient.flush()
    }

    def update(key, function) {
        def value = function()
        if (value == null) value = NULL
        set(key, value)
        return value
    }

    def get(key, function) {
        def value = get(key)
        if (value == null) {
            value = update(key, function)
        }
        return (value == NULL) ? null : value;
    }
}

The majority of the methods in Listing 5 are probably exactly what you would expect to see — get() set(), delete(), and clear() — but there are some less usual elements as well. Let's look at them.

First, consider the line:

static final Object NULL = "NULL"

You will use this value whenever a null is being stored in memcached. You need this line because you cannot serialize null and all objects that are placed in memcached have to be serializable.

Next is the afterPropertiesSet() method. As I previously mentioned, this method is invoked by Spring after all the properties have been set. Note that in this method, I also added code to create a new instance of the MemcachedClient and connect to the memcached server.

The last two methods worth noting are update() and get(), which each take the properties key and a function. These methods demonstrate an interesting approach to using a cache, called memoization.

What I've done is to pass the key of the item I want to look up, along with the function that should execute if the get() does not find the key. This technique is used to avoid repeatedly calculating the results of previously processed inputs. Instead, a simple get() call in your code both retrieves the object desired and creates the object if it is not found in the cache. It also then stores the results. Later in the article, you'll learn more about how memoization simplifies cache interactions.

MemcachedService meets ContactController

To add the newly created MemcachedService into your ContactController, add the following line to your ContactController:

class ContactController {

    def memcachedService

    ...

}

Adding this simple line automatically injects an instance of the MemcachedService into the Controller. Now the service is available in the Controller, but what should the Controller cache?

Using memcached

When determining which application data to cache, it's good to keep two simple guidelines in mind. These are not etched in stone and will not apply to every case, but they provide a good foundation for deciding what to cache:

• Do not cache data that changes frequently. If the data you want to cache changes often, you will be constantly changing the values stored in the cache, which will limit the caches value.
  
  
• If you have an id that identifies a value directly, then you don't need to cache that value. The database will be able to look up a value using its id very quickly.

In the case of the Contact Manager application, these guidelines make it apparent that you want to cache the data returned when paging through contacts. Remember that this data is queried in the list closure of the ContactController.

Two values are returned by the list closure:

• contactInstanceList is a list of Contacts retrieved from the database.
  
  
• contactInstanceTotal represents the total number of Contacts in the database.

Both of these items are candidates for caching, but let's start with contactInstanceTotal.

Caching the contact-instance total

In order to cache the contact-instance data, start by adding a method to the ContactController, which will cache the total Contacts:

def getUsername() {

    // dumb method to return a username
    // you are not implementing any security in this example
    return "my.username"
}

def getContactInstanceTotal(username) {

    def contactInstanceTotal = memcachedService.get("${username}:contactInstanceTotal") {

        def contactInstanceTotal = Contact.count()
        return contactInstanceTotal
    }
    return contactInstanceTotal
}

getUsername() is a dumb method. It is only necessary because security is out of scope for this article, and so the sample application doesn't have the concept of a user. It's in getContactInstanceTotal() that the first real interaction with the cache begins. This method invokes the memcachedService.get() method with the key "my.username:contactInstanceTotal." If the key is found in the cache, then the value will be returned to the caller. If not, the closure passed to the get() will be invoked and the returned value will be stored in the cache with the passed-in key. Go ahead and add this code to the ContactController, and then replace the following:

def contactInstanceTotal = Contact.count()

with a call to the getContactInstanceTotal():

def contactInstanceTotal = getContactInstanceTotal(getUsername())

Now restart the application, telnet into memcached, and execute the flush_all command. This will reset the cache to a clean state:

telnet localhost 11211
Trying ::1...
Connected to localhost.
Escape character is '^]'.
flush_all
OK

Now open your browser to: http://localhost:8080/contactmanager/contact/list.

After you hit this link once, telnet into memcached and perform a get on the key used by the getContactInstanceTotal(), as shown here:

get my.username:contactInstanceTotal
VALUE my.username:contactInstanceTotal 512 2
'
END

You can see that a value is stored in the cache. You won't necessarily see the value you expected, however. This is because the value is serialized and will not be in String form.

Generating keys

In order to cache the Contact Manager application's Contacts, as you'll do next, you need to be able to create a unique key for each request. If you only use a single key (as was the case with the contactInstanceTotal), then you will end up writing over cached data with each new request. You also need a way to ensure that you can reproduce the key with each matching request.

The simplest and most reliable way to generate keys is to use a digest of the parameters passed into the request with the username pre-pended to the key. Assume that if you receive the same parameters twice, the results should be the same (assuming the data has not changed, which I will discuss in the next section). This technique also guarantees that you will be able to generate a consistent key by generating a digest of the parameters.

Caching data

Now you're ready to start putting data into the cache. As previously mentioned, you'll cache the results of each request to the list closure. The list closure is pretty simple and is used to page through a collection of Domain objects in the database. When a request is made to this closure a collection of parameters is passed to it. These parameters indicate how Grails is to query the database. You can see examples of these parameters by watching the URL change as you page through the application's Contacts. You can also see the parameters in the following URL:

http://localhost:8080/contactmanager/contact/list?offset=10&max=10

This request contains two parameters: offset and max. The result of this request is a list of 10 Contacts. The list starts with the tenth Contact and continues through the nineteenth. Unless the data changes, assume that you will always get the same results from the database, given these parameters. Therefore, you can take the params from this request and generate a digest and store this key/value pair in the cache. Listing 7 shows the method I created for this purpose:

def getCachedContactInstanceList(username) {

    params.max = Math.min(params.max ? params.max.toInteger() : 10, 100)

    println "PARAMS == ${params.toString()}"
    MessageDigest md = MessageDigest.getInstance("SHA");
    md.update(params.toString().getBytes('UTF-8'))
    def key = username + new BASE64Encoder().encode(md.digest())

    println "Using key ${key}."

    def cachedContactInstanceList = memcachedService.get(key) {

        def contactInstanceList = Contact.list(params)

        def serializableList = new ArrayList()

        contactInstanceList.each {

            serializableList.add([id: it.id, firstName: it.firstName, lastName: 
                it.lastName, email: it.email])
        }
        return serializableList
    }
    return cachedContactInstanceList
}

The getCachedContactInstanceList() method is fairly simple. It takes the parameters passed into the controller and converts its String value to a byte array, then passes it to the MessageDigest. It then uses the BASE64Encoder to generate a key with the username pre-pended to the key — thus guaranteeing that you will get a reproducible key given the same set of params.

The only other thing to note about Listing 7 is the iteration over the results, converting the contents of each Contact into a map and storing each of the maps in an ArrayList. This is necessary because Grails Domain objects are not serializable, and maps are.

After you have added this method to your sample Grails application, replace:

def contactInstanceList = Contact.list(params)

with a call to getCachedContactInstanceList():

def cachedContactInstanceList = getCachedContactInstanceList(getUsername())

Restart Grails and page through the Contacts a few more times, repeating a few of the requests. As you are doing this, monitor the consoles stdout. You will notice response times go way down as you repeat requests.

Invalidating the cache

Response times for cached data are going down dramatically, but what will happen when you update the application's contacts? Before adding a new Contact, browse to the very last page and take a look at the Contact list there. Now add a couple of new Contacts and then browse to the last page again. You will notice that the new Contacts are not listed. The problem is that you have already cached the results of the request in which they would have been returned.

The solution is to invalidate the cache — but how? You don't even have a record of the keys that you have already put into the cache. The first thing you need to do is make sure you can look up all the keys you need to invalidate. A simple approach to this would be to cache a list of keys and associate this list to the user.

Try adding the snippet from Listing 8 to the getCachedContactInstanceList() method immediately before returning the cached Contacts:

def getCachedContactInstanceList(username) {

    ...

    // before I return the contacts, I need to add this key to the user's keyList
    def contactKeyList = memcachedService.get(username + ":contactKeyList")
    if (!contactKeyList) {

        contactKeyList = []
    }
    contactKeyList.add(key)
    memcachedService.set(username + ":contactKeyList", contactKeyList)

    return cachedContactInstanceList
}

Next, add the code to invalidate the cache. The method that will invalidate the cache must perform four steps:

• Retrieve the cached keys
• Delete the key/value pairs associated with the retrieved keys
• Delete the array of cached keys
• Delete the key/value pair representing the contactInstanceTotal

Listing 9 shows the complete method to invalidate the cache:

def invalidateContacts(username) {

    // delete the cached contacts
    def contactKeyList = memcachedService.get(username + ":contactKeyList")
    contactKeyList.each {

        memcachedService.delete(it)
    }

    // delete the list of keys
    memcachedService.delete(username + ":contactKeyList")

    // delete the contactInstanceTotal
    memcachedService.delete(username + ":contactInstanceTotal")
}

The code itself is pretty straightforward. It completes the four steps necessary and removes all cached information about Contacts associated with the username. Calls to this method need to be added to all methods/closures that result in a change in the Contacts data. The closures in this application include delete and save. Changes to the closures are shown in Listing 10:

def delete = {
    def contactInstance = Contact.get(params.id)
    if (contactInstance) {
        try {
            contactInstance.delete(flush: true)
            flash.message = "Contact ${params.id} deleted"
            invalidateContacts(getUsername())
            redirect(action: list)
        }
        catch (org.springframework.dao.DataIntegrityViolationException e) {
            flash.message = "Contact ${params.id} could not be deleted"
            redirect(action: show, id: params.id)
        }
    }
    else {
        flash.message = "Contact not found with id ${params.id}"
        redirect(action: list)
    }
}

def save = {
    def contactInstance = new Contact(params)
    if (!contactInstance.hasErrors() && contactInstance.save()) {
        flash.message = "Contact ${contactInstance.id} created"
        invalidateContacts(getUsername())
        redirect(action: show, id: contactInstance.id)
    }
    else {
        render(view: 'create', model: [contactInstance: contactInstance])
    }
}

Notice in Listing 10 that you should not invalidate the cached Contacts until you are certain that the data has been successfully changed. Invalidating the cache at the beginning of either of these closures would risk invalidating the cache prematurely.

Testing the results

After you have made these changes, restart your Contact Manager application and telnet into memcached, where you'll check the results of your updates. The first thing you want to do is reset the cache to a completely empty state, so execute the flush_all command:

flush_all
OK

Next, open your browser and start paging through the Contacts list. As you do this, watch the console. You will see the keys being used to store the Contacts in the cache. Copy a few of these keys for later use, then add, delete, and edit some Contacts. First you will see that the data is being properly displayed as you are modifying it. As your last action, add a new contact and stop your browsing with the Show Contact page.

Go back to your telnet session and do a get on the keys you copied away, the contactKeyList and the contactInstanceTotal. You will notice that all of these values have been removed from the cache and the cache is in the proper state for all upcoming requests.

Back to top

In conclusion

In this article, I've written about one way to effectively incorporate caching into your Grails applications. Using memcached to cache individual request results, as you've done here, keeps all of Grails's built-in paging magic in place. An alternative would be to cache all of the user' contacts, and then write all of your own paging code. This longhand approach would make sense in some cases; for instance, I did it once when working on a GWT/Grails application that passed JSON back and forth. Storing the cached data in its JSON representation made it ever faster, because I did not have to convert the results into JSON. For most purposes, however, the quick-and-dirty technique introduced in this article is effective.

Resources

Learn

• "memcached and Grails, Part 1: Installing and using memcached" (James Goodwill, developerWorks, September 2009): Gets you started with memcached installation, configuration, and client commands, as well as a quick introduction to evaluating the effectiveness of your cache.
  
  
• "Mastering Grails: Build your first Grails application" (Scott Davis, developerWorks, January 2008): Need an introduction to Grails? Start here and build your skills from the ground up.
  
  
• Memoization: Get the Wikipedia definition and some links for further reading.
  
  
• "Memoization in Java using dynamic proxy classes" (Tom White, OnJava, August 2003): An older article that presents a Java Memoizer class.
  
  
• HSQLDB: Learn more about the default database that comes packaged with Grails.
  
  
• "Distributed Caching with Memcached" (Brad Fitzpatrick, Linux Journal, August 2004): Danga Interactive's Brad Fitzpatrick introduces memcached.
  
  
• "Performance tuning considerations in your application server environment" (Sean Walberg, developerWorks, January 2009): An overview of how the various components of a Web application interact, including common performance bottlenecks and solutions such as caching.
  
  
• developerWorks Java technology zone: Find hundreds of articles about every aspect of Java programming.
  
  

Get products and technologies

• Spymemcached: Download it.
  
  

Discuss

• Get involved in the My developerWorks community.
  
  

About the author

James Goodwill is a well-known author and technologist living in the Rocky Mountain region of the United States. His main focus is scaling Java and Grails Web applications. His published titles include Developing Java Servlets, Mastering Jakarta Struts, and Mastering JSP Custom Tags and Tag Libraries.

Rate this article

Comments

Back to top

Table of contents

• The Grails application
• Injecting memcached into the Grails app
• Using memcached
• Caching data
• In conclusion
• Resources
• About the author
• Comments