Connecting to the cloud, Part 2: Realize the hybrid cloud model

Pull JMS queue data to an Amazon SQS queue

This is Part 2 of a three-part series on connecting to the cloud. To determine the best solution for creating a hybrid cloud application, Part 1 examined some of the offerings from the major cloud platform vendors. In this article, Part 2 of the series, you will implement the hybrid cloud application, which combines local application components with cloud computing. The application makes use of a JMS queue locally as well as an SQS queue in the cloud, combining the two in a single hybrid application.

Share:

Mark O'Neill, CTO, Vordel

Mark O'Neill is CTO at Vordel, an XML Networking company. He is also author of the book "Web Services Security" and contributing author to "Hardening Network Security", both published by McGraw-Hill/Osborne Media. Mark is responsible for overseeing Vordel's product development roadmap and also advises Global 2000 firms and governments worldwide on their tactical and strategic adoption of XML, Web Services and SOA technologies. He holds degrees in mathematics and psychology from Trinity College Dublin and graduate qualifications in neural network programming from Oxford University. Mark lives in Boston, Massachusetts.



28 April 2009

Also available in Chinese Japanese Vietnamese Spanish

The hybrid model

For this article, I narrowed the focus for creating a hybrid cloud application to one cloud provider: Amazon. The example application, called HybridCloud, takes data from a JMS queue and passes the data up to an Amazon SQS queue, which is hosted in the Amazon cloud. Once in the Amazon SQS queue, the data can be pulled down by another application with the rights to that queue.

To put this into context, look at an example. Consider a medical organization that requires processing of X-ray images by an image-processing application. The medical organization securely writes the X-ray images to an Amazon SQS queue, where they are temporarily stored. Even if the file size of each X-ray image is very large, that is not a problem for a cloud computing provider. At the same time, the image-processing application consumes the queue, reading in the X-ray images as they are placed there. The image-processing application, which requires significant processing power, is ideally suited to deployment in a virtualized environment (a private cloud) to which hardware capacity can be added quickly and easily. Once processed by the image-processing application, the X-ray images are placed back onto a different queue, to be received back by the medical application. Although X-ray images are the subject matter of this example, any data can be used. Clearly, X-ray images introduce significant privacy and security issues. Part 3 of this article series will examine security and governance for the cloud.

Frequently used acronyms

  • FTP: File Transfer Protocol
  • API: application programming interface
  • HTTP: Hypertext Transfer Protocol
  • JMS: Java™ Message Service
  • JNDI: Java Naming and Directory Interface
  • REST: Representational State Transfer
  • SQS: Simple Queue Service
  • URL: Uniform Resource Locator
  • XML: Extensible Markup Language

The advantages of this architecture include:

  • The system does not rely on the queue-writer (the medical application) or the queue-reader (the image processing application) being online at the same time. If the files were simply FTPed between the two applications, the system breaks down if either party goes offline. By using the Amazon SQS queue, the system has resiliency.
  • Capacity can be added at either side of the solution, without affecting the working of the system as a whole. For example, you can increase the computing capacity on the image-processing application side, which might then double the image processing speed. This change is transparent to the medical applications, which continue to write their X-ray image files to the Amazon SQS queue.
  • This hybrid model is the widely-recommended route for architects who are considering the use of cloud computing. Rather than an all-or-nothing solution that either puts everything up onto a cloud computing platform, or completely eschews cloud computing as a whole, the better option is to choose what is useful about cloud computing and use that tactically. On the local application side, virtualization (a private cloud) makes it relatively easy to add capacity dynamically, which is important for processor-intensive applications like image processing. On the cloud side, the Amazon SQS service provides a queuing service between applications, even when they are on unconnected networks.

Identifying yourself to the Amazon SQS cloud service

Your HybridCloud Java application uses Amazon SQS (see Download for the application source code). The connection from the HybridCloud application up to the Amazon SQS cloud uses Web services, which is authenticated. There are two broad reasons for this authentication: first, Amazon charges money for the use of Amazon SQS, and therefore must keep track of individual usage; second, access to individual queues must be controlled. In the case of our HybridCloud application, clearly it would not be appropriate for third-parties to access the Amazon SQS queues containing private X-ray images.

The first step in order to use the Amazon SQS queue is to become a user of Amazon Web Services (AWS), which requires an Amazon.com login. Possession of an Amazon.com login is not unusual—anyone who has ever purchased a book on the Amazon.com site already has an Amazon.com login. However, the Amazon.com login is just the first step. AWS requires the use of an "access key ID" and an associated secret key.

The Secret Access Key in the Amazon Web Services model can be thought of as a shared secret between Amazon.com and the developer who is connecting to Amazon Web services. The Access Key ID, by contrast, is not secret, since it is used for identification not authentication. In Part 3, you will examine these two keys in more detail, as well as alternative authentication models used by other cloud computing providers.

Once a developer has signed up for Amazon Web Services, and obtained their Access Key ID and their Secret Access Key, they can then subscribe for particular services. In this case, we are going to subscribe to the Amazon.com SQS service. A developer must register a credit card with Amazon Web Services in order to use services such as SQS. The credit card details must be stored with Amazon, they cannot simply be entered when a bill is due—it is a pay-as-you-go model. If you attempt to access the SQS service without previously registering for the service, you will see an exception with the text "The AWS Access Key Id needs a subscription for the service".


Designing your hybrid application

As described at the outset, the hybrid application processes data locally and makes use of the Amazon SQS cloud service. Therefore, it has a local component and a cloud component. In this application design, data is retrieved locally from a JMS Queue (perhaps from a mainframe application) and then sent up to the Amazon SQS queue using HTTP GETs and POSTs. You will use Java as the language of your application.

The application has three main parts:

  1. Create the Amazon SQS queue.
  2. Read local data (from a JMS queue) and put it onto the Amazon SQS queue.
  3. Retrieve response data from the Amazon SQS queue.

In the code snippets for the application in Making use of Amazon SQS, notice the similarities between the handling of the connection to the local JMS queue and the connection to the Amazon SQS queue.


Making use of Amazon SQS

In your HybridCloud Java application, first pull in the Amazon SQS classes (see Listing 1).

Listing 1. Importing Amazon SQS classes
Import com.amazonaws.queue.*;

The HybridCloud application uses Amazon SQS to write data to a queue, and then to later read data back from the queue. The connection to Amazon SQS is over an authenticated Web services connection, where the client is identified using the Amazon Access Key ID and authenticated using the Amazon Secret Key ID. To use the Amazon SQS from your Java code, first put the Amazon Access Key ID and Amazon Secret Key ID into the code (see Listing 2).

Listing 2. Setting Amazon keys
String accessKeyId = "12345678901234567890";
String secretAccessKey = "abcdefghijklmnopqrstuvwxyz";

Next, instantiate the HTTP Client implementation of Amazon SQS, passing in the Access Key ID and Secret Access key as variables (see Listing 3).

Listing 3. Instantiating the Amazon SQS http client
AmazonSQS service = new AmazonSQSClient(accessKeyId, secretAccessKey);

At this point, you now have instantiated an Amazon SQS client object, but you have not yet connected to the Amazon SQS service over the Internet. That is the next step, in which you create the queue on the Amazon SQS cloud service.


Creating the Amazon SQS queue

This application makes use of a queue to store X-ray image files. Before you can use an Amazon SQS Queue, you must create it. The queue must have a name, which, in this case, is "imageQueue". This is where your HybridCloud application puts its X-ray image files (see Listing 4).

Listing 4. Creating an Amazon queue
CreateQueueRequest request = new CreateQueueRequest();
request.setQueueName("imageQueue");

You now run a method of the Amazon SQS service object called createQueue, which creates the HTTP GET (REST-style) request to Amazon Web Services to create your queue (see Listing 5).

Listing 5. Creating an Amazon queue
CreateQueueResponse response = service.createQueue(request);

This creates a REST-style invocation of a Web service on Amazon SQS. When you run the code to create the Amazon SQS queue, it is interesting to examine the REST-style URL which is sent on the wire to Amazon SQS. In this URL, you can see the Access Key ID in the clear, the signature (makes use of the Secret Key ID), and the actual SQS queue, which is created and called imageQueue. The actual Secret Key ID is not, of course, sent. The signature component of the message (see Listing 6) provides the proof of possession of the Secret Key ID. Only the holder of the key could have created the signature component.

Listing 6. REST Style Web service request sent to Amazon Web Services
queue.amazonaws.com?Action=CreateQueue&SignatureMethod=HmacSHA256&AWSAccessKeyId
=12345678901234567890&QueueName=imageQueue&SignatureVersion=2&Version
=2008-01-01&Signature=U859J2Hoi5qBqlQx1R18dKPgSgrgjlOiJIDD8ug9FPI%3D&Timestamp
=2009-04-01T03%3A25%3A13.575Z

The signature is created using not only the QueueName, but it is also computed over a timestamp. This ensures that the request to Amazon SQS cannot simply be captured by an attacker and replayed, thus mimicking the valid user. If an attacker re-sends the request to the Amazon SQS service, the repetition of its signature identifies the request as a capture-replay attack, and Amazon Web Services blocks the it.

At this point, you have created the imageQueue queue, into which you will load image data. You now source that data from a local JMS queue. This local queue can itself come from a virtualized environment (a local cloud).


Retrieve data from the local JMS queue

Now that you've created you Amazon SQS queue, turn your attention to the local aspects of this hybrid application. You must read data from a local queue, prior to feeding the data to the Amazon SQS queue. It is instructive to compare the steps used for the local queue with the steps required to connect to Amazon's SQS queue.

The Java libraries required are imported, first all of the JMS jars, and then the JNDI jar. These files allow your Java application to make use of JMS (see Listing 7).

Listing 7. Importing JMS classes
import javax.jms.ConnectionFactory;
import javax.jms.Connection;
import javax.jms.Session;
import javax.jms.MessageProducer;
import javax.jms.MessageConsumer;
import javax.jms.Queue;
import javax.jms.Session;
import javax.jms.Message;
import javax.jms.TextMessage;
//Required for JNDI.
import javax.naming.*;

You use JNDI to set the context for your JMS connection. For example, if you read data from the file system, you can use the code in Listing 8.

Listing 8. Reading files from a local queue
ConnectionFactory myConnFactory;
Queue myQueue;
Hashtable env;
Context ctx = null; 
env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.fscontext.RefFSContextFactory");
env.put(Context.PROVIDER_URL, "file:///C:/Images");	
ctx = new InitialContext(env);

Now create your queue and connection factory (see Listing 9):

Listing 9. Create the queue and connection factory
String MYCONNECTIONFACTORY = "MyConnectionFactory";
String MYQUEUE = "MyQueue";
myConnFactory = (javax.jms.ConnectionFactory) ctx.lookup(MYCONNECTIONFACTORY);
myQueue = (javax.jms.Queue)ctx.lookup(MYQUEUE);

Next, create a connection, and then a session within the connection (see Listing 10).

Listing 10. Creating a connection to the local JMS queue
Connection myConn = myConnFactory.createConnection();
Session mySess = myConn.createSession(false, Session.AUTO_ACKNOWLEDGE);

To read the message from the queue (which in this case ultimately involves reading from the file system since we use the JNDI identifier for the file system), use the code in Listing 11.

Listing 11. Reading from a local JMS queue
MessageConsumer myMsgConsumer = mySess.createConsumer(myQueue);
myConn.start();
Message msg = myMsgConsumer.receive();

Now examine the contents of the message that you retrieved locally (see Listing 12).

Listing 12. Examining the contents of the locally retrieved message
if (msg instanceof TextMessage) {
                TextMessage txtMsg = (TextMessage) msg;
                String inputText = txtMsg.getText());
}

Finally, close the connection to the JMS Queue (see Listing 13).

Listing 13. Closing the connection to the local JMS queue
mySess.close();
myConn.close();

At this point, you have the data that you want to write to the Amazon SQS queue in the string called inputText. This is what you write to the Amazon SQS queue next.


Writing to the Amazon SQS queue

The data passed to the Amazon SQS queue is sent as a string. Although your data is an image, it is still sent up to Amazon SQS as a string. You see it set below in the SendMessageRequest object used to send the message to the Amazon SQS queue (see Listing 14).

Listing 14. Sending a message to the Amazon SQS queue
SendMessageRequest sendRequest = new SendMessageRequest();
sendRequest.setMessageBody(inputText);

You must also set the appropriate queue name, which is the same as the queue name you created earlier (see Listing 15).

Listing 15. Setting the queue name on Amazon SQS
sendRequest.setQueueName("imageQueue");

Now send your message to the Amazon SQS Queue by using the sendMessage method of your Amazon SQS object (see Listing 16).

Listing 16. Sending our data to the Amazon SQS queue
SendMessageResponse response = service.sendMessage(sendRequest);

Reading the Amazon SQS response

The next step is to read the response from the Amazon SQS Queue. First create a ReceiveMessageRequest object (see Listing 17).

Listing 17. Creating a ReceiveMessageObject to retrieve data from the Amazon SQS queue
receiveRequest = new ReceiveMessageRequest();

Next, set the queue name (see Listing 18).

Listing 19. Creating a ReceiveMessageObject to retrieve data from the Amazon SQS queue
receiveRequest.setQueueName("imageQueue");

Now use the receiveMessage method to attempt to receive a message from the Amazon SQS queue (see Listing 19).

Listing 19. Receiving a response from the Amazon SQS queue
ReceiveMessageResponse response = service.receiveMessage(receiveRequest);

To examine the result, including the message ID and message body of the messages retrieved from the cloud (see Listing 20), pass it to standard output.

Listing 20. Examining the response from the Amazon SQS queue
if (response.isSetReceiveMessageResult()) {
ReceiveMessageResult  receiveMessageResult = response.getReceiveMessageResult();
java.util.List<Message> messageList = receiveMessageResult.getMessage();
for (Message message : messageList) {
if (message.isSetMessageId()) {
                        System.out.print("            MessageId");
                        System.out.print("                " + message.getMessageId());
                        System.out.println();
                    }
                    if (message.isSetBody()) {
                        System.out.print("            Body");
                        System.out.println();
                        System.out.print("                " + message.getBody());
                        System.out.println();
                    }
}

Using an XML gateway to connect local applications to the cloud

In this article, you used Java code to connect a local application to a cloud computing environment. This is an ideal solution for developers, but as part of an application, the connection to the Amazon cloud computing service does not come under the control of a network operations team, who might monitor usage, traffic, and availability. Additionally, any change to the cloud computing connection involves a code change. Although it is outside the scope of this article, it's possible to achieve the same connection between a local JMS queue and the Amazon SQS cloud service using a network infrastructure, that places the connection under the control of network operations staff.

To connect a local application component, such as a JMS queue, to the cloud, requires an XML gateway that includes cloud computing connectors (see Resources for an example of such a gateway). The task of connecting a JMS queue to an Amazon SQS queue involves dragging-and-dropping connection filters together, so data is pulled from the JMS Queue and then trafficked to the Amazon cloud service.

A useful analogy is the familiar cable TV box. This box connects local infrastructure (televisions) to the cloud (the cable TV operator). An XML gateway provides the role of the local cable box. In addition to providing the connection to the cloud services, it also provides monitoring and security for that connection.


Summary

You saw how a Java application can bridge a local JMS Queue and the Amazon SQS cloud. Your HybridCloud application uses local resources (a JMS Queue) and cloud-based resources (Amazon SQS) to perform an example image-sharing task. Relatively similar methods connect to the local queue and to the Amazon SQS queue, although on the network level, the connection to Amazon SQS is sent over HTTP GETs and PUTs with parameters passed within URLs.

In Part 3, the final part of this series, you will examine the issues of governance and security that come with cloud computing. You will see the authentication models used by various cloud providers, and examine the challenges which arise from privacy, regulatory compliance, and protection from Denial-of-Service threats.


Download

DescriptionNameSize
Hybrid Cloud Sample AppHybridCloud.zip3KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into XML on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=XML, SOA and web services
ArticleID=384952
ArticleTitle=Connecting to the cloud, Part 2: Realize the hybrid cloud model
publish-date=04282009