Configure and run a multiregion Bluemix application with IBM Cloudant and Dyn
Incorporate geographic load balancing, active failover, and master-to-master data replication into your app
IBM Bluemix instant runtimes (based on Cloud Foundry) support application availability by running multiple instances of your application on separate nodes within a single region. However, to deliver a truly robust cloud application on Bluemix, you should consider a multiregion architecture that takes advantage of the global reach of Bluemix.
Currently, delivering a proper multiregion application on Bluemix (one that load balances across the regions) requires an external router or routing service. However, even when you get this working, you still have the challenge of keeping your dynamic content in sync across the different regions. With a single database approach, your application can suffer from database latency issues. With a distributed database approach, you might find that your distributed database isn't designed to be distributed across the globe.
I'll show you how to configure and run a multiregion Bluemix application by using:
- IBM Cloudant NoSQL Database's master-to-master replication to synchronize your data
- Dyn Managed DNS with Traffic Director to provide geographic load balancing and failover
Register a domain
To use Dyn Managed DNS, you must control a domain that you can delegate to your assigned Dyn nameservers. If you already own a domain that you can delegate, skip to Step 2. If not, register for one with any certified domain registrar, such as Dyn's domain registrar.
Create your application in multiple Bluemix regions
- Navigate to Bluemix (register for your free trial account or log in to Bluemix if you already have an account).
- From the Bluemix console, expand the Account and Support panel in the
upper right and note your current region. As of the writing of this
article, IBM Bluemix (Public) is hosted in four different regions:
Each Bluemix region is operated independently, but with a unified billing and account system. After completing the following steps 3-6, you will need to repeat them in the other Bluemix regions.
Region Location API endpoint Germany Frankfurt, Germany https://api.eu-de.bluemix.net Sydney Sydney, Australia https://api.au-syd.bluemix.net United Kingdom London, England https://api.eu-gb.bluemix.net US South Dallas, Texas https://api.ng.bluemix.net
- Select Manage Organizations, and then select the DOMAINS tab and add your domain:
- If you want to serve HTTPS traffic (in addition to HTTP), then you also need to obtain a wildcard SSL Certificate for your domain (for example, *.ibmjstart.biz) and upload it to Bluemix. (Find more information at Securing apps.)
- Click SAVE, and then navigate to the CATALOG and create a new instance of the Node.js Cloudant DB Web Starter boilerplate. Give the application a name, and ensure that your domain is selected from the Domain drop-down list:
- While the application is staging, navigate to the
Overview page and add a second route with the
default domain of your Bluemix region (for example, mybluemix.net).
This route gives you a back door for accessing the version of your application that is running in this specific region before or after configuring DNS for your custom domain. However, don't forget to remove this route in the future if you want to restrict access to a single URL.
Now repeat steps 3-6 for each of the Bluemix regions where you want to host your app, giving the application the same name and domain in each region.
Configure Cloudant replication
To synchronize the application data across regions, configure each Cloudant database to continuously replicate from the database in each of the other regions.
This can be accomplished manually from the Cloudant dashboard according to the steps below, or it can be automated through the use of this command-line utililty created by the IBM jStart team.
Note: Configuring continuous replication as described in this article will result in frequent API calls between the configured regions. Please consider setting Spending notifications to avoid unexpected charges.
- From application details page in each region, navigate to the
Connections view, click the View credentials button
of your Cloudant NoSQL DB service tile, and note the username and
To keep the different Cloudant accounts straight, consider
temporarily writing them down with a format similar to the following:
Region Username Password URL US South xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix United Kingdom xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix Sydney xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix Germany xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx-bluemix
- After noting the password, click the tile itself and then LAUNCH the Cloudant Dashboard. Each region's Cloudant instance should have a single database with the name my_sample_db.
- In each Cloudant dashboard, open the Permissions view for the my_sample_db database. Grant Replicator permissions for the Cloudant user names in each of the other regions, or use the Generate API key button to create a key/pass combo for each of the other regions to use.
- Next, open the Replication section for each region,
and append "/my_sample_db" to the end of the URL (including username
and password) of the Cloudant DB in the other regions to configure a
continuous replication task from each of the other regions:
This should create a new database in each region with the name "_replicator" and a single document per replication task.
Test the replication
After you have configured the replications, test them by opening your application in each of the configured regions using the "back door" routes that you assigned earlier:
Upload a file to one region and refresh your app in the other regions to see the replication in action.
Master-to-master data replication is a great feature of Cloudant/CouchDB, but you should consider what to do about document conflicts. The default strategy is to pick a conflict winner at random, but applications are encouraged to handle conflicts for themselves. The easiest way to do this is to architect your application to avoid conflict by ensuring unique document _ids and avoiding document updates.
Deploying updates to your application
From any one of the Bluemix regions, navigate back to the application Overview page, click the Enable button from the Continuous delivery tile, and proceed to Create a Continuous Delivery Toolchain.
Once the toolchain has been created, click the Delivery Pipeline tile. The project already has a working pipeline for the region where you created the project, but you might want to configure the Deploy Stage to deploy to additional Bluemix regions.
You can deploy to additional Bluemix regions by adding additional jobs to the Deploy Stage. Be sure to set the Target properly for each region, and save the new configuration.
Now that the pipeline is configured, let's try it out with a small change. Navigate back to the Toolchain, click the Eclipse Orion Web IDE tile, click EDIT CODE near the top of the page, open the manifest.yml file in the built-in editor, and modify the memory threshold and the number of instances:
applications: - path: . memory: 256M instances: 2 ...
This code tells Bluemix to run 2 instances of the application (per region) instead of 1. With or without DNS failover, running multiple instances of your application is strongly encouraged to avoid unexpected downtime in the event of isolated incidents or platform maintenance.
Note: Non-trial Bluemix users currently receive a free tier that corresponds to running at 512 MB throughout the month. Because usage is aggregated across the regions, this means that the configuration described above will put you beyond this tier.
Switch to the GIT view, commit, and push your changes.
Pushing your changes should automatically invoke the delivery pipeline, deploying your update to all of the configured regions. Navigate back to the delivery pipeline to check the progress.
To access your application through your custom domain, you must first configure the Domain Name System to resolve your route to Bluemix. To do so using Dyn Managed DNS, register for a 7-day free trial of their Managed DNS offering. Alternatively, if you want to configure geographical load balancing and failover, contact Dyn for the pricing and availability of their Enterprise plan.
After you have registered, log in to the Dyn Managed DNS portal and create a new zone for your custom domain.
Note the name servers assigned to you by Dyn. You will need to add these as NS records in your DNS registrar from Step 1 to delegate your domain to Dyn. Further information on delegating your domain can be found by:
- Delegating your dyn.com domain
- Delegating your domain from a different registrar
After you have created a zone for your domain and delegated that domain to Dyn, it's time to configure a subdomain for your Bluemix application. You can do this by adding a new node that matches the host name of your application in Bluemix.
If you have enrolled in the enterprise plan, skip to the Traffic Director section to configure load balancing and failover for your node. If not, you will not be able to configure load balancing without an additional routing layer, but you may still proceed to add a single CNAME Record from the Dyn node editor:
Set the CNAME field to the secure endpoint of your preferred Bluemix region:
Note: These CNAME records resolve to a different IP than the default "mybluemix" domains in each region. If you wish to configure A Records instead, be sure to use the IPs of these secure CNAME endpoints in order to support HTTPS from your custom domain.
Dyn Traffic Director is a highly configurable DNS traffic management service that provides geolocation load balancing and failover through Response Pools, Rulesets, and Monitors. To configure Traffic Director for your Bluemix application, select it from the "Add a New Service" drop-down list on the Dyn node editor, give it a label, and click CREATE SERVICE.
Next, create a separate Response Pool for each of the Bluemix regions:
Using the previous table, add a CNAME Record with the secure endpoint of the corresponding Bluemix region for each response pool.
Leave the defaults for most of the fields, but consider modifying the "Record Serve Mode" to "Monitor & Remove." With a value of "Monitor & Obey," failover traffic will be automatically restored to the response pool after a single successful connection from the monitor. This can be problematic for issues characterized by repeated interruption in service. By setting the mode to "Monitor & Remove," recovery will become a manual intervention.
Next, add a Ruleset for each geography to configure which response pool is used to serve which requests.
As a starting point for your Rulesets, you can copy the following configuration:
|Ruleset||Geographic areas||Response pool failover|
|America||North America, South America||US South > United Kingdom > Sydney|
|Europe/Africa||Africa, Europe, and Russia||United Kingdom > Sydney > US South|
|Asia/Pacific||Antarctica, Asia, and Australia||Sydney > United Kingdom > US South|
|Match All||You can leave this empty to match undefined areas||Sydney > United Kingdom > US South|
In addition to the geographic configuration, the Response Pool Failover field lets users customize which Response Pools to consult in the case of failure. To use this feature, you must also configure a Monitor for your application.
To configure where the Monitor will probe, leave the Protocol as HTTP, set the Port to 80, and specify the Host for your Bluemix app. Set the Probe Interval low to shorten the potential downtime, and configure the Expected Data field with information that will appear in the working app. In the case of our sample app, you can use the text "Favorites Organizer powered by Cloudant."
Tip: In a real scenario, you might want to include a dedicated status page in your application that returns only the expected text if all of the application's service dependencies are also functioning.
Optionally, configure a notifier to receive an email when the Monitor detects a break in service. You might also want to open the Probe Results link in a separate tab to verify the status checks for your site as you test the configuration.
After configuring the Response Pools, Rulesets, and Monitors, publish your changes for both the Traffic Director and your zone. If everything is configured correctly, you should see the Traffic Director service (and the status of your Response Pools) listed in the Node configuration editor:
Try it out
- Load your application in a browser, using your custom domain this time, and upload a file. Refresh the page to ensure the file was uploaded properly.
nslookupfrom a command terminal (or use your browser's developer tools) to see which region is serving your request.
If you are using Traffic Director for geographic load balancing and failover, proceed to:
- Log in to the Bluemix console for the region serving your requests and stop the application.
- Refresh your application in the browser, and you should receive a 404 Error.
- Refresh the Dyn website with your Node Editor, Traffic Director, or Probe Results page open until you see the status updated. (Note that this will depend on the Probe Interval you configured above.)
- After Dyn has identified the failure, the region will fail over to the next Response Pool for your geography.
nslookupfrom a command terminal to see if your DNS changes have taken effect. The site should resolve to its new address at some point within the TTL (time to live) that's configured for your A Records.
- Once the site is resolving to its new IP, load the page in a new browser. Note that some browsers, like Google Chrome, are known to cache the DNS response even beyond the configured TTL. However, even in these browsers, the new IP should take effect within a few minutes.
- Your site should now load from the configured failover region. Thanks to Cloudant, the file you uploaded in the original region should be waiting there for you!
Limitations of DNS failover
While Dyn Traffic Director provides a useful approach for mitigating region-specific outages, it is worth noting that DNS failover comes with significant limitations:
- The Dyn Monitor polls your application at a maximum of once per minute.
- DNS TTL statements shorter than 30 seconds might not be respected on the web.
- Some browsers, like Google Chrome, implement "DNS Pinning," which will cache the DNS response even longer than the configured TTL (up to 5 minutes per my local testing).
When combined, this represents a worst case window of 6.5 minutes (when the user's browser pins an IP just before failover occurs), although the average window should be significantly shorter.
Additionally, in some failure scenarios, the application might go up and down over the course of the incident. To avoid restoring traffic in the midst of these scenarios, I recommend using the "Monitor & Remove" serve mode to take care of the traffic restoration behavior.
Note: Although not explored in this tutorial, you can further reduce the 60-second monitoring window by invoking the failover from your own custom monitor via the Dyn API. This can also be used to implement more sophisticated rules in regard to the automatic restoration of failover traffic.
Using Dyn's geographic load balancing, visitors to your URL from across the globe will be served by the Bluemix data center most appropriate to their location. By configuring failover, you'll have peace of mind knowing that if one of the Bluemix regions begins to experience an issue, your users will be directed to one of the working sites instead. When combined with the Cloudant master-to-master synchronization capabilities, this provides a recipe for running fault-tolerant, cloud-native applications on Bluemix.
- More about Cloudant replication
- More about conflict resolution
- The Cloudant NoSQL service
- IBM Bluemix DevOps Services