How IBM PureApplication System uses DNS to deploy workloads onto the data center's network
IBM PureApplication System (W1500 and W1700 v1.0 and v1.1) is a cloud computing system-in-a-box, with integrated hardware and software to deploy and execute workloads in a cloud – in other words, everything an enterprise data center needs to add a private cloud environment. Like any computer connected to a TCP/IP network, PureApplication System uses DNS to locate resources on the network so that it can connect to them.
This article describes how DNS is used by IBM PureApplication System v1.1 (referred to hereafter as “the system”). It explains how to configure the system’s network settings to use the data center’s DNS servers, the dependencies the system has on DNS, what queries it expects DNS to be able to answer, and what goes wrong with the system when it can’t use DNS properly.
This information is intended primarily for:
- Network administrators who make DNS servers available on the data center’s network and ensure they are configured with the necessary DNS data.
- IBM technical professionals who help clients configure their system and connect it properly to the data center’s network.
The illustrations included in this article reflect v126.96.36.199. To perform the configurations shown here, you need these permissions:
- Hardware administration with Manage hardware resources (Full permission).
- Cloud group administration with Manage cloud resources (Full permission).
The next sections explain what DNS is and where it’s specified in the system. Later, you will see how these settings are used and how to troubleshoot when things are not properly set up.
What is DNS?
DNS (Domain Name System) is a standardized optional feature of Internet
Protocol (IP). A DNS server listens for requests on port 53 (both UDP and TCP). It translates
domain names (like
mycomputer) into IP addresses (like
and back. Translating a host name to an IP address is called a forward lookup,
whereas a reverse lookup uses an IP address to look up its host name.
A particular network’s DNS server is configured with a set of data specific to the locations of the resources on its network. This data, called DNS records, contains mappings between host names and IP addresses. DNS records have a time to live (TTL) property, which specifies the maximum amount of time DNS clients should cache the record.
The DNS server is configured and made available on the network by the IP network administrator. The administrator configures a DNS server with a set of DNS records and hosts the server on the network on a particular IP address. For availability, a second DNS server can be hosted in addition to the first one. The two DNS servers are attached to the network with different IP addresses but contain the same DNS records, and so provide the same lookup answers. A DNS client that wants redundancy can be configured to use one DNS server as a primary and the other as a secondary. The two DNS servers can be fronted by a third IP address, a so-called virtual IP address (VIP), which acts as a proxy to the two DNS servers’ addresses. This way, DNS clients can be configured with the VIP and automatically connect to whichever server the VIP chooses for load balancing and failover.
For a client to be able to use a DNS server successfully, there are two main challenges to overcome:
- Connectivity: The client must be able to connect to the DNS server across the
This means that the client’s IP address must have a pathway to the DNS server’s IP address. If the client is connected to the network as a member of a virtual local area network (VLAN), the VLAN must have a pathway to the DNS server’s IP address. A DNS server listens for requests on port 53, so any firewalls along the pathway must permit valid DNS packets to pass through on port 53.
- Content: The DNS server must contain the DNS records that map the network resources to
which the client wants to connect.
If the client expects to connect to certain host names, the IP addresses for those host names must be defined in the DNS server. If the client wants to be able to lookup a host name from its IP address, the DNS server must define reverse lookups.
When DNS is working properly, the client can connect to the DNS server and perform lookups that convert host names to IP address and vice versa.
DNS network settings
IBM PureApplication System does not implement or contain a DNS server; rather, it requires access to at least one DNS server on the data center’s network. The network settings, including the addresses for the DNS servers, are configured in two places:
- PureSystems Manager, in its system settings.
- Each IP group.
The PureSystems Manager, the central management component within PureApplication System, uses DNS, which means that the IP address for the PureSystems Manager must be able to reach the DNS server on the system management VLAN.
When IBM installs the system, the PureSystems Manager is configured with the network settings for connecting to DNS. Configuration is performed using the system setup wizard, which uses the configuration data gathered in the Technical and Delivery Assessment (TDA). Therefore, if you’re using a system that’s already been setup and attached to the data center’s network, the system’s PureSystems Manager should already be configured with a DNS server. Nevertheless, if the system seems unable to use DNS successfully, then you need to understand these settings so you can verify they are correct and debug what might be wrong.
The PureSystems Manager must be able to reach the DNS server on the system management network VLAN. Let’s review where you find each of these settings.
The DNS server used by the PureSystems Manager is configured on the System
Settings page (navigate to System Console > System > Settings
> Domain Name Service (DNS)). Enter the IP address (not a host name) for at
least one DNS server. You can specify multiple entries and they will be searched in the order
listed. Figure 1 shows a system configured with a DNS server on
Figure 1. Domain Name Services (DNS) settings
The DNS settings page also includes a Lookup widget to look up host names and IP addresses. You will use this in Troubleshooting.
The DNS servers must be reachable on the system management network, which is the VLAN assigned
to port 64 in the top of rack switches (ToRs). This network can be any valid VLAN on the data
center’s network, although it should be different from the VLANs used for application data. That
VLAN ID is configured on the Customer Network Configuration page in the
Management LAN Port section (navigate to System Console > System >
Customer Network Configuration > Management LAN Port). Figure 2 shows a system
configured with a system management network whose VLAN ID is
Figure 2. Management LAN Port – Aggregate Port 64 settings
The PureSystems Manager’s IP addresses are configured in the same Management LAN Port section.
The PureSystems Managers use three IP addresses, one for each PureSystems Manager (the system
contains two: used active/standby) and a VIP (also known as floating IP address) that is used to
access whichever one is currently primary. Figure 3 shows a system configured with a management
IP address of
172.18.112.32 (and a corresponding host name of
rack16psm.purescale.raleigh.ibm.com; this mapping should be included in the DNS
Figure 3. Management LAN Port – Aggregate Port 64 settings
Again, to summarize, the IP address for the PureSystems Manager must be able to reach the DNS server on the system management VLAN. In this example:
- Figure 1 shows a DNS server at 172.17.248.2.
- Figure 2 shows a system management network on VLAN 716.
- Figure 3 shows a PureSystems Manager on 172.18.112.32.
This means that any TCP/IP client computer running on the PureSystems Manager’s IP address (172.18.112.32 in Figure 3) needs to have a path on the management network (VLAN 716 in Figure 2) through port 53 to reach that DNS server’s IP address (172.17.248.2 in Figure 1). If the PureSystems Manager can’t reach the DNS server, it can’t use DNS.
IP group’s DNS network settings
Each of the IP addresses in an IP group must be able to reach the DNS server on the IP group’s VLAN. Let’s review where you find each of these settings.
You can view an IP group on the IP Groups page (navigate to System Console > Cloud > IP Groups and select the IP group). The tables below show the data displayed for an IP group.
As shown in Table 1, each IP group contains settings for:
- A pair of DNS servers, for a primary server (required) and a secondary server (optional).
- A VLAN ID.
- A network address.
Table 1. IP group network settings
The VLAN must be defined on the network with that address, and the DNS servers’ IP addresses must be reachable on that VLAN.
Be aware that the DNS server specified for an IP group can be different from that specified for the PureSystems Manager. Furthermore, each IP group can specify a different DNS server. The system requires only one DNS server, where that server is used for the PureSystems Manager and all IP groups. But the system also supports a network architecture where the PureSystems Manager has one DNS server and each IP group also has its own DNS server.
As shown in Table 2, each IP group contains a list of IP addresses. When the system console displays an IP group, it lists not only the IP addresses but also each one’s corresponding host name. The system gets the host name for each IP address by performing a lookup using the DNS server.
Table 2. IP group network settings
|Status||IP address||Host name||Actions|
In Table 2, an IP group also shows not only the host name of each IP address, but also whether the IP address is currently being used (by a deployed virtual machine). An "*" in the Status column indicates use. An "X" in the Actions column indicates the user can delete an IP address from the IP group. An address can be removed only if it is not in use.
Again, each of the IP addresses in an IP group must be able to reach the DNS server on the IP group’s VLAN. In this example, this means that each of the IP addresses listed in Table 2 needs to have a path on the specified VLAN (VLAN 636 in Table 1) through port 53 to reach that DNS server’s IP address (188.8.131.52 in Table 1). If the IPs can’t reach the DNS server, pattern deployment using those IPs will go badly (see Troubleshooting).
How DNS is used
Now that you know which parts of the system configure network settings including DNS, you might ask: Why does the system need DNS? Why does both the PureSystems Manager and the IPs in an IP group need DNS? What data does the DNS server need to contain?
DNS comes into play in three aspects of using the system. The first two aspects require DNS, DNS is optional in the third. After reviewing how the system uses DNS, we’ll look at the data the DNS server needs to contain so that it can answer its queries.
Displaying IP groups
The main task for which the PureSystems Manager uses DNS is to display IP groups. As shown in Table 2, when the system console displays an IP group, it shows both the list of IP addresses and each address’ corresponding host name. The IP group object itself only contains the list of IP addresses. When the system console displays the list, it uses the DNS server specified in the PureSystems Manager’s settings to look up each IP address’ host name.
When the system deploys a pattern — provided the environment profile used to deploy the pattern has its IP addresses provided by property set to IP Groups (which is typical and makes the deployment better automated than the Pattern Deployer setting; see Creating environment profiles in the IBM Knowledge Center) — the deployment engine uses the DNS server. Each time the engine creates a virtual machine (VM), it gets an IP address for it from the IP group specified by the deployer, and then gets the host name for the VM by looking up the IP address in the DNS server specified in the IP group.
Furthermore, when the system restarts a VM, the virtualization manager uses the DNS server. This occurs for every VM when it is restarted, whether or not its IP address was obtained from an IP group.
Table 3 shows the steps involved in deploying a simple virtual system pattern containing a single Core OS part. These steps are shown in the pattern instance’s History section during and after deployment. Many of the steps make use of DNS. One in particular is the maestro script package. As Table 3 shows, this script package normally runs very quickly (11 seconds in this example). As you will see in Troubleshooting, when DNS isn’t set up properly, deployment will either fail entirely or will succeed but take much longer than it should.
Table 3. History of a pattern deployment
|The virtual system has been deployed||May 1, 2014, 4:13:58 PM|
|Script package Must Gather Logs on virtual machine VM123 completed successfully||May 1, 2014, 4:13:58 PM|
|Executing script package Must Gather Logs on virtual machine VM123||May 1, 2014, 4:13:40 PM|
|Script package maestro on virtual machine VM123 completed successfully||May 1, 2014, 4:13:39 PM|
|Executing script package maestro on virtual machine VM123||May 1, 2014, 4:13:28 PM|
|Executing script packages||May 1, 2014, 4:13:28 PM|
|Starting virtual machine VM123||May 1, 2014, 4:11:45 PM|
|Starting virtual machines in virtual system VSys123.||May 1, 2014, 4:11:45 PM|
|Registering virtual system VSys123||May 1, 2014, 4:08:02 PM|
|Pattern deployment starting.||May 1, 2014, 4:08:01 PM|
|Generating model for topology and network||May 1, 2014, 4:07:46 PM|
|Reserving cloud resources||May 1, 2014, 4:07:45 PM|
|Deployment has been queued||May 1, 2014, 4:07:43 PM|
Once a pattern instance is deployed, whether or not the application requires DNS is totally dependent on whether the application (including its middleware) uses host names and how the OS is configured to resolve host names. By default, when the system deploys a VM, it configures the OS with the IP address’ DNS server so that the OS can use it when desired. But if the application prefers not to use the DNS server, the OS can be configured not to use DNS. This would typically be configured during pattern deployment, such as by using a script package that runs on the VM to configure its operating system’s hosts file and disconnect it from DNS.
Given that the PureSystems Manager and each IP group can reach its corresponding DNS server, what data (known as DNS records) needs to be configured in those servers? (If they all use a single server, all of this data needs to be available in that server.)
- First, understand that when a DNS server lists an IP address/host name pair to be used by the VMs deployed on the system, it needs to define both forward and reverse lookups for that pair. (See Administering the DNS server in the IBM Knowledge Center.) This is true both for the DNS server used by the PureSystems Manager and for the DNS server used by an IP group.
- The DNS used by the PureSystems Manager must list all of the IP addresses in all of the IP groups, since any of those groups can be displayed in the system console. For each IP address, that DNS server must list the IP’s host name, since that is also displayed in the system console. Both forward and reverse lookups must be provided for all of the addresses. The system console uses these to confirm that it is displaying the IP group properly.
- The DNS used by an IP group must list all of the IP addresses in that particular IP group, since any of those IPs can be used to deploy a VM. For each IP address, that DNS server must list the IP’s host name, since that is also assigned to the VM. Both forward and reverse lookups must be provided for all of the addresses. The deployment engine uses both lookup directions while building the pattern instance.
- If the DNS server used by the PureSystems Manager is different from that used for any or all of the IP groups, each IP address will need to be listed twice, once in the PureSystems Manager’s DNS and once in the IP group’s DNS. Both DNS servers will need to list the host name for the IP address; they should both list the same host name for a given IP address.
Like most DNS clients, the system caches DNS data. It expires each record from the cache according to the record’s TTL setting in the DNS server. Once a record is changed in the server, the system will not reflect this change until the TTL expiration time passes.
Now that you understand the network settings the system uses to connect to DNS and how the system uses DNS, all of this explanation leads to the question: What goes wrong when DNS isn’t set up properly for the system? Generally, three types of problems can be encountered:
- Trouble displaying IP groups
One challenge for the system when DNS isn’t working properly is that it won’t be able to display IP groups properly. When you select an IP group in the system console, rather than displaying the IP group immediately, the console will either hang trying to display all details about the group, or it will display all of the details other than the IP address list (Table 2) but will hang displaying the section for the IP address list (Table 3). This hanging will last a couple of minutes as the console times out on one reverse lookup after another. When all of the addresses have timed out, the console will only display the IP addresses and will not display their host names. That’s because the IP addresses, like the other properties, are values stored in the IP group’s data – but the host names must be looked up from DNS.
An IP group whose display hangs and then displays the addresses but not the host names indicates one of two problems is occurring:
- The PureSystems Manager cannot reach the DNS server.
- The DNS server does not define forward and reverse lookups for all of the IP addresses in the IP group (the ones whose host names aren’t shown).
If the console cannot properly display any IP groups, then either the PureSystems Manager cannot reach the DNS server or the DNS server does not define any of the IP addresses’ mappings properly. If the console can properly display some IP groups but not others, the PureSystems Manager is able to reach DNS, so the server must define the mappings properly for some IP addresses but not others.
To diagnose the problem more precisely, use the Lookup widget shown in Figure 1. This widget is not doing anything special or proprietary; it is simply included in the console for convenience. It just runs the
nslookupcommand, acting like any UNIX® computer that is connected to the system management VLAN using the PureSystems Manager‘s IP address and configured with the DNS server’s IP address. Use the widget to see if it can look up the host name for one of the IP addresses that will not display correctly. Make sure to use it again to enter the host name and look up the IP address as well:
- If the widget can successfully look up the address for the host name and vice versa, DNS is set up correctly for that IP address.
- If the widget cannot perform the lookup, it cannot reach DNS.
- If the widget says an IP address or host name is not found for the data entered, the system is reaching DNS but the data is not defined in the DNS server properly.
Remember to test this for a representative sample of IP addresses, both forward and reverse, to make sure they’re all working. If an IP group cannot be displayed properly, the failed addresses in that group will fail in this widget.
- Problems deploying patterns
Another challenge for the system when DNS isn’t working properly is that pattern deployment will either fail or eventually succeed but take far longer than it should. When neither the PureSystems Manager nor the IP group can reach their DNS server or neither contains the host name for an IP address, deployment will fail. When the IP group cannot resolve the host name for the IP address but the PureSystems Manager can, the deployment will succeed but DNS timeouts will delay it. For example, on one system, when DNS for its IP group was finally set up properly, the deployment time for the simple pattern shown in Table 3 went from 34 minutes to 6 minutes.
During deployment, each DNS lookup that fails takes 10 minutes to time out. Thus a telltale sign of this happening is a step that normally takes a few seconds to run takes more than 600 seconds to run. For the simple pattern shown in Table 3, the maestro script package should take 11-12 seconds to run. When DNS wasn’t working properly, that script package took 10 minutes and 12 seconds to run. The script package took 10 minutes for DNS to time out, then 12 seconds to do its work without the results of the lookup.
You might expect that if an IP group’s DNS isn’t set up properly, the deployment would fail, or at least that the VMs wouldn’t have host names. Actually, as long as the PureSystems Manager can use DNS properly, the deployment still succeeds (barring other causes of failure) and each VM has the appropriate IP address/host name pair. When the DNS lookup on the IP group’s server times out, the PureSystems Manager then uses its DNS server to get the host name (assuming the PureSystems Manager can display the IP group successfully). So the problem is not that the pattern deployment fails, but that it takes much longer than it should.
- Virtual machine cannot connect to network resources
Once a VM is deployed, it can be tested to confirm that it can use its IP address to reach DNS properly. Assuming no additional configuration has been performed on the operating system to change its hosts settings or network settings for DNS, it’s a safe bet that if the VM cannot reach its DNS server and get successful lookups from it once it was deployed, it probably couldn’t during deployment either.
To test the deployed VM, SSH into its OS. From within the VM’s OS, try network tasks like these:
nslookupto find the PureSystems Manager by its IP address or host name.
pingto find the PureSystems Manager or the DNS server.
SSHto connect to the PureSystems Manager or the DNS server.
Even if the ping and SSH services are disabled on these other servers, the VM should at least be able to find them on the network and display errors that the services won’t work. If the VM gives errors that it can’t even find these servers on the network, or just hangs and times out, then that indicates DNS isn’t set up properly.
This article reviewed the basics of how DNS works as part of any IP network, the functionality in PureApplication System that uses DNS and how DNS is used, and how to troubleshoot problems involving the system’s use of DNS. With this information, you can configure DNS servers for PureApplication System and configure PureApplication System to use them, thereby helping to ensure that the system will deploy your workloads onto your data center’s network properly.
The author thanks fellow IBMers Alan Booth, Ivan Heninger, Mike Law, Brad Broge, Santiago Ortega, Adam Geiger, Erhan Ekici, and Hendrik van Run.
- IBM PureApplication System backup and restore, Part 1: Recovery scenarios
- IBM PureApplication System
- IBM Knowledge Center
- IBM PureApplication System resources on developerWorks
- IBM PureSystems Centre
- Adoption Scenarios for IBM PureApplication System
- Achieving business continuity in IBM PureApplication System
- Managing application runtime environments in IBM PureApplication System