Use this algorithm to map deployment of virtual machines

A simplified and efficient way to deploy virtual machines in bulk

Deploying virtual machines in bulk is a very tedious and cumbersome job. Deploying a virtual machine on a data center involves various tasks, such as installing the operating system, prepping the machine for a system (sysprep), adding a system to the domain, and giving proper access to users. One of the key challenges is planning what to deploy where. The authors propose an algorithm that provides a simplified and efficient solution to the problem.

Share:

Rahul Choudhary (rahul.choudhary@in.ibm.com), Software Engineer, IBM

Author1 photoRahul has been part of IBM India Software Labs for five years and is presently working as a developer for Rational Functional Tester. He has hands-on knowledge in the virtualization field and has worked on automating virtual machine deployment.



Riti Saxena (ritisaxe@in.ibm.com), Software Engineer, IBM

author photoRiti Saxena is working for TXSeries and WXTR. She has been an active member in tech vitality forums. She presented her ideas in IBM's internal Hack Day 2008 and got first prize for the implementation. She has also made a presentation for Hack Day in India Software Labs, Shakti. She was a finalist in Tech Connect 2010 and her Location-based Task Scheduler idea was well-received. She also submitted a poster for Regional Technical Leadership Exchange (RTLE) 2010.



04 August 2012

Also available in Russian

A proposed solution to virtualization challenges

Virtualization is similar to abstraction, where background processes are hidden. Users do not have any clue about the host machine, other users, or their environment. Each virtual machine can interact independently with other devices, applications, data, and users as though it were a separate physical resource.

When it comes to virtualization, there is not only one way to do it. In fact, there are several ways that achieve the same result through different levels of abstraction.

The example in this article is based on full virtualization, which is an almost complete simulation of the actual hardware. This is to allow software, which typically consists of a guest operating system, to run smoothly.

Figure 1. Full virtualization example
Diagram of a typical full virtualization environment

Challenges with virtualization

Deploying virtual machines in bulk is a very tedious and cumbersome job. It is a nightmare for administrators. Deploying a virtual machine on a data center involves various tasks, such as installing an operating system, prepping the machine for a system (sysprep), adding a system to the domain, and giving proper access to users.

One of the key challenges is planning what to deploy and where to deploy, especially when the deployment needs to be done in bulk. This is one of the major problems in client consolidation. In nutshell, the problem is what to deploy, and where

Solution

This article describes a simplified, efficient, and optimal solution to the problem in the form of a proposed algorithm: Virtual Machine Deployment Map. Considering the various servers and storage resources available, the algorithm tries to distribute each group of virtual machines along the available servers in the ratio of their available resources. The algorithm also ensures that the minimal number of base virtual machines is required for each group, thus reducing of the storage required for the deployment significantly.

The storage required is also distributed proportionally on the available storage devices. The result is a map that depicts how many virtual machines of each group to create on each server and how many luns (logical unit numbers), and what size, are to be created on each storage device.

The algorithm also manages the load balancing of virtual machine groups (golden images) across the servers and storage devices, while taking into account creating as few golden images as possible. Therefore, it provides an efficient resource optimization mapping technique of golden image groups across various servers and storage, which depends on a configurable distribution factor. It even enables the ease in use of high-availability feature of virtual machine deployment.

Our algorithm plans deployment architecture and creates a deployment mapping between virtual clients, servers, and storage (mapping of virtual clients to servers and virtual disks to storage). This helps the deployment engineer and the system administrator know which virtual machine would be created on which server. It also suggests the way that the storage would need to be partitioned to accommodate the storage requirements of all of the virtual machines.

You can use this map in XML format or as a GUI screen, depending on your business requirements.


Flow chart of the algorithm

The algorithm is divided into two parts:

  • Server-VM map
  • Storage-VM map

Server-VM map

This generates the server to VM mapping. Figure 2 shows the overview of server-VM mapping.

Figure 2. Flowchart for Server-VM mapping
Diagram of Server-VM mapping algorithm

Storage-VM map

It takes the output of Server-VM map and maps the VMs to luns on storage boxes. These luns need to be created on respective storage boxes and mapped to the servers as depicted in the output of the Storage-VM map.

Figure 3. Flowchart for Storage-VM mapping
Diagram of Storage-VM mapping algorithm

Algorithm details

The algorithm assumes that the administrator has the knowledge of the following inputs that need to be passed to the algorithm.

Inputs

Server list

Each server has information such as CPU cores, sockets, memory and so forth for that server.

User type

The type of hardware requirements each user group has. Basically, there are three user types in an organization:

  • Power users: The ones who require maximum hardware resources (CPU, memory). For example, developers.
  • Knowledge users: The ones who require average resources for their daily work. Testers, for example.
  • Task users: The ones who don’t have a very high requirement for resource requirements for their day-to-day work. Managers, for example.

Golden image list

The list contains information about the number of users each golden image has and their types. Every organization has various projects and a set of software and hardware required specifically for each of them. Each project has different storage requirements, as well as for user types (Power, Knowledge, or Task). For example, an organization might have two golden images (projects) with the following configuration:

  • Golden Image 1 needs 5 Power, 3 Knowledge, and 1 Task user.
  • Golden Image 2 needs 4 Power, 0 Knowledge, and 1 Task user.

Storage list

Each storage entry has information such as the total size available on that storage area.

The algorithm uses the ratio and proportion to map users to servers.

Server-VM map

  1. Initialization
    1. Initialize available CPU cores and available memory on each server.
    2. Set TotalAvailableCore and TotalAvailableMemory,combining the cores and memory from the server list.
    3. Initialize the number of users mapped onto each server to be zero.
  2. Iterate through the Golden Image List, and calculate the total Power (totalPUsers), Knowledge (totalKUsers), and Task (totalTUsers) users.
  3. Sort the list of servers in descending order, based on cores available for resource allocation. This is to ensure load balancing
  4. Create a status variable (see Listing 1), and set it to have the entire user Types as unallocated. This variable needs to be checked during the course of the algorithm to ensure that all users have been mapped to the servers.
Listing 1. Code snippet to allocate the status variable
status.setPUnallocated(totalPUsers);
status.setKUnallocated(totalKUsers);
status.setTUnallocated(totalTUsers);
  1. Perform initial validation (see Listing 2). This step ensures that total resources on servers are more than the resources required to map all of the users to the system.
Listing 2. Code snippet for initial validation
performInitialValidation()
{
	
tCoresRequired = totalTUsers * coreTUsage
kCoresRequired = totalKUsers * coreKUsage
pCoresRequired = totalPUsers * corePUsage
totalCoresRequired = tCoresRequired + kCoresRequired + pCoresRequired;

tMemoryRequired = totalTUsers * memoryTUsage
kMemoryRequired = totalKUsers * memoryKUsage
pMemoryRequired = totalPUsers * memoryPUsage
totalMemoryRequired = tMemoryRequired + kMemoryRequired + pMemoryRequired;
	
 If(totalAvailableMemory >= totalMemoryRequired && totalAvailableCores >= 
                                    totalCoresRequired)
	        Go To Step 6(defined below);
	 else
                     Exit algorithm stating insufficient resources;
}

Key:

  • Where coreTUsage, coreKUsage, corePUsage are the cores required by the Task, Knowledge, and Power users, respectively
  • Where memoryTUsage, memoryKUsage, and memoryPUsage are the memory required by Task, Knowledge, and Power users, respectively.
  1. Calculate the ratio. The following module tries to incorporate load balancing, golden image (GI) balancing, and base virtual machine optimization by taking chunks of the divisor unit during mapping. The module returns true if all users can be distributed across all servers in terms of units of divisor, where divisor is a configurable variable that the administrator wants to set, depending on needs. It ensures that you map only that many users to a server at a time. Therefore, you can optimize the use of golden image cloning and ensure that you don’t create too many golden images.
    1. Add each GI under each server and initialize respective Power, Knowledge, and task counts to zero, as well as storage required by server to zero
    2. Browse through each GI to be distributed across the servers in terms of ratio.
      Listing 3. Code snippet to assign remaining users
      for(k = 0;k<goldenImageList.size();k++)
      {
          pUsersRemaining = goldenImageList.get(k).getPUserCount(); 
          kUsersRemaining = goldenImageList.get(k).getKUserCount(); 
          tUsersRemaining = goldenImageList.get(k).getTUserCount();
      1. Calculate units of Power, Knowledge, and Task users to be allocated at the same time.
        Listing 4. Code snippet to initialize divisor unit for each user type
          pUnit = pUsersRemaining / divisor ; 
            kUnit = kUsersRemaining / divisor ;
            tUnit = tUsersRemaining / divisor 
            totalUsersUnit = 128 / divisor; //maximum users on a server =128
      2. Calculate the pratio, tratio, and kratio:
        Iterate through the server list.
        {
        • pratio on ith server = (cores on ith server/total cores) * (pUnit)
        • If ratio is more than number of Power users left
          pratio on server i and Golden Image k = pUnit-(GiPCount/divisor);
        • If one divisor unit cannot be allocated:
          pratio on server i and Golden image k = 0;
        • If total users on this server is greater than 128
          pratio = totalUsersUnit - number of users on server i;
        • // Check if pratio can be accommodated on the server
          // Incrementing with 0.1 as pratio is calculated up to one decimal point
          for (j =1 ;j <= pratio on server i and Golden Image k ;j = j + 0.1)
          {
              Calculate the actual number of pusers that can be actually
              accommodated on the ith server
          }
        • If none can be allocated then
          pratio = 0
          else
          pratio = number of users that cannot be allocated.
        • Update the server info: Decrement its CPU and memory resources and increase the storage required depending on pratio.
        • Update the count of users of the Golden Image that has been allocated on the server.
        • Perform the same for Knowledge and Task users.
        }//End of step ii
      3. Map unallocated users:
        • Assign the users which were not allocated during the ratio (for p, k, and t users).
           pUnallocated+=pUnit - (GiPCount /divisor);
        • Traverse through the bubble-sorted server list, and try to map chunks of Power, Knowledge, and Task users left for that golden image, based on the divisor.
      4. Return red flag if any one user cannot be allocated in any server.
         } // Step b ends
    3. Check the status variable. If any of the users remain unallocated, display a red flag saying "insufficient resource." Otherwise, proceed to Storage-VM map.

Storage-VM map

Inputs

Server VM map generated from above algorithm will be taken as input to this storage list.

  1. Initialize TotalStorage by adding the sizes of all available storage from the storageList. Create a status variable, and set it to have all of the user types as unallocated. This variable needs to be checked during the course of the algorithm to ensure that all users have been mapped to the servers. (See Listing 1.)
  2. Perform initial validation. Check whether storage required by all virtual machines is greater than total storage available
  3. Calculate Storage Ratio. This computes the number of Power, Knowledge, and Task users to be allocated on each storage device, with the key factor being the available storage.
    1. Iterate through the StorageList.
      For i = 0 to StorageList size
      • Calculate the ratio of Power, Knowledge, and Task users on that storage device.
Listing 5. Code snippet to initialize ratios on storage devices
    pratio of storage i = capacity of i/total capacity * totalPUsers 
    kratio of storage i = capacity of i/total capacity * totalKUsers 
    tratio of storage i = capacity of i/total capacity * totalTUsers
  • Round off those calculated ratios, and check whether the users can actually be allocated on the storage.
Listing 6. Code snippet to check ratio allocation
For (j = 1; j <= Storage (i).getPCount (); j++) 
                { 
                if((j * PUserStorageRequired) <= Storage i.getAvailableStorage()) 
                
                { 
                // continue; 
                } 
                else 
                { 
                Storage(i).PCount = (j - 1); 
                break; 
                } 
                }// End of for j
  • Update: available storage on server i and status flag for mapped p users.
  • Repeat these steps for Knowledge and Task users.

//End of step a

  1. Map unallocated users. Check the status flag for unallocated users. If there are any, sort the storage list, in descending order, in terms of available storage, and map the users onto them.
  2. Map VM2Lun. If any user still remains unmapped, raise a red flag saying "insufficient resources." Otherwise, create luns, and map the VM onto them.
    1. Browse through the serverVMMap and iterate through each golden image that needs to be mapped onto that server.
    2. Initialize pRemaining = the number of p users of that golden image that need to be mapped onto server (i).
    3. Initialize lun size and number of users on the lun to be zero.
    4. For p = 1 to p<= pRemaining:
      • Increase the lun size for pUser that can be accommodated while taking into account the following considerations.
      • Lun size is less than 500 GB.
      • Number of users on the lun does not exceed 30.
      • Number of p users does not exceed the store user mapping generated from step 3.
      • Update the resource after each decrement of p.
      • Update the resource after each decrement of p.
      • If lun size = 500 or the number of users of lun = 30 , map lun to the server and create a new lun.

Note:
Repeat the same for Knowledge and Task users.

  1. Display the server VM as well as storage lun map in XML or GUI format.

Sample output

Figure 4 depicts a sample output of the Virtual Map Deployment algorithm in XML format.

Input supplied: Server 1 and Server 2

Golden image:

  • GI1 has 20 Power, 30 Knowledge, and 12 Task users
  • GI2 has 7 Power, 1 Knowledge and 2 Task users

Storage: Storage1

Figure 4. Sample output of the Virtual Map Deployment algorithm in XML
Screen capture of f algorithm output in XML

Output generated:

  • Server1 needs 2 luns. Lun1 hosting 10 P, 15 K, and 7 T users of GI1, and Lun2 hosting 7P, 1K, and 2 T users of GI2.
  • Server2 needs 1 lun. Lun3, which is hosting 10 P, 15 K and 5 T users GI1.
  • There are 3 luns to be created on Storage1, Lun1, Lun2, and Lun3 of sizes 120, 75, and 100 GB, respectively.

Share your reaction

We invite and welcome your feedback and suggestions in the Comments area for this article.

Resources

Learn

Get products and technologies

  • Download a free trial version of Rational software.
  • Evaluate other IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational
ArticleID=832841
ArticleTitle=Use this algorithm to map deployment of virtual machines
publish-date=08042012