Introduction

The first part of this series explained how you can set up and configure IBM® Network Authentication Service (NAS) master and slave KDCs using the legacy database. Having a master-slave setup in an IBM NAS environment helps ensure the reliability and availability of the Kerberos realm. The next step is to understand how you can promote a slave KDC to be a master KDC and why you would need to.

Many times in production, the master KDC system is subject to be retired and replaced by a newer system for scalability reasons. In other scenarios, maintenance is planned for the server, or the server hosting the master KDC is undergoing intermittent hardware problems. For these and other practical reasons, a facility for promoting an existing slave KDC to take over the master's responsibility is essential. The promotion requires the new master KDC to manage tasks that can be handled only by the master, apart from its regular KDC requests. Thus, migration of a master KDC to a slave KDC is a critical activity that directly impacts the reliability and availability of all the kerberized applications running over the Kerberos realm.

Below are the detailed steps for an administrator to promote a slave KDC to a master KDC using the legacy database. The article assumes a running Kerberos realm having a master-slave setup with a legacy database as explained in Part 1 of this series (see Resources). The article explains the required steps with a sample setup, which should assist administrators who are planning for the migration.

Distinction between master KDC and slave KDC

In a typical production environment, Kerberos setups always have the master-slave KDC configuration. In this section, let's look at the main differences between the master KDC and slave KDC.

From Part 1 of this series, you know that there can be only one master KDC in the Kerberos realm, but multiple slave KDCs could be present. These slave KDCs perform load balancing and ease the burden off the master KDC.

Each of the slave KDCs has a read-only copy of the Kerberos principal database, whereas the master KDC has access to the main writable copy of the database where the changes can be made. The master KDC is responsible for propagating the updated copy of the database to each slave KDC after a fixed interval.

Also, the Kerberos administration server (kadmind) runs on the master KDC only. So migration needs to take care of this as well.

Issues while upgrading

Here are some common questions you may have, and suggestions for handling them.

• When there is more than one slave KDC, how do you decide which one to choose as the next master KDC?
  • Choose a slave machine that has a recent and easily upgradable hardware configuration.
  • Choose a slave machine with less overall workload.
  • Choose a slave machine with relatively fewer IBM NAS clients. (Tip: Compare the various krb5.conf files across the clients to find a slave machine with fewer clients configured for it.)
    
    
• How do you minimize the effective downtime of the master KDC (that is, the time between shutting down the KDC on the master KDC and starting it on the slave KDC)?
  • You can minimize the impact by taking time to understand the migration process in its entirety.
  • Practice the steps below in a non-production machine (in a test cell) to avoid any surprises in the production cell.
  • If one master KDC is being migrated to an existing slave KDC (in a single master-slave setup), we recommend that you have an additional slave KDC configured to handle the requests during the downtime of the old master KDC. This option may require you to update the Kerberos client configuration file (/etc/krb5/krb5.conf) across the realms.
    
    
• How do you minimize users’ pain?
  • Announce the migration activity well in advance and also disclose the services that will be affected (NFS V4, Kerberized telnet, Kerberized SSH, etc).
  • Estimate and announce the expected downtime for users, and finish the activity within the planned time frame. Add buffer time, just in case something goes wrong.
  • After a successful migration, check and re-check the new configuration and proper working of new master KDC, including all of the kerberized services like NFS V4, Kerberized telnet, and etc. before announcing the resumption of services.
  • After migration, make changes as needed to the configuration files of all Kerberos clients. To ease this task, use the "KDC discovery via LDAP" feature of IBM NAS (see Resources for more information on this).

How to upgrade

The process for migrating from the slave KDC to the master KDC is straightforward, but it requires a good amount of Kerberos administrative knowledge so you can understand why each step is necessary. In the example migration, we will use the same example Kerberos setup we used in Part 1 of this series. Our master KDC (along with kadmind server) is initially on the host machine fsaix12.in.ibm.com (we are going to refer this as "old master KDC") and our slave KDC is on the machine huntcup.in.ibm.com (our "new master KDC," hereafter). At the end of the example, we will have the master KDC and administration server running on huntcup.in.ibm.com.

Here are the steps to upgrade the slave KDC to the master KDC, along with examples and command outputs.

1. On the old master KDC, stop both the Kerberos daemons by using the /usr/krb5/sbin/stop.krb5 command.

   -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# /usr/krb5/sbin/stop.krb5 Stopping /usr/krb5/sbin/krb5kdc... /usr/krb5/sbin/krb5kdc was stopped successfully. Stopping /usr/krb5/sbin/kadmind... /usr/krb5/sbin/kadmind was stopped successfully. The command completed successfully. -bash-2.05b#

   
   
2. On the old master KDC, disable the cron job that propagates the Kerberos database to all slave KDCs. Use the crontab AIX command to do this. Remember that we have already added our database propagation cron script for the old master KDC in Part 1 of this series. Let’s remove that entry now. First, check the entry using the crontab -l command.

   -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# crontab -l 0 11 * * * /usr/bin/errclear -d S,O 30 0 12 * * * /usr/bin/errclear -d H 90 0 15 * * * /usr/lib/ras/dumpcheck >/dev/null 2>&1 # SSA warning : Deleting the next two lines may cause errors in redundant # SSA warning : hardware to go undetected. 01 5 * * * /usr/lpp/diagnostics/bin/run_ssa_ela 1>/dev/null 2>/dev/null 0 * * * * /usr/lpp/diagnostics/bin/run_ssa_healthcheck 1>/dev/null 2>/dev/null # SSA warning : Deleting the next line may allow enclosure hardware errors to go undetected 30 * * * * /usr/lpp/diagnostics/bin/run_ssa_encl_healthcheck 1>/dev/null 2>/dev/null # SSA warning : Deleting the next line may allow link speed exceptions to go undetected 30 4 * * * /usr/lpp/diagnostics/bin/run_ssa_link_speed 1>/dev/null 2>/dev/null 0 00,12 * * * wall%rc.powerfail:1::WARNING!!! The system is now operating with cooling problem. This message will be walled every 12 hours. Remove this crontab entry after the problem is resolved. 59 23 * * * /var/krb5/krb5kdc/prop_krb5_database.sh -bash-2.05b#

   
   To remove this entry, use crontab -e command.

   -bash-2.05b# crontab -e

   
   The above command will give you a text editor ('vi' by default) with a file opened. Locate the above highlighted line in that file and remove it using the 'vi' command 'dd' in command mode. Save the file using :wq! and at the command line, check that the cron job has been removed.

   -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# crontab -l 0 11 * * * /usr/bin/errclear -d S,O 30 0 12 * * * /usr/bin/errclear -d H 90 0 15 * * * /usr/lib/ras/dumpcheck >/dev/null 2>&1 # SSA warning : Deleting the next two lines may cause errors in redundant # SSA warning : hardware to go undetected. 01 5 * * * /usr/lpp/diagnostics/bin/run_ssa_ela 1>/dev/null 2>/dev/null 0 * * * * /usr/lpp/diagnostics/bin/run_ssa_healthcheck 1>/dev/null 2>/dev/null # SSA warning : Deleting the next line may allow enclosure hardware errors to go undetected 30 * * * * /usr/lpp/diagnostics/bin/run_ssa_encl_healthcheck 1>/dev/null 2>/dev/null # SSA warning : Deleting the next line may allow link speed exceptions to go undetected 30 4 * * * /usr/lpp/diagnostics/bin/run_ssa_link_speed 1>/dev/null 2>/dev/null 0 00,12 * * * wall%rc.powerfail:1::WARNING!!! The system is now operating with cooling problem. This message will be walled every 12 hours. Remove this crontab entry after the problem is resolved. -bash-2.05b#

   
   
3. From the old master KDC, run the database propagation script manually for the last time, to make sure that you have the latest copy on your slave KDCs. Please refer to Part 1 of this series for details on the database propagation script and how to run it manually. For now, you can see how this can be done:

   -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# cat prop_krb5_database.sh #!/bin/sh KDCLIST="huntcup.in.ibm.com" DUMP_FILE="/var/krb5/krb5kdc/krb5_dump" /usr/krb5/sbin/kdb5_util dump $DUMP_FILE for kdc in $KDCLIST do /usr/krb5/sbin/kprop -f $DUMP_FILE $kdc done -bash-2.05b# ./prop_krb5_database.sh Database propagation to huntcup.in.ibm.com: SUCCEEDED -bash-2.05b#

   
   
4. On the old master KDC, edit the /etc/krb5/krb5.conf file to reflect the new master KDC entries. So, change the first 'kdc = …' line under [realms] stanza and enter the new master KDC name (huntcup.in.ibm.com in this example). If you want to keep the old master KDC as a slave KDC, then keep its entry after the new master KDC entry; otherwise, remove the entry of the old master KDC. Also, the administration server (kadmind) will be running on the new master KDC after the successful upgrade, so change the 'admin_server = …' line under the [realms] stanza to reflect the new master KDC name.
   After the changes, the /etc/krb5/krb5.conf file should look like this:

   -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# cat /etc/krb5/krb5.conf [libdefaults] default_realm = ISL.IN.IBM.COM default_keytab_name = FILE:/etc/krb5/krb5.keytab default_tkt_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc default_tgs_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc [realms] ISL.IN.IBM.COM = { kdc = huntcup.in.ibm.com:88 admin_server = huntcup.in.ibm.com:749 default_domain = in.ibm.com } [domain_realm] .in.ibm.com = ISL.IN.IBM.COM fsaix12.in.ibm.com = ISL.IN.IBM.COM [logging] kdc = FILE:/var/krb5/log/krb5kdc.log admin_server = FILE:/var/krb5/log/kadmin.log default = FILE:/var/krb5/log/krb5lib.log -bash-2.05b#

   
   
5. From the old master KDC, copy the following files to the new master KDC:
   /var/krb5/krb5kdc/kadm5.keytab
/var/krb5/krb5kdc/kadm5.acl
/etc/krb5/krb5_cfg_type

NOTE 1: Please make sure to use the ‘binary mode’ to transfer the files, if you are using any FTP client.
   
   NOTE 2: If you had configured your old master KDC to use the "Enhanced password strength" feature, then the password rules file also needs to be copied to the new master KDC (see Resources for more information on enhanced password strength).
   
   

   -bash-2.05b# hostname fsaix12.in.ibm.com -bash-2.05b# ftp huntcup.in.ibm.com Connected to huntcup.in.ibm.com. 220 huntcup.in.ibm.com FTP server (Version 4.1 Fri Nov 17 10:41:25 CST 2006) ready. Name (huntcup.in.ibm.com:root): root 331 Password required for root. Password: 230 User root logged in. ftp> bi 200 Type set to I. ftp> ha Hash mark printing on (1024 bytes/hash mark). ftp> put /var/krb5/krb5kdc/kadm5.keytab 200 PORT command successful. 150 Opening data connection for /var/krb5/krb5kdc/kadm5.keytab. # # 226 Transfer complete. 574 bytes sent in 0.001897 seconds (295.5 Kbytes/s) local: /var/krb5/krb5kdc/kadm5.keytab remote: /var/krb5/krb5kdc/kadm5.keytab ftp> put /var/krb5/krb5kdc/kadm5.acl 200 PORT command successful. 150 Opening data connection for /var/krb5/krb5kdc/kadm5.acl. # # 226 Transfer complete. 186 bytes sent in 0.000631 seconds (287.9 Kbytes/s) local: /var/krb5/krb5kdc/kadm5.acl remote: /var/krb5/krb5kdc/kadm5.acl ftp> put /etc/krb5/krb5_cfg_type 200 PORT command successful. 150 Opening data connection for /etc/krb5/krb5_cfg_type. # # 226 Transfer complete. 7 bytes sent in 0.000594 seconds (11.51 Kbytes/s) local: /etc/krb5/krb5_cfg_type remote: /etc/krb5/krb5_cfg_type ftp> bye 221 Goodbye. -bash-2.05b#

   
   
6. On the new master KDC, restart the Kerberos daemons.

   -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# /usr/krb5/sbin/stop.krb5 Stopping /usr/krb5/sbin/krb5kdc... /usr/krb5/sbin/krb5kdc was stopped successfully. Stopping /usr/krb5/sbin/kadmind... The command completed successfully. -bash-2.05b# /usr/krb5/sbin/start.krb5 Starting krb5kdc... krb5kdc was started successfully. Starting kadmind... kadmind was started successfully. The command completed successfully. -bash-2.05b#

   
   Once both the daemons are up and running, please take a few trials to get the initial ticket, and check that all the kerberized services and applications are working as expected.
7. Please remember that the new master KDC also contains the /etc/krb5/krb5.conf (as the client is by default configured if you configure IBM NAS master or slave KDC). So don’t forget to change this file to reflect the new entries of KDC and kadmind server, like this:

   -bash-2.05b# hostname huntcup.in.ibm.com -bash-2.05b# cat /etc/krb5/krb5.conf [libdefaults] default_realm = ISL.IN.IBM.COM default_keytab_name = FILE:/etc/krb5/krb5.keytab default_tkt_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc default_tgs_enctypes = des3-cbc-sha1 arcfour-hmac aes256-cts des-cbc-md5 des-cbc-crc [realms] ISL.IN.IBM.COM = { kdc = huntcup.in.ibm.com:88 admin_server = huntcup.in.ibm.com:749 default_domain = in.ibm.com } [domain_realm] .in.ibm.com = ISL.IN.IBM.COM fsaix12.in.ibm.com = ISL.IN.IBM.COM huntcup.in.ibm.com = ISL.IN.IBM.COM [logging] kdc = FILE:/var/krb5/log/krb5kdc.log admin_server = FILE:/var/krb5/log/kadmin.log default = FILE:/var/krb5/log/krb5lib.log -bash-2.05b#

   
   
8. Similarly, don’t forget to change the /etc/krb5/krb5.conf file as shown above on all the clients under this realm.
9. Finally, as a thoughtful and responsible Kerberos administrator, set up the cron job on the new master KDC to ensure timely propagation of the Kerberos database. See Part 1 of this series for details.

Conclusion

By following the steps in this article, you have successfully upgraded your slave KDC to act as a master KDC. Part 3 in this series will cover how to configure the slave KDC setup using LDAP as a back end to store the Kerberos database.

Resources

Learn

• IBM NAS KDC Configuration: Part 1: Configuration and Management of Slave KDC in IBM NAS on AIX (developerWorks, September 2008): The first part under this series 'IBM NAS KDC configuration' covers the basic Master-Slave KDC configuration and management in detail.
  
  
• Enhanced password strength in IBM Network Authentication Service for AIX (developerWorks, July 2008): Discover what is the 'Enhanced password strength' feature in IBM Network Authentication Service for AIX and how to make use of it in your Kerberos environment.
  
  
• IBM Network Authentication Service KDC and administration servers discovery using LDAP for AIX (developerWorks, Dec 2007): Read this article to learn about IBM NAS KDC discovery via LDAP.
  
  
• Configuring AIX 5L for Kerberos Based Authentication Using IBM Network Authentication Service: Read this white paper to learn about using Kerberos as an alternative authentication mechanism to AIX.
  
  
• A Kerberos Primer (developerWorks, Nov 2001): This article introduces Kerberos technology and Distributed Computing Environment-based applications.
  
  
• AIX and UNIX : Want more? The developerWorks AIX and UNIX zone hosts hundreds of informative articles and introductory, intermediate, and advanced tutorials.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  

Get products and technologies

• IBM Network Authentication Service for AIX: Download the IBM Network Authentication Service for AIX from IBM AIX Web Download Pack Programs.
  
  
• IBM Network Authentication Service for Linux,Solaris: Download the IBM Network Authentication Service for Linux,Solaris from here.
  
  
• IBM GUI-based Administration Tool for Network Authentication Service: Experience the GUI to perform the IBM NAS related administration tasks. Download from the IBM alphaWorks today.
  
  
• AIX 5L Expansion Pack and Web Download Pack: Start downloading now.
  
  

Discuss

• Participate in the AIX and UNIX forums:
  • AIX Forum
  • AIX Forum for developers
  • Cluster Systems Management
  • IBM Support Assistant Forum
  • Performance Tools Forum
  • Virtualization Forum
  • More AIX and UNIX Forums
  • AIX Networking
  
  

About the authors

Sandeep Ramesh Patil is an Advisory Software Engineer for the IBM India System and Technology Lab. His professional experience has been on distributed technology and security products such as the IBM Network Authentication Services (IBM Kerberos). He is an IBM developerWorks Professional Author with most of his articles on information security. He also plays a active role in IP generation. Sandeep holds a BE degree in computer science and engineering from the University of Pune, India. You can contact him at rsandeep@in.ibm.com .

Vipin Rathor has been with IBM India System and Technology Lab for three years. He works for IBM Network Authentication Service Development and Support activities. His areas of interest include network security, particularly Kerberos and other authentication protocols.

Comments (Undergoing maintenance)

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Issues while upgrading
• How to upgrade
• Conclusion
• Resources
• About the authors
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers