Introduction

Maintaining system administration files such as /etc/passwd is quite straightforward for an environment that consists of a small number of systems. However, it can be quite time consuming for an environment that boasts a large number of systems and end users. This article discusses two commonly practiced methods to distribute and share such administration files among multiple systems in a network environment and shows in detail how these two methods are actually implemented.

The methods discussed here are generally available on most Unix systems, including all versions of AIX, and would be of particular interest to system administrators who not only spend a great deal of time maintaining these files but find this aspect of their job quite annoying. This article would also be useful to those application developers who, at times, have to set up their own development environment.

Note that the Network Information Services (NIS), the second of the two methods discussed in this article, offers a way to enforce some aspects of security in terms of system access.

Back to top

Overview

A properly functioning Unix system depends on a number of configuration files. These files normally reside on the local system. When you multiply these files by the number of hosts in the network, the result can be hundreds of files -- too many to easily manage.

Fortunately, however, systems are often quite similar from an administrative point of view and this situation makes it easier to manage the systems. Instead of having to maintain the configuration files on each machine, one could take advantage of the similar administrative behavior of the systems and cluster the machines into groups that would share configuration information.

There are two generally accepted ways to do this grouping. The simplest way is to keep a master copy of each configuration file in one place and distribute it to members of the group whenever it changes. We will call this method Distribution by file copying. It has the advantages of being simple and of working on every Unix system.

Another approach is to eliminate text files altogether and have each machine obtain its configuration information from a central server. This is the approach used by the Network Information Services (NIS) or the follow-on product, NIS+. This method is more complicated than copying files but it is much more reliable. For example, clients do not miss updates even if they are down when a change is made to the configuration information. However, the downside of this approach is that, if not properly implemented, the entire network can become crippled if the server goes down.

In this article we will look at both approaches including how they are implemented.

Back to top

Configuration files -- what to share

Of the many configuration files on a Unix system, most server-based systems are generally set up to work with the following subset:

Table 1. Configuration file functions

The files listed above are usually accessed through routines defined in the Standard C library. For example, the /etc/passwd file is searched with the getpwuid, getpwnam, and getpwent routines. These routines take care of opening, reading, and parsing the passwd file so that user-level programs do not have to.

Back to top

Distribution by file copying

Maintaining a network by copying the master configuration files may not be an elegant solution, but it works on every kind of machine and is easy to set up and maintain. If the requirements are not complex or the network is small and simple, this method may be good enough. One does not always have to use a complex system such as NIS.

File copying systems use either a "push" model or a "pull" model. With push, the master server periodically distributes the most up-to-date files to each client, whether the client wants them or not. The server explicitly pushes files whenever a change occurs or whenever the regular distribution schedule specifies.

The push model has the advantage of keeping the distribution system centralized on one machine. All files are stored in one place, making the scheme easy to control. One disadvantage is that each client must allow the master to modify its system files, creating a possible security risk.

In a pull system, each client is responsible for updating itself from the server. This method is less centralized but it is more adaptable and more secure.

The rdist command to push files

The master server pushes files by executing the rdist command by itself. By default, rdist looks for a control file named distfile (or Distfile) in the current directory. If the control file is not in the current directory, you can explicitly specify the pathname to the control file with the -f option:

rdist -f distfile

An example of a distfile is given later in the discussion. In the distfile, the separators (tabs, spaces, and newlines) are interchangeable. Comments are introduced with the pound sign (#).

The statements in a distfile use the following syntax:

label: pathname -> destinations commands

The label field associates a name to the statement. From the shell, you can issue the following command to distribute only the files appearing in the statement associated with that label:

rdist label

The pathname and destinations files contain a list of files to be copied and a list of hosts to copy them to, respectively. If there is more than one entry in a list, it must be surrounded with parentheses and the elements must be separated with white space.

By default, rdist copies the files and directories listed in pathname to the equivalent paths on each destination machine. You can modify this behaviour by using commands from the following list when creating the distfile (note that each command ends with a semicolon):

install options [destdir];
notify namelist;
except pathlist;
except_pat patternlist;
special [pathlist] string;

The install command sets options that affect the way rdist copies files. Options typically control such things as symbolic links.

The destdir argument specifies an installation directory on the destination hosts.

The notify command takes a list of email addresses as its argument. rdist sends mail to these addresses whenever a file is updated. Any addresses that do not contain an at sign (@) are suffixed with the name of the destination host.

The except and except_pat commands remove pathnames from the list of files to be copied. The arguments of except are matched literally, while those of except_pat are interpreted as regular expressions.

The special command executes an sh command (the string argument should be quoted) on each remote host. If a pathlist argument is present, rdist executes the command, specified by the string argument, once after copying each of the files specified in the pathlist. Without a pathlist, rdist executes the command.

The following is an example of a distfile:

	SYS_FILES = (/etc/passwd /etc/group /etc/aliases)
	GET_ALL = (homer bart marge)
	GET_SOME = (krusty moe)

	all: ${SYS_FILES} -> $(GET_ALL)
		notify jim;
		special /etc/aliases "/usr/ucb/newaliases";

	some: ${SYS_FILES} -> ${GET_SOME} 
		except /etc/aliases;
		notify ted@moe;

This distfile replicates the three listed system files on machines homer, bart, and marge, and sends mail to jim@destination describing any updates or errors that occur. After /etc/aliases is copied, rdist runs newaliases on each destination. Only two files are copied to machines krusty and moe, with a report being mailed to ted@moe.

The rdist command copies files only when they are out of date. Therefore, you can create the distfile as if all files were to be copied and let rdist decide what really needs to be copied.

The rdist command preserves the file attributes (owner, group, mode, and modification time). When rdist updates an existing file, it deletes the old version before installing the new one.

The expect command to pull files

There are several ways to implement a pull system. One way is to make system files available via ftp from a central server and to use expect to retrieve and install them.

The expect command is a set of extensions to Tcl (Tool Command Language) that is used to write control scripts for interactive programs. However, expect is different from a normal scripting language in that it provides for incremental control of subprocesses. The output produced by each operation can be examined to determine what input should be sent next.

Back to top

The Network Information Service

NIS, released by Sun in the 1980s, was the first major administrative database for the system configuration files. Originally called Yellow Pages, NIS received a new name for legal reasons. NIS commands still begin with the letters yp.

A follow-on product from Sun is called NIS+, but despite the similarity of the names, NIS+ and NIS are not related to one another.

With NIS, a master server maintains the authoritative copies of system files. A server process makes the contents of the files (or as NIS calls them, "maps") available over the network. NIS maps are preprocessed by the ndbm extensible hashing routines to improve the efficiency of lookups. You can edit the system files on the master server with a text editor. Then when makedbm is executed on them, NIS converts them to ndbm format.

A server and its clients comprise an NIS domain.

The ndbm routines allow only one key to be associated with each entry, so a system file may have to be translated into several NIS maps. For example, the /etc/passwd file is translated into two maps called passwd.byname and passwd.byuid. Use the first one to look up entries by user name and the second one to look up entries by UID. Either map can be used to enumerate all the entries in the passwd file.

NIS allows replication of the network maps on a set of slave servers. There can be multiple slave servers. Providing more than one server helps to relieve load on the master server and to keep clients functioning even when some servers become unavailable. Whenever a file is changed on the master server, the corresponding NIS maps must be pushed out to the slaves so that the data is synchronized on all servers. You can do the push either manually or through cron as described later. Clients do not distinguish between the master server and the slaves.

Every physical network must contain at least one NIS server. Clients use IP broadcasting to locate servers, but broadcast packets are usually not forwarded by routers and gateways. Use the ypset command to point a client at a particular server as described later. However, at the first sign of trouble, the client will attempt to locate a new server. Unless there is another server in the client's network, this failed attempt may cause the client to hang.

NIS data and local data -- to merge or not to merge

On most systems, NIS defines a complicated scheme that allows clients to supplement the configuration information obtained from NIS with their own local information. Some vendors have removed this function from NIS and provide a separate configuration file that specifies the order in which the sources of administrative information should be searched. However, a discussion of this file is beyond the scope of this article.

Under the traditional NIS, merging of NIS information with the local information is supported for some files but not for others. These two categories of files are defined as follows:

• Local priority files - The information in the local configuration files overrides that in the NIS configuration files.
• Global priority files - The information in the NIS configuration files supersedes that in the local configuration files.

The /etc/passwd and /etc/group files are usually the only local-priority files. NIS data must be "invited" into these files by including a special token in the file itself. A plus sign (+) on a line by itself at the bottom of these files would append the entire NIS map, while an entry of the form +name would append only the entry for the named account or group. A third form is used with netgroup as described below.

For example, the following /etc/passwd file would add local accounts davidw and stevenr to the accounts defined in the NIS passwd maps:

	davidw:!:123:201:David Williams:/home/davidw:/bin/ksh
	stevenr:!:124:201:Steven Roberts:/home/stevenr:/bin/ksh
	+

Local copies of global-priority files are completely ignored, with the exception of /etc/hosts, which may be consulted at boot time. The /etc/hosts, /etc/networks, /etc/protocols, /etc/services, and /etc/netgroup files are all global-priority.

What is netgroup?

NIS introduced a popular concept known as netgroup. Netgroups are used to name sets of hosts, users, and domains for easy reference in other system files. They are defined in /etc/netgroup but are also shared as a global-priority NIS map.

The format of a netgroup entry follows:

groupname list-of-members

Members are separated by white space. A member is either a netgroup name or a triplet of the form:

(hostname, username, domain name)

Any empty field in a triplet is a wild card; thus the entry (cyclop,,) refers to all users in all domains on the host cyclop. Since a minus sign (-) in a field indicates negation, the entry (cyclop,-,) refers to the machine cyclop and no users. Group definitions can nest.

Following is a simple example of an /etc/netgroup file:

	jimmy	(cobra,,)	(support,,)
	systems	(chain,,)	(lisa,,)	(apu,,) (cyclop,,)
	clients	(charli,,)	(pushkin,,)	(bush,,)
	daniel	(chain,,)	(bourbon,,)	clients
	hosts	daniel		jimmy		systems

Netgroups can be used in various files that define permissions. Used in /etc/exports, they can specify which hosts may mount filesystems. Used in /etc/hosts.equiv or in a user's .rhosts file, they can grant or revoke login permissions for groups of users or machines.

Netgroups used in an NIS client's /etc/passwd file import account information for the members of a particular netgroup. The syntax is +@netgroup.

How to set up NIS

NIS must be configured on the master server, on the slave servers, and on each client. In the following discussions, dutta is the NIS domain name for the servers and clients.

Execute the following steps on the host that is to serve as the master server:

1. Create a domain directory called /var/yp/dutta:

   cd /var/yp /* The NIS directory */ /usr/bin/domainname dutta /* create a domain named dutta */

   
   
   
   
2. Clean up the following system admin files:
   • /etc/passwd
   • /etc/group
   • /etc/hosts
   • /etc/networks
   • /etc/services
   • /etc/protocols
   • /etc/ethers
   • /etc/aliases
   • /etc/rpc
   • /etc/netgroup
   
   
3. Check the /etc/passwd file on master server. If it contains +, remove it.
   
   
4. Run /usr/etc/yp/ypinit -m.

   The ypinit command builds a complete set of admin maps and places them in the domain directory /var/yp/dutta. The first map is ypservers. Then ypinit prompts for a list of all servers that will run NIS and adds these servers to ypservers. There should be only one master server. Note that ypinit does not check if there is another. Having more than one master server confuses the procedures that contact the NIS master for map transfers and NIS password file updates.

   The ypservers file does not correspond to any flat file. Its contents are examined whenever the master server needs to distribute maps to slaves.

5. When ypinit finishes, start NIS manually by executing the command /usr/etc/ypserv or by rebooting the server.

In AIX, the NIS domain name as well as the initiation of ypinit are usually set in the /etc/rc.nfs script for every server and client.

Create the slave servers

When the master is up and running, configure each slave server by executing the following steps on each slave:

cd /var/yp				/* The NIS directory */
/usr/bin/domainname dutta		/* This slave uses dutta domain*/
/usr/etc/yp/ypinit -s masterServer     /* masterServer -> master's hostname */
/usr/etc/ypserv>		        /* Start the Slave Server */

Running ypinit -s transfers the system configuration data from the master server and builds local copies of the maps on the slave. No ASCII source files are used to build the local maps.

The presence of the domain's data files is enough to let ypserv know that it should serve the domain.

If the master and slave are on two different IP networks, you can perform this initialization by setting up the slave as an NIS client (described below) and using ypset command to explicitly point it at the NIS master:

ypset -d dutta -h homer krusty

In this example, the ypset command causes the host named homer to bind to the server named krusty in the domain named dutta.

Adding slave servers later

As mentioned before, the list of servers is contained in the file ypservers. You can add new servers to this list by executing the following steps on the master server:

1. cd /var/yp/dutta
2. ypcat -k ypservers > /tmp/ypservers
3. Edit the file /tmp/ypservers and add the new server name and save the file.
4. cat /tmp/ypservers | makedbm - /var/yp/`domainname`/ypservers

Now, execute the following commands on the new slave to initialize it and to start NIS:

/usr/etc/yp/ypinit -s masterServer
/usr/etc/ypserv

Enable NIS on clients

Take the following steps on each client to enable NIS:

1. Make sure at least one server is running ypserv.
2. Make sure each of the configuration files on the clients includes + at the end of the files.
3. Execute /usr/bin/domainname dutta.
4. Execute /usr/etc/ypbind.

These steps start the ypbind daemon, which is responsible for locating NIS servers and maintaining bindings of domain names to the servers.

When NIS is running, client references to the basic administrative information are resolved in two different ways depending on whether that information is available in Local Priority files or Global Priority files:

1. The NIS database ignores local copies of the following files and uses its own copies as soon as the ypbind daemon is started:
   • /etc/hosts
   • /etc/networks
   • /etc/services
   • /etc/protocols
   • /etc/ethers
   • /etc/rpc
   • /etc/netgroup
2. NIS appends information to these files: /etc/passwd, /etc/bootparams, /etc/group, and /etc/aliases. NIS looks up the local files first. If it does not find an entry there, NIS consults the corresponding NIS map. For example, when a user logs in, an NIS client first looks up the user's login name in the local passwd file; if it does not find it, it looks up the NIS passwd map.

Each client must have at minimum a private version of the passwd, group, and hosts files. The passwd and group files are necessary to allow root to log in when no NIS server is available. They should contain the standard system accounts and groups (root, bin, daemon, and wheel). The hosts file must be present to answer boot-time queries that occur before NIS is up and running.

The internals of NIS

The NIS data files and most of its commands are stored in one directory, usually /var/yp, /usr/etc/yp, or /etc/yp. This is sometimes called the NIS directory. Each NIS map is stored as a pair of ndbm files, one called map.dir and the other called map.pag, in a subdirectory of the NIS directory named for the NIS domain. For example, in the domain dutta, the ndbm files for the /etc/passwd maps might be the following:

/usr/etc/yp/dutta/passwd.byname.dir
/usr/etc/yp/dutta/passwd.byname.pag
/usr/etc/yp/dutta/passwd.byuid.dir
/usr/etc/yp/dutta/passwd.byuid.pag


Note that for each field, a separate map is required with which to search the file. The passwd file is searchable by both name and user ID, so two maps (four files) are derived from it.

You can copy maps from the master server to the slave servers by executing the ypxfr command on the slave servers. (ypxfr is a pull command.) A daemon called ypxfrd runs on the master server to respond to the requests from the slaves. ypxfr does not transfer a map unless it is out of date. Slaves usually execute ypxfr every so often just to verify that they have the most recent maps. You can also set up ypxfr to run from cron. On each slave, you could set up crontab entries that would pull fresh copies of all maps from the master. You can use ypxfr map, where map is a name such as passwd.byuid, to transfer the specified map from the master server. This command has to be run once for each map. In most circumstances, transferring all the maps once or twice a day (perhaps late at night) is good enough. Run the following Korn Shell script from cron on the slaves to transfer maps from the master server:

	#!/bin/ksh
	mydomain = '/usr/bin/domainname'
	cd /var/yp/$mydomain
	for map in '/usr/bin/ls'
	do
		/usr/etc/ypxfr  $map
	done

Note that yppush, a push version of the ypxfr command, is run on the master server. It actually does not transfer any data, but rather instructs each slave to execute a ypxfr. The makefile command uses yppush in the NIS directory to ensure that newly updated maps are propagated to slaves.

After the initial configuration, the only active components of the NIS system are the ypserv and ypbind daemons. ypserv runs only on servers (both master and slave); it accepts queries from clients and answers them by looking up information in the ndbm maps. ypbind runs on every machine in the NIS domain, including servers. The C library contacts the localypbind daemon whenever it needs to answer an administrative query. ypbind locates a ypserv in the appropriate domain and returns its identity to the C library, which then contacts the server directly. A ypbind on a server machine does not give itself preferential treatment, so servers do not necessarily bind to themselves.

NIS commands you cannot live without

Table 2. NIS commands

Back to top

Conclusion

In a multi-system environment, there is no need to maintain identical copies of system configuration files on each system. Based on the complexity of environment, either of the two methods discussed here would facilitate maintenance of these files without any difficulty. By carefully designing the password files, you could even build access security into the system.

Resources

• For background reading, check out the "Network Information Services (NIS and NIS+) Guide".
  
  
• You'll find useful information in Managing NFS and NIS by Hal Stern (O’Reilly & Associates, Inc., ISBN 0-937175-75-7).
  
  
• Another good reference is the Unix System Administration Handbook by Evi Nemeth et al. (Prentice Hall, ISBN 0-13-151051-7).
  
  

About the author

Comments

Back to top

Trademarks  |  My developerWorks terms and conditions

Table of contents

• Introduction
• Overview
• Configuration files -- what to share
• Distribution by file copying
• The Network Information Service
• Conclusion
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

My developerWorks community

Dig deeper into AIX and UNIX on developerWorks

Calling all developers

Download no charge Lotus Domino Designer

Special offers