Introduction

Processes that the UNIX kernel manages operate autonomously, which leads to a more stable system. However, every developer eventually runs into a situation where one group of processes needs to talk to another group of processes, perhaps to exchange data or send commands. This communication is called Inter-Process Communication (IPC). The System V (SysV) UNIX specification describes these three mechanisms for IPC, collectively referred to as SysV IPC:

• Message queues
• Semaphores
• Shared memory

In addition, processes can communicate through other means, such as:

• Reading, writing, and locking files
• Signals
• Sockets
• Pipes
• FIFOs (First In, First Out)

The latter group is also generically referred to as IPC. This article focuses on the SysV IPC methods because of their simplicity and effectiveness.

Understanding the SysV model

The three SysV IPC methods have similar syntax despite their different purposes. In general, do the following:

1. Determine the proper IPC key to use with ftok(3).
   
   
2. Get the IPC-specific identifier associated with the IPC key using msgget(2), semget(2), or shmget(2) for message queues, semaphores, or shared memory, respectively.
   
   
3. Modify attributes of the IPC instance with msgctl(2), semctl(2), or shmctl(2).
4. Make use of the specific IPC instance.
   
   
5. At the end, destroy the IPC instance with msgctl(2), semctl(2), or shmctl(2) and the IPC_RMID flag.

Each instance of an IPC is given an identifier to distinguish it from other instances of the IPC that exist on the system. For example, two different applications might each decide to use shared memory segments, so the system-wide IPC ID distinguishes between the two instances. It might not be immediately obvious, but the first challenge is to figure out how to distribute the information regarding how to attach to a common IPC instance without having an IPC mechanism in the first place.

The ftok library call uses inode information from a given file and a unique identifier to come up with a key that stays the same as long as the file exists and the identifier is constant. Thus, two processes could use their configuration files and compile-time constants to come up with the same IPC key. The presence of the constant allows the same application to create multiple instances of an IPC mechanism by varying the constant.

Once a set of processes have independently come up with their IPC keys, they must then get a specific identifier associated with the particular IPC instance using one of the get system calls. The get calls all require the IPC key and a set of flags, as well as some size information for semaphores and shared memory. Because UNIX is a multiuser system, the flags include file permissions in the familiar octal format (so 666 means anyone can read and write). If the IPC_CREAT flag is also set, the IPC instance will be created if it does not exist. If the IPC_CREAT flag is not set and the IPC instance has not been created, the get call returns an error.

A simpler way to distribute the IPC instance identifier exists for those applications with the capability to do so themselves. If you use a key of IPC_PRIVATE when calling get to create the IPC, the instance identifier will be unique. The other processes that need to attach to the IPC do not need to call get because they already have the identifier.

Applications are free to use the IPC instances once they have the identifier. Each IPC method is different and is addressed in its own section.

Passing messages through queues

Message queues provide a mechanism for one process to post a message that another process can pick up. Once the message is picked up, it is removed from the queue. Message queues are unique in that both processes do not have to exist simultaneously -- one process could post a message and exit, and the message could be picked up days later.

Messages must consist of a long integer followed by the message data. Listing 1 shows such a structure in C using a 100-byte message.

struct mq_message {
    long type;  /* The type or destination */
    char text[100]; /* Data */
};

The receiver of the message uses the message type. When pulling a message from the queue, you can select the first available message or you can look for a particular message type. The message types to use are application-specific, making queues unique from other forms of IPC in that the kernel has some understanding of the application data being passed by reading the type field.

Listing 2 shows the message submission part of message queuing.

#include <sys/types.h>
#include <sys/msg.h>
#include <sys/ipc.h>
#include <string.h>
#include <stdio.h>

int main (void) {

        key_t ipckey;
        int mq_id;
        struct { long type; char text[100]; } mymsg;

        /* Generate the ipc key */
        ipckey = ftok("/tmp/foo", 42);
        printf("My key is %d\n", ipckey);

        /* Set up the message queue */
        mq_id = msgget(ipckey, IPC_CREAT | 0666);
        printf("Message identifier is %d\n", mq_id);

        /* Send a message */
        memset(mymsg.text, 0, 100); /* Clear out the space */
        strcpy(mymsg.text, "Hello, world!");
        mymsg.type = 1;
        msgsnd(mq_id, &mymsg, sizeof(mymsg), 0);

}

The code in Listing 2 includes the necessary header files and then defines the variables to be used within the main function. The first order of business is to determine the IPC key using /tmp/foo as the common file and the number 42 as the ID. For demonstration purposes, this key is displayed on the screen using printf(3c). Next, the message queue is created using msgget. The first parameter to msgget is the IPC key, and the second is a set of flags. In the example, the flags include the octal permissions, which allow anyone with the IPC key to fully use this IPC, and the IPC_CREAT flag, which causes msgget to create the queue. Again, the result is printed to the screen.

Sending the message to the queue is simple. After zeroing out the memory space in the message, a familiar string is copied to the text part of the buffer. The message type is set to 1, and then msgsnd is called. msgsnd expects to be passed the queue ID, a pointer to the data, the size of the data, and a flag indicating whether the call should block or not. If the flag is IPC_NOWAIT, the call returns even if the queue is full. If the flag is 0, the call blocks until space is free on the queue, the queue is deleted, or the application receives a signal.

The client side of the equation is similar. Listing 3 shows code that retrieves the message that the server sent.

#include <sys/types.h>
#include <sys/msg.h>
#include <sys/ipc.h>
#include <string.h>
#include <stdio.h>

int main (void) {

        key_t ipckey;
        int mq_id;
        struct { long type; char text[100]; } mymsg;
        int received;

        /* Generate the ipc key */
        ipckey = ftok("/tmp/foo", 42);
        printf("My key is %d\n", ipckey);

        /* Set up the message queue */
        mq_id = msgget(ipckey, 0);
        printf("Message identifier is %d\n", mq_id);

        received = msgrcv(mq_id, &mymsg, sizeof(mymsg), 0, 0);

        printf("%s (%d)\n", mymsg.text, received);
}

The procedure to get the IPC key and the message queue identifier is similar to the server's code. The call to msgget does not specify any flags, because the server has already created the queue. If the application were designed such that the client might be started before the server, then both the client and server would have to specify permissions and the IPC_CREAT flag so that whichever application started first would create the queue.

mq_client.c then calls msgrcv to pull a message off the queue. The first three arguments specify the message queue identifier, a pointer to the memory space that will hold the message, and the size of the buffer. The fourth parameter is the type parameter, and it allows you to be selective about which messages you get:

• If the type is 0, the first message in the queue is returned.
• If the type is a positive integer, the first message in the queue with the same type is returned.
• If the type is a negative integer, the first message in the queue with the lowest value that is less than or equal to the absolute value of the specified type is returned. For example, if 2 and then 1 were to be added to the queue, calling msgrcv with a type of -2 would return the 1 because it is the smallest, even though it was second in the queue.

The fifth parameter to msgrcv is the blocking flag again. Listing 4 shows the client and server in action.

sunbox$ ./mq_server
My key is 704654099
Message identifier is 2
sunbox$ ./mq_client
My key is 704654099
Message identifier is 2
Hello, world! (104)

The output of the client and server show that they both came up with the same IPC key because they referenced the same file and identifier. The server created the IPC instance, which the kernel assigned the value of 2, and the client application learned this. It should then come as no surprise that the client pulls "Hello, world!" back from the message queue.

This example shows the simplest of situations. A message queue would be helpful for a short-lived process such as a Web transaction that submits work to a heavier back-end application such as some batch processing. A client can also be a server, and multiple applications can submit messages to the queue. The message type field allows applications to direct messages to particular readers.

Back to top

Locking resources with semaphores

Communication between processes does not need to involve sending large amounts of data. Indeed, a single bit might suffice to indicate that a process is using a certain resource. Consider two processes that need to access a piece of hardware, but only one of them at a time may use it. Each could agree on a spot to hold a reference counter. If one process reads the counter and sees 1, it is understood that another process is using the hardware. If the value of the counter is 0, then the process is free to use the hardware resource as long as the counter is set to 1 for the duration of the hardware operation and reset to 0 at the end.

There are two problems with this scenario. The first is the simple matter of setting up the shared counter and agreeing upon its location, which is more of an inconvenience than anything else. The second is that the get and set operations needed to lock the hardware resource are not atomic. If one process were to read the counter as 0 but become preempted by another process before it had the chance to set the counter to 1, the second process might be able to read and set the counter. Both processes would think they could use the hardware. There is no way to know if the other process (or processes) also set the counter. This is called a race condition. Semaphores solve both these problems by providing a common interface to applications and by implementing an atomic test or set operation.

The SysV implementation of semaphores is more general than the described solution. First, the value of a semaphore does not need to be 0 or 1; it can be 0 or any positive integer. Second, a series of semaphore operations is possible, much like the type parameter used with msgrcv. These operations are given as a set of instructions to the kernel, and either they all run or none of them run. The kernel requires that these instructions go in a structure called sembuf, which has the following members (in order):

1. sem_num: A description of which semaphore from the set is being acted on.
2. sem_op: A signed integer containing the instruction or test to be performed.
3. sem_flg: A combination of the familiar IPC_NOWAIT flag, which indicates if the test should return immediately or block until it passes, and SEM_UNDO, which causes the semaphore operations to be undone if the process should exit prematurely.

sem_op is where the bulk of the configuration goes:

• If sem_op is 0, sem_num is tested to see if it is 0. If sem_num is 0, the next test runs. If sem_num is not 0, either the operation blocks until the semaphore becomes 0 if IPC_NOWAIT is not set, or the rest of the tests are skipped if IPC_NOWAIT is set.
  
  
• If sem_op is a positive integer, the value of sem_op is added to the value of the semaphore.
  
  
• If sem_op is a negative integer and the value of the semaphore is greater than or equal to the absolute value of sem_op, then the absolute value is subtracted from the value of the semaphore.
  
  
• If sem_op is a negative integer and the value of the semaphore is less than the absolute value of sem_op, then the execution of tests immediately stops if IPC_NOWAIT is true, or blocks until the semaphore's value becomes greater than the absolute value of sem_op if false.

The example in Listing 5 clarifies the use of semaphores by examining a program that can be run several times simultaneously, but it ensures that only one process is in the critical section at a time. The simple case of semaphores is used; the resource is free if the value of the semaphore is 0.

#include <sys/types.h>
#include <sys/sem.h>
#include <sys/ipc.h>
#include <string.h> /* For strerror(3c) */
#include <errno.h> /* For errno */
#include <unistd.h> /* rand(3c) */
#include <stdio.h>

int main (int argc, char **argv) {

        key_t ipckey;
        int semid;
        struct sembuf sem[2]; /* sembuf defined in sys/sem.h */

        /* Generate the ipc key */
        ipckey = ftok("/tmp/foo", 42);

        /* Set up the semaphore set. 4 == READ, 2 == ALTER */
        semid = semget(ipckey, 1, 0666 | IPC_CREAT);
        if (semid < 0) {
                printf("Error - %s\n", strerror(errno));
                _exit(1);
        }

        /* These never change so leave them outside the loop */
        sem[0].sem_num = 0;
        sem[1].sem_num = 0;
        sem[0].sem_flg = SEM_UNDO; /* Release semaphore on exit */
        sem[1].sem_flg = SEM_UNDO; /* Release semaphore on exit */
        while(1) { /* loop forever */
                printf("[%s] Waiting for the semaphore to be released\n", argv[1]);
                /* Set up two semaphore operations */
                sem[0].sem_op = 0; /* Wait for zero */
                sem[1].sem_op = 1; /* Add 1 to lock it*/
                semop(semid, sem, 2);
                printf("[%s] I have the semaphore\n", argv[1]);

                sleep(rand() % 3); /* Critical section, sleep for 0-2 seconds */

                sem[0].sem_op = -1; /* Decrement to unlock */
                semop(semid, sem, 1);
                printf("[%s] Released semaphore\n", argv[1]);

                sleep(rand() % 3); /* Sleep 0-2 seconds */
        }

}

Listing 5 starts out the way the message queue example does. Where msgget specifies the size of the message queue in the second parameter, semget specifies the size of the semaphore set. The semaphore set is a group of semaphores sharing a common IPC instance. The number of semaphores in the set cannot be changed. If the semaphore set has been created, the second parameter to semget is effectively ignored. If semget returns a negative integer indicating failure, the reason is printed, and the program exits.

Just before the main while loop, sem_num and sem_flg are initialized, because they stay consistent throughout this example. SEM_UNDO is specified so that if the holder of the semaphore exits before the semaphore can be released, all the other applications are not locked out.

Within the loop, a status message is printed to indicate that the application has begun to wait for the semaphore. This output is prefixed by the first command-line argument to distinguish it from other instances. Before entering the critical section, the application locks the semaphore. Two semaphore instructions are specified. The first is 0, which means that the application waits until the semaphore value returns to 0. The second is 1, which means that after the semaphore returns to 0, 1 is added to the semaphore. The application calls semop to run the instructions, passing it the semaphore ID, the address of the data structure, and the number of sembuf instructions to use.

After semop returns, the application knows it has locked the semaphore and prints a message to indicate this. The critical section then runs, which in this case is a pause for a random number of seconds. Finally, the semaphore is released by running a single sembuf command with a semop value of -1, which has the effect of subtracting 1 from the semaphore and returning its value to 0. More debugging output is printed, the application pauses randomly, and execution continues. Listing 6 shows the output of two instances of this application.

sunbox$ ./sem_example a & ./sem_example b &
[a] Waiting for the semaphore to be released
[a] I have the semaphore
[b] Waiting for the semaphore to be released
[a] Released semaphore
[b] I have the semaphore
[a] Waiting for the semaphore to be released
[b] Released semaphore
[a] I have the semaphore
[a] Released semaphore
[a] Waiting for the semaphore to be released
[a] I have the semaphore

Listing 6 shows two instances of the example being run, with the names of a and b, respectively. First, a obtains the semaphore, and while holding it, b tries to obtain a lock. As soon as a releases the semaphore, b is given the lock. The situation reverses itself with a waiting for b to finish. Finally, a gets the semaphore again after releasing it, because b was not waiting.

The last item to note about semaphores is that they are known as advisory locks. This means that the semaphores themselves do not prevent two processes from using the same resource simultaneously; rather, they are there to advise anyone willing to ask if the resource is already in use.

Back to top

Sharing memory space

Shared memory is perhaps the most powerful of the SysV IPC methods, and it is the easiest to implement. As the name implies, a block of memory is shared between processes. Listing 7 shows a program that calls fork(2) to split itself into a parent process and a child process, communicating between the two using a shared memory segment.

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>

int main(void) {

        pid_t pid;
        int *shared; /* pointer to the shm */
        int shmid;

        shmid = shmget(IPC_PRIVATE, sizeof(int), IPC_CREAT | 0666);
        if (fork() == 0) { /* Child */
                /* Attach to shared memory and print the pointer */
                shared = shmat(shmid, (void *) 0, 0);
                printf("Child pointer  %p\n", shared);
                *shared=1;
                printf("Child value=%d\n", *shared);
                sleep(2);
                printf("Child value=%d\n", *shared);

        } else { /* Parent */
                /* Attach to shared memory and print the pointer */
                shared = shmat(shmid, (void *) 0, 0);
                printf("Parent pointer %p\n", shared);
                printf("Parent value=%d\n", *shared);
                sleep(1);
                *shared=42;
                printf("Parent value=%d\n", *shared);
                sleep(5);
                shmctl(shmid, IPC_RMID, 0);
        }
}

The parameters to shmget should be familiar by now: key, size, and flags. The size of the shared memory area in this example is a single integer. Listing 7 differs from previous examples in its use of IPC_PRIVATE for the IPC key. When IPC_PRIVATE is used, a unique IPC ID is guaranteed to be created, and it is expected that the application distributes the ID itself. In the example, shmid is known by both the parent and child because they are copies of each other. The fork system call creates a second copy of the current process called the child process, which is virtually identical to the parent. Execution of both processes resumes after the fork. The return value is used to determine if the current process is the parent or child.

Both the parent and child look similar. First, the shmat system call is used to get a pointer to the shared memory segment. shmat requires the shared memory ID, a pointer, and some flags. The pointer is used to request a particular address of memory. By passing 0, the kernel is free to choose whatever it wants. The flags are mostly vendor-specific, however SHM_RDONLY is a common flag that indicates the segment not written to. As shown in Listing 7, the common usage of shmat is to let the kernel decide everything.

shmat returns a pointer to the shared memory segment, which for debugging purposes is printed to the screen. Each process then takes a turn modifying the shared memory segment and printing out the value. Finally, the parent process removes the shared memory segment using shmctl(2). Listing 8 shows the output of this program.

 
sunbox$ ./shared_memory
Child pointer  ff390000
Child value=1
Parent pointer ff380000
Parent value=1
Parent value=42
Child value=42

You can see the sharing of the same memory space from the output. First, the value in the shared memory is 1, which the child set and the parents read. Then the parent set the value to 42, which the child read. Note how the parent and child processes had a different pointer address to the shared memory segment, even though they were accessing the same physical memory. This causes a problem for some data structures, such as linked lists, when the physical address is used, so you can use relative addressing if you are building complex structures in shared memory.

This example relies on one process pausing while the other is writing to the shared memory segment. In a real application, this would not be practical, so if your application could potentially have multiple processes writing to the same memory location, consider locking that area with a semaphore.

Back to top

Conclusion

UNIX provides several methods for IPC. The SysV IPC methods are message queues, semaphores, and shared memory. Message queues allow one application to submit a message that other applications can pick up later, even after the sending application has finished. Semaphores ensure that multiple applications can lock resources and avoid race conditions. Shared memory allows multiple applications to share a common segment of memory, which provides a fast method of communicating and sharing large amounts of data. You can also use these methods together. For example, you can use a semaphore to control access to a shared memory segment.

IPC methods are helpful to application developers because they provide a standard way to communicate between applications and are portable across different UNIX flavors. The next time you find yourself needing to lock resources or share data between processes, try the SysV IPC mechanisms.

Resources

Learn

• "Port Windows IPC apps to Linux, Part 1: Processes and threads" (developerWorks, April 2005): This article is the first in a three-part series about porting Microsoft ® Windows® apps to Linux® that focuses on IPC.
  
  
• "Secure programmer: Prevent race conditions" (developerWorks, October 2004): This article takes a much deeper look at race conditions, which also can cause security problems.
  
  
• ipcs and ipcrm: Use these two commands to list and delete IPC instances from the command line.
  
  
• The Open Group: Visit the Open Group site for UNIX standards as well as a description of Interprocess Communication.
  
  
• AIX and UNIX articles: Check out other articles written by Sean Walberg.
  
  
• Search the AIX and UNIX library by topic:
  • System administration
  • Application development
  • Performance
  • Porting
  • Security
  • Tips
  • Tools and utilities
  • Java™ technology
  • Linux
  • Open source
  
  
• AIX and UNIX: The AIX and UNIX developerWorks zone provides a wealth of information relating to all aspects of AIX systems administration and expanding your UNIX skills.
  
  
• New to AIX and UNIX: Visit the New to AIX and UNIX page to learn more about AIX and UNIX.
  
  
• AIX 5L™ Wiki: A collaborative environment for technical information related to AIX.
  
  
• Safari bookstore: Visit this e-reference library to find specific technical resources.
  
  
• developerWorks technical events and webcasts: Stay current with developerWorks technical events and webcasts.
  
  
• Podcasts: Tune in and catch up with IBM technical experts.
  
  

Get products and technologies

• Software Distributed Shared Memory: This project from Rutgers University extends the concept of shared memory across machines. The project's goal is to use commodity hardware to solve parallel processing tasks normally run on massively parallel hardware.
  
  
• IBM trial software: Build your next development project with software for download directly from developerWorks.
  
  

Discuss

• Participate in the developerWorks blogs and get involved in the developerWorks community.
  
  
• Participate in the AIX and UNIX forums:
  • AIX 5L -- technical forum
  • AIX for Developers Forum
  • Cluster Systems Management
  • IBM Support Assistant
  • Performance Tools -- technical
  • Virtualization -- technical
  • More AIX and UNIX forums
  
  

About the author

Sean Walberg has been working with Linux and UNIX systems since 1994 in academic, corporate, and Internet service provider environments. He has written extensively about systems administration over the past several years. You can contact him at sean@ertw.com.

Rate this article

Comments

Back to top

Table of contents

• Introduction
• Understanding the SysV model
• Passing messages through queues
• Locking resources with semaphores
• Sharing memory space
• Conclusion
• Resources
• About the author
• Comments

Local resources

• IBM Innovation Center - Austin, TX
• IBM Innovation Center - Chicago, IL
• IBM Innovation Center - Dallas, TX
• IBM Innovation Center - San Mateo, CA
• IBM Innovation Center - Waltham, MA
• Technical Briefings in North America

Dig deeper into AIX and UNIX on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers