WHY DOES SEND() RETURN EAGAIN / EWOULDBLOCK?

Body

This small article tries to answer the following question that came via support channel:

  "What does it mean really when send() fails with EAGAIN/EWOULDBLOCK?"

So... without going to much into details let's try to answer this. First let's walk over a few definitions, once again trying to keep a simple vocabulary here.

- socket -

A socket is special file used in network transports. In a very simple way you could see a socket as a 'pipe' that can be used to send data over the network. So it has 2 ends, usually a server and a client that reside on the same or two different machines. When you write data to a regular file you can write as long as the container (file system - or - quotas) allows you to do so. For a socket data goes into a buffer and then that buffer is transported over the network. This leads us to the 'blocking / non blocking' concept.

- blocking / non blocking -

The 'sender' can send data using the 'send()' call. As said above, that data goes into a buffer that will then be transmitted to the 'receiver' via the socket. Of course the buffer has a fixed size (configurable) and you want to make sure that the 'receiver' read all the data you sent before starting to 'overwrite' the buffer with new data. So what happens if you decide to 'send()' data but the buffer is full? Well... if your socket is in 'blocking' mode then the 'send()' will 'hang' until there is room in the buffer, that is until the 'receiver' has read data from the current buffer and therefore made some space available in the buffer. This is because once data is read the space it was using in the buffer is made available for new data. If your socket is in 'non blocking' mode then the 'send()' will fail with 'EAGAIN' or 'EWOULDBLOCK' (often the same error number)
meaning 'try again later / it would block'. Both 'blocking' and 'non blocking' have their plus and minus. For example you might not want to 'hang' if the receiver did not read all the data yet so that you can go on and do something else, that
would be the 'non blocking' mode. Or you can decide that there is no point to do anything else as long as the data is not sent and 'hang' waiting for the send to complete, that would be the 'blocking' mode.

To illustrate this you have 2 small programs below. One server and one client. Those are simple programs just to demonstrate 'blocking' versus 'non blocking'. Real life client/server programs should most likely be a bit more elaborate but those will do for what we have here. And to make it even simpler both have to be run on the same box. You can alter them if you want, of course, to work on different machines.

Both client and server programs need to be started with a 'port number' for the socket and an extra argument, 0 or 1, indicating respectively 'blocking' or 'non blocking' mode. The server keeps sending data in a loop to the client over the socket and the buffer size for the socket is set to 16K.

To compile the programs:

  AIX:

  cc -q64 server3.c -o server3
  cc -q64 client3.c -o client3

  Solaris:

  cc -m64 -lnsl -lsocket server3.c -o server3
  cc -m64 -lnsl -lsocket client3.c -o client3

  Linux (X86):

  cc -m64 server3.c -o server3
  cc -m64 client3.c -o client3

You then need to run 'server3' in one window and 'client3' in another window.

Let's start with 'blocking mode':

  WINDOW 1                                    WINDOW 2

  [/home/dalla/tmp]->server3 61433 0

                                              [/home/dalla/tmp]->client3 61433 0
                                              [info] press return to receive data from server:

  [info] send(aaaa...) size 8192 [sent so far 0]: sent
  [info] send(bbbb...) size 8192 [sent so far 8192]: sent
  [info] send(cccc...) size 8192 [sent so far 16384]: sent
  [info] send(dddd...) size 8192 [sent so far 24576]: sent

  As soon as the client is connected the server starts sending data.
  Our buffer is 16K but as you can see, under the covers the size seem
  to be doubled and we can accommodate a bit more.

  At this time the server is 'hanging' because our socket is 'blocking'
  and server has no other choice but wait until some space is made
  available in the buffer. And this will happen if we press 'return'
  in WINDOW 2 where the client is running.

                                               [info] received 8192 bytes [so far 8192] [aaaa]
                                               [info] press return to receive data from server:

  And sure enough, as soon as we hit 'return' on the client
  side we see that the server is able to send more data and then
  'hang' again waiting for some more room in the buffer.

  [info] send(eeee...) size 8192 [sent so far 32768]: sent

  And this will go on and on...

Now let's see what happens when we are using 'non blocking' mode for the socket. We use the same 2 windows but the last argument to both 'server3' and 'client3' will be '1' to indicate 'non blocking':

  WINDOW 1                                    WINDOW 2

  [/home/dalla/tmp]->server3 61433 1

                                              [/home/dalla/tmp]->client3 61433 1
                                              [info] press return to receive data from server:

  [info] send(aaaa...) size 8192 [sent so far 0]: sent
  [info] send(bbbb...) size 8192 [sent so far 8192]: sent
  [info] send(cccc...) size 8192 [sent so far 16384]: EAGAIN
  [info] send(cccc...) size 8192 [sent so far 16384]: sent
  [info] send(dddd...) size 8192 [sent so far 24576]: sent
  [info] send(eeee...) size 8192 [sent so far 32768]: EAGAIN
  [info] send(eeee...) size 8192 [sent so far 32768]: EAGAIN
  [info] send(eeee...) size 8192 [sent so far 32768]: EAGAIN
  [info] send(eeee...) size 8192 [sent so far 32768]: EAGAIN

  Similar to the previous case, as soon as the client establishes
  the connection we see the server sending data. However, when there
  is no room left in the buffer the server doesn't hang anymore but
  instead the 'send()' fails with EAGAIN. In the server code you
  will see that it does a 'sleep()' before trying again. This goes
  on until the client reads the first batch of data which is done
  by pressing 'return' in WINDOW 2.

                                              [info] received 8192 bytes [so far 8192] [aaaa]
                                              [info] press return to receive data from server:

  Client read some data and the space is made available
  in the buffer. So server is able to send some more. Then it
  fails again with EAGAIN and this goes on and on...

  [info] send(eeee...) size 8192 [sent so far 32768]: sent
  [info] send(ffff...) size 8192 [sent so far 40960]: EAGAIN
  [info] send(ffff...) size 8192 [sent so far 40960]: EAGAIN
  [info] send(ffff...) size 8192 [sent so far 40960]: EAGAIN

Now on to the source code for 'server3.c' and 'client3.c'. Those are simple programs you can play with. Once again in real life client/server code would probably be a bit more sophisticated but I tried to keep them simple just for the purpose of this small article.

== server3.c ==

#include <stdio.h>
#include <stdlib.h>
#include <strings.h>
#include <unistd.h>
#include <time.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <poll.h>
#include <signal.h>

#define BUFSIZE           8192
#define MAXSEND           512

int
main(int ac, char **av)
{
    char                 *bf;
    int                   max;
    int                   status;
    int                   port;
    int                   serverfd;
    int                   clientfd;
    int                   optval;
    int                   nonblock;
    unsigned int          addrsize;
    ssize_t               size;
    size_t                totalsent;
    struct sockaddr_in    server_addr;
    struct sockaddr_in    client_addr;

    /*
     * Check arguments. Collect the user supplied port number to use.
     */
    if (ac < 3) {
        printf("[error] Usage: %s <portnumber> <block/nonblock (0/1)>\n", av[0]);
        exit(1);
    }

    port = atoi(av[1]);
    nonblock = atoi(av[2]);

    /*
     * Allocate a working buffer.
     */
    if ((bf = (char *) malloc(BUFSIZE)) == 0) {
        printf("[error] malloc(bf), [errno %d]\n", errno);
        exit(1);
    }

    /*
     * For this example we will only work on the local server.
     */
    if (gethostname(bf, BUFSIZE)) {
        printf("[error] gethostname(), [errno %d]\n", errno);
        exit(1);
    }

    /*
     * Open a socket for receiving connections. We are the server.
     */
    if ((serverfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
        printf("[error] socket(), [errno %d]\n", errno);
        exit(1);
    }

    status = 1;

    (void) memset((void *) &server_addr, 0, sizeof(struct sockaddr_in));
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = INADDR_ANY;
    server_addr.sin_port = htons(port);

    addrsize = sizeof(struct sockaddr_in);

    if (bind(serverfd, (struct sockaddr *) &server_addr, addrsize) < 0) {
        printf("[error] bind(), [errno %d]\n", errno);
        goto cleanup;
    }

    /*
     * For this small example we only accept 1 connection. So we use a backlog
     * queue to fit only 1 entry...
     */
    if (listen(serverfd, 1) < 0) {
        printf("[error] listen(), [errno %d]\n", errno);
        goto cleanup;
    }

    /*
     * Check incoming connection. We are only expecting 1 here.
     * By default the socket is blocking so we can step into the 'accept()'
     * call and it will wait until a connection request comes in. In real
     * life a better approach would be to 'poll' the socket to check if
     * if we have any pending request.
     */
    addrsize = sizeof(struct sockaddr_in);

    clientfd = accept(serverfd, (struct sockaddr *) &client_addr, &addrsize);
    if (clientfd < 0) {
        printf("[error] accept(), [errno %d]\n", errno);
        goto cleanup;
    }

    optval = BUFSIZE * 2;
    if (setsockopt(clientfd, SOL_SOCKET, SO_SNDBUF,
                   (void *) &optval, sizeof(int))) {
        printf("[error] setsockopt(SO_SNDBUF), [errno %d]\n", errno);
        goto cleanup;
    }

    /*
     * Check if we should go for non blocking mode...)
     */
    if (nonblock) {
        if (fcntl(clientfd, F_SETFL, O_NONBLOCK) < 0) {
            printf("[error] fcntl(O_NONBLOCK), [errno %d]\n", errno);
            goto cleanup;
        }
    }

    /*
     * We now have a connection. In real life we would either fork() or create
     * a new thread to handle the connection. Here, since we won't receive
     * any other connection we simply handle it in the main program.
     * The idea is to show what would generate a 'EWOULDBLOCK/EAGAIN' error
     * on a send() call. So what we do is write data to the client until we
     * hit the error.
     */
    totalsent = 0;
    max = 0;

    (void) memset((void *) bf, (int) ((max % 26) + (int) 'a'), BUFSIZE);

    while (max < MAXSEND) {
        printf("[info] send(%c%c%c%c...) size %ld [sent so far %ld]: ",
               bf[0], bf[1], bf[2], bf[3], BUFSIZE, totalsent);

        if ((size = send(clientfd, (void *) bf, BUFSIZE, 0)) != BUFSIZE) {
            if (errno == EWOULDBLOCK || errno == EAGAIN) {
                printf("%s\n", errno == EAGAIN ? "EAGAIN" : "EWOULDBLOCK");
                sleep(5);
            } else {
                printf("errno %d\n", errno);
                goto cleanup;
            }
        } else {
            printf("sent\n");
            max++;
            totalsent += BUFSIZE;
            (void) memset((void *) bf, (int) ((max % 26) + (int) 'a'), BUFSIZE);
        }
    }

    /*
     * Close our connection.
     */
    close(clientfd);

    status = 0;

cleanup:
    close(serverfd);

    exit(status);
}

== client3.c ==

#include <stdio.h>
#include <stdlib.h>
#include <strings.h>
#include <unistd.h>
#include <time.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <netinet/in.h>
#include <poll.h>
#include <signal.h>

#define BUFSIZE           8192
#define MAXRECV           512

int
main(int ac, char **av)
{
    char                 *bf;
    int                   max;
    int                   status;
    int                   cc;
    int                   port;
    int                   connfd;
    int                   clientfd;
    int                   optval;
    int                   nonblock;
    unsigned int          addrsize;
    ssize_t               size;
    size_t                totalrecv;
    struct hostent       *server;
    struct sockaddr_in    server_addr;

    /*
     * Check arguments. Collect the user supplied port number to use.
     */
    if (ac < 3) {
        printf("[error] Usage: %s <portnumber> <block/nonblock (0/1)>\n", av[0]);
        exit(1);
    }

    port = atoi(av[1]);
    nonblock = atoi(av[2]);

    /*
     * Allocate a working buffer.
     */
    if ((bf = (char *) malloc(BUFSIZE)) == 0) {
        printf("[error] malloc(bf), [errno %d]\n", errno);
        exit(1);
    }

    /*
     * For this example we will only work on the local server.
     */
    if (gethostname(bf, BUFSIZE)) {
        printf("[error] gethostname(), [errno %d]\n", errno);
        exit(1);
    }

    /*
     * Open a socket for connecting to the server. We restrict the size
     * of buffers so that we can easily demonstrate our test.
     */
    if ((connfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) {
        printf("[error] socket(), [errno %d]\n", errno);
        exit(1);
    }

    status = 1;

    optval = BUFSIZE * 2;
    if (setsockopt(connfd, SOL_SOCKET, SO_RCVBUF,
                   (void *) &optval, sizeof(int))) {
        printf("[error] setsockopt(SO_RCVBUF), [errno %d]\n", errno);
        goto cleanup;
    }

    /*
     * Check if we should go for non blocking mode...)
     */
    if (nonblock) {
        if (fcntl(connfd, F_SETFL, O_NONBLOCK) < 0) {
            printf("[error] fcntl(O_NONBLOCK), [errno %d]\n", errno);
            goto cleanup;
        }
    }

    /*
     * Collect our server address for the 'connect()' call.
     */
    if ((server = gethostbyname(bf)) == 0) {
        printf("[error] gethostbyname(), [errno %d]\n", errno);
        goto cleanup;
    }

    /*
     * Connect to the server.
     */
    size = server->h_length;
    (void) memset((void *) &server_addr, 0, sizeof(struct sockaddr_in));
    memcpy((void *) &server_addr.sin_addr.s_addr, (void *) server->h_addr, size);
    server_addr.sin_family = AF_INET;
    server_addr.sin_port = htons(port);

    addrsize = sizeof(struct sockaddr_in);

    /*
     * The while loop is only for Linux where sometimes the connect
     * would fail with EINPROGRESS. We just wait and retry.
     */
    while (connect(connfd, (struct sockaddr *) &server_addr, addrsize) < 0) {
        if (errno == EINPROGRESS) {
            sleep(1);
            continue;
        }
        printf("[error] connect(), [errno %d]\n", errno);
        goto cleanup;
    }

    /*
     * We now have a connection. For this small example what we need to
     * do is 'recv()' only when the user does request it by pressing return.
     */
    totalrecv = 0;
    max = 0;
    do {
        size = 0;
        (void) memset((void *) bf, 0, BUFSIZE);
        printf("[info] press return to receive data from server: ");
        cc = getc(stdin);

        while ((size = recv(connfd, (void *) bf, BUFSIZE, 0)) < BUFSIZE) {
            if (errno == EWOULDBLOCK || errno == EAGAIN) {
                printf("[warning] recv(), [errno %s]\n",
                       errno == EAGAIN ? "EAGAIN" : "EWOULDBLOCK");
                sleep(5);
            } else {
                printf("[error] recv(), [errno %d]\n", errno);
                break;
            }
        }

        max++;
        totalrecv += BUFSIZE;
        printf("[info] received %ld bytes [so far %ld] [%c%c%c%c]\n",
               BUFSIZE, totalrecv, bf[0], bf[1], bf[2], bf[3]);
    } while (max < MAXRECV);

    /*
     * Close our connection.
     */
    close(connfd);

    status = 0;

cleanup:
    close(connfd);

    exit(status);
}