Asynchronous I/O

You may need to use asynchronous I/O for speed and efficiency in scientific programs that perform I/O for large amounts of data. Synchronous I/O blocks the execution of an application until the I/O operation completes. Asynchronous I/O allows an application to continue processing while the I/O operation is performed in the background. You can modify applications to take advantage of the ability to overlap processing and I/O operations. Multiple asynchronous I/O operations can also be performed simultaneously on multiple files that reside on independent devices. For a complete description of the syntax and language elements that you require to use this feature, see the XL Fortran Language Reference under the topics:

Execution of an asychronous data transfer operation

The effect of executing an asynchronous data transfer operation will be as if the following steps were performed in the order specified, with steps (6)-(9) possibly occurring asynchronously:

  1. Determine the direction of the data transfer.
  2. Identify the unit.
  3. Establish the format if one is present.
  4. Determine whether an error condition, end-of-file condition, or end-of-record condition has occurred.
  5. Cause the variable that you specified in the IOSTAT= specifier in the data transfer statement to become defined.
  6. Position the file before you transfer data.
  7. Transfer data between the file and the entities that you specified by the input/output list (if any).
  8. Determine whether an error condition, end-of-file condition, or end-of-record condition has occurred.
  9. Position the file after you transfer data.
  10. Cause any variables that you specified in the IOSTAT= and SIZE= specifiers in the WAIT statement to become defined.

Usage

You can use Fortran asynchronous READ and WRITE statements to initiate asynchronous data transfers in Fortran. Execution continues after the asynchronous I/O statement, regardless of whether the actual data transfer has completed.

A program may synchronize itself with a previously initiated asynchronous I/O statement by using a WAIT statement. There are two forms of the WAIT statement:

  1. In a WAIT statement without the DONE= specifier, the WAIT statement halts execution until the corresponding asynchronous I/O statement has completed:
    	integer idvar
    	integer, dimension(1000):: a
    	....
    	READ(unit_number,ID=idvar) a
    	....
    	WAIT(ID=idvar)
    	....
  2. In a WAIT statement with the DONE= specifier, the WAIT statement returns the completion status of an asynchronous I/O statement:
    	integer idvar
    	logical done
    	integer, dimension(1000):: a
    	....
    	READ(unit_number,ID=idvar) a
    	....
    	WAIT(ID=idvar, DONE=done)
    	....
    The variable you specified in the DONE= specifier is set to "true" if the corresponding asynchronous I/O statement completes. Otherwise, it is set to "false".

The actual data transfer can take place in the following cases:

Because of the nature of asynchronous I/O, the actual completion time of the request cannot be predicted.

You specify Fortran asynchronous READ and WRITE statements by using the ID= specifier. The value set for the ID= specifier by an asynchronous READ or WRITE statement must be the same value specified in the ID= specifier in the corresponding WAIT statement. You must preserve this value until the associated asynchronous I/O statement has completed.

The following program shows a valid asynchronous WRITE statement:

        program sample0
        integer, dimension(1000):: a
        integer idvar
        a = (/(i,i=1,1000)/)
        WRITE(10,ID=idvar) a
        WAIT(ID=idvar)
        end

The following program is not valid, because XL Fortran destroys the value of the asynchronous I/O identifier before the associated WAIT statement:

        program sample1
        integer, dimension(1000):: a
        integer idvar
        a = (/(i,i=1,1000)/)
        WRITE(10,ID=idvar) a
        idvar = 999     ! Valid id is destroyed.
        WAIT(ID=idvar)
        end

An application that uses asynchronous I/O typically improves performance by overlapping processing with I/O operations. The following is a simple example:

        program sample2
        integer  (kind=4), parameter :: isize=1000000, icol=5
        integer  (kind=4) :: i, j, k
        integer  (kind=4), dimension(icol) :: handle
        integer  (kind=4), dimension(isize,icol), static :: a, a1

!
!       Opens the file for both synchronous and asynchronous I/O.
!
        open(20,form="unformatted",access="direct", &
           status="scratch", recl=isize*4,asynch="yes")

!
!       This loop overlaps the initialization of a(:,j) with
!       asynchronous write statements.
!
!       NOTE: The array is written out one column at a time.
!             Since the arrays in Fortran are arranged in column
!             major order, each WRITE statement writes out a
!             contiguous block of the array.
!
        do 200 j = 1, icol
           a(:,j) = (/ (i*j,i=1,isize) /)
           write(20, id=handle(j), rec=j) a(:,j)
200     end do

!
!       Wait for all writes to complete before reading.
!
        do 300 j = 1, icol
           wait(id=handle(j))
300     end do

!
!       Reads in the first record.
!
        read(20, id=handle(1), rec=1) a1(:,1)

        do 400 j = 2, icol
           k = j - 1
!
!          Waits for a previously initiated read to complete.
!
           wait(id=handle(k))
!
!          Initiates the next read immediately.
!
           read(20, id=handle(j), rec=j) a1(:,j)
!
!          While the next read is going on, we do some processing here.
!
           do 350 i = 1, isize
              if (a(i,k) .ne. a1(i,k)) then
                 print *, "(",i,",",k,") &
                 &  expected ", a(i,k), " got ", a1(i,k)
              end if
350        end do
400     end do

!
!       Finish the last record.
!
        wait(id=handle(icol))

        do 450 i = 1, isize
           if (a(i,icol) .ne. a1(i,icol)) then
              print *, "(",i,",",icol,") &
              &  expected ", a(i,icol), " got ", a1(i,icol)
           end if
450     end do

        close(20)
        end

Performance

To maximize the benefits of asynchronous I/O, you should only use it for large contiguous data items.

It is possible to perform asynchronous I/O on a large number of small items, but the overall performance will suffer. This is because extra processing overhead is required to maintain each item for asynchronous I/O. Performing asynchronous I/O on a larger number of small items is strongly discouraged. The following are two examples:

  1. WRITE(unit_number, ID=idvar) a1(1:100000000:2)
  2. WRITE(unit_number, ID=idvar) (a2(i,j),j=1,100000000)

Performing asynchronous I/O on unformatted sequential files is less efficient. This is because each record might have a different length, and these lengths are stored with the records themselves. You should use unformatted direct access or unformatted stream access, if possible, to maximize the benefits of asynchronous I/O.

Compiler-generated temporary I/O items

There are situations when the compiler must generate a temporary variable to hold the result of an I/O item expression. In such cases, synchronous I/O is performed on the temporary variable, regardless of the mode of transfer that you specified in the I/O statement. The following are examples of such cases:

  1. For READ, when an array with vector subscripts appears as an input item:
    1.         integer a(5), b(3)
      
              b = (/1,3,5/)
              read(99, id=i) a(b)
    2.         real a(10)
              read(99,id=i) a((/1,3,5/))
  2. For WRITE, when an output item is an expression that is a constant or a constant of certain derived types:
    1.      write(99,id=i) 1000
    2.      integer a
           parameter(a=1000)
      
           write(99,id=i) a
    3.      type mytype
           integer a
           integer b
           end type mytype
      
           write(99,id=i) mytype(4,5)
  3. For WRITE, when an output item is a temporary variable:
    1.      write(99,id=i) 99+100
    2.      write(99,id=i) a+b
    3.      external ff
           real(8) ff
      
           write(99,id=i) ff()
  4. For WRITE, when an output item is an expression that is an array constructor:
         write(99,id=i) (/1,2,3,4,5/)
  5. For WRITE, when an output item is an expression that is a scalarized array:
         integer a(5),b(5)
         write(99,id=i) a+b

Error handling

For an asynchronous data transfer, errors or end-of-file conditions might occur either during execution of the data transfer statement or during subsequent data transfer. If these conditions do not result in the termination of the program, you can detect these conditions via ERR=, END= and IOSTAT= specifiers in the data transfer or in the matching WAIT statement.

Execution of the program terminates if an error condition occurs during execution or during subsequent data transfer of an input/output statement that contains neither an IOSTAT= nor an ERR= specifier. In the case of a recoverable error, if the IOSTAT= and ERR= specifiers are not present, the program terminates if you set the err_recovery runtime option to no. If you set the err_recovery runtime option to yes, recovery action occurs, and the program continues.

If an asynchronous data transfer statement causes either of the following events, a matching WAIT statement cannot run, because the ID= value is not defined:

XL Fortran thread-safe I/O library

The XL Fortran runtime library libxlf90_r.so provides support for parallel execution of Fortran I/O statements.

Synchronization of I/O operations

During parallel execution, multiple threads might perform I/O operations on the same file at the same time. If they are not synchronized, the results of these I/O operations could be shuffled or merged or both, and the application might produce incorrect results or even terminate. The XL Fortran runtime library synchronizes I/O operations for parallel applications. It performs the synchronization within the I/O library, and it is transparent to application programs. The purpose of the synchronization is to ensure the integrity and correctness of each individual I/O operation. However, the runtime does not have control over the order in which threads execute I/O statements. Therefore, the order of records read in or written out is not predictable under parallel I/O operations. Refer to Parallel I/O issues for details.

External files

For external files, the synchronization is performed on a per-unit basis. The XL Fortran runtime ensures that only one thread can access a particular logical unit to prevent several threads from interfering with each other. When a thread is performing an I/O operation on a unit, other threads attempting to perform I/O operations on the same unit must wait until the first thread finishes its operation. Therefore, the execution of I/O statements by multiple threads on the same unit is serialized. However, the runtime does not prevent threads from operating on different logical units in parallel. In other words, parallel access to different logical units is not necessarily serialized.

Functionality of I/O under synchronization

The XL Fortran runtime sets its internal locks to synchronize access to logical units. This should not have any functional impact on the I/O operations performed by a Fortran program. Also, it will not impose any additional restrictions to the operability of Fortran I/O statements except for the use of I/O statements in a signal handler that is invoked asynchronously. Refer to Use of I/O statements in signal handlers for details.

Parallel I/O issues

The order in which parallel threads perform I/O operations is not predictable. The XL Fortran runtime does not have control over the ordering. It will allow whichever thread that executes an I/O statement on a particular logical unit and obtains the lock on it first to proceed with the operation. Therefore, only use parallel I/O in cases where at least one of the following is true:

In these cases, results of the I/O operations are independent of the order in which threads execute. However, you might not get the performance improvements that you expect, since the I/O library serializes parallel access to the same logical unit from multiple threads. Examples of these cases are as follows:

For multiple threads to write to or read from the same sequential-access file, or to write to or read from the same stream-access file without using the POS= specifier, the order of records written out or read in depends on the order in which the threads execute the I/O statement on them. This order, as stated previously, is not predictable. Therefore, the result of an application could be incorrect if it assumes records are sequentially related and cannot be arbitrarily written out or read in. For example, if the following loop is parallelized, the numbers printed out will no longer be in the sequential order from 1 to 500 as the result of a serial execution:

     do i = 1, 500
       print *, i
     enddo

Applications that depend on numbers being strictly in the specified order will not work correctly.

The XL Fortran runtime option multconn=yes allows connection of the same file to more than one logical unit simultaneously. Since such connections can only be made for reading (ACCESS='READ'), access from multiple threads to logical units that are connected to the same file will produce predictable results.

Use of I/O statements in signal handlers

There are basically two kinds of signals in the POSIX signal model: synchronously and asynchronously generated signals. Signals caused by the execution of some code of a thread, such as a reference to an unmapped, protected, or bad memory (SIGSEGV or SIGBUS), floating-point exception (SIGFPE), execution of a trap instruction (SIGTRAP), or execution of illegal instructions (SIGILL) are said to be synchronously generated. Signals may also be generated by events outside the process: for example, SIGINT, SIGHUP, SIGQUIT, SIGIO, and so on. Such events are referred to as interrupts. Signals that are generated by interrupts are said to be asynchronously generated.

The XL Fortran runtime is asynchronous signal unsafe. This means that an XL Fortran I/O statement cannot be used in a signal handler that is entered because of an asynchronously generated signal. The behavior of the system is undefined when an XL Fortran I/O statement is called from a signal handler that interrupts an I/O statement. However, it is safe to use I/O statements in signal handlers for synchronous signals.

Sometimes an application can guarantee that a signal handler is not entered asynchronously. For example, an application might mask signals except when it runs certain known sections of code. In such situations, the signal will not interrupt any I/O statements and other asynchronous signal unsafe functions. Therefore, you can still use Fortran I/O statements in an asynchronous signal handler.

A much easier and safer way to handle asynchronous signals is to block signals in all threads and to explicitly wait (using sigwait()) for them in one or more separate threads. The advantage of this approach is that the handler thread can use Fortran I/O statements as well as other asynchronous signal unsafe routines.

Asynchronous thread cancellation

When a thread enables asynchronous thread cancellability, any cancellation request is acted upon immediately. The XL Fortran runtime is not asynchronous thread cancellation safe. The behavior of the system is undefined if a thread is cancelled asynchronously while it is in the XL Fortran runtime.