Using the IBM Spectrum LSF API Python wrapper

The Python wrapper for IBM® Spectrum LSF APIs allows users to call the LSF APIs from Python. You create a Python wrapper using the Simplified Wrapper and Interface Generator (SWIG), which is the tool used for interfacing the LSF C APIs and Python.

About this task

You can contribute your own Python wrappers to the open source LSF APIs for Python on GitHub: https://github.com/PlatformLSF/platform-python-lsf-api.

Mapping from a string array in the Python wrapper to char** in the C LSF API

About this task

In the Python wrapper for the LSF 9.1.2 APIs, the function typemap(in) char** that converts a Python string array to a C char** in C is removed because of a bug in this function when the input string arrays are not NULL terminated.

Procedure

Use another interface to handle the mapping from string arrays in Python to char** in C.

For example,

$ cat test1.py
from pythonlsf import lsf

strArr = lsf.new_stringArray(1);              # Create a new stringArray, length 1
lsf.stringArray_setitem(strArr, 0, "normal"); # Set its value
intp_num_queues = lsf.new_intp();             # Create an int ptr
lsf.intp_assign(intp_num_queues, 1);          # Set the int value, which is the queue number

if lsf.lsb_init("test") > 0:                  # LSB initialization
    exit(-1);

queueInfo = lsf.lsb_queueinfo(strArr,intp_num_queues,None,None,0); # Query queue info
if queueInfo == None:
    exit(0);

print ’queue name = %s’ % queueInfo.queue;
print ’queue description = %s’ % queueInfo.description;

$ python test1.py
queue name = normal
queue description = For normal low priority jobs, running only if hosts are lightly loaded.

Querying multiple entities returned from LSF APIs

About this task

LSF APIs support multiple entities to be queried and returned, and similar APIs in Python are needed the LSF Python wrapper, for example, to return multiple queueinfo entities from lsf.lsb queueinfo(). You need to generate the required Python interface in accordance with LSF C APIs and SWIG. The following example uses the LSF C API lsb queueinfo() to demonstrate how to extend the Python LSF wrapper.

Procedure

Add the following code into the SWIG interface file <python-wrapper-install-dir>/pythonlsf/lsf.i.

PyObject * get_queue_info_by_name(char** name, int num) {
    struct queueInfoEnt* queueinfo;
    int numqueues = num;
    int options = 0;

    queueinfo = lsb_queueinfo(name,               // Return queries as C queueInfoEnt*
                  &numqueues, NULL, 0, options);  

    PyObject *result = PyList_New(numqueues);     // Create PyObject * to get C returns
    int i;
    for (i = 0; i < numqueues; i++) {             // Save queries in a loop to result
        PyObject *o = SWIG_NewPointerObj(SWIG_as_voidptr(&queueinfo[i]),
        SWIGTYPE_p_queueInfoEnt, 0 | 0 );
        PyList_SetItem(result,i,o);
    }

    return result;
}

In <python-wrapper-install-dir>/, rebuild the Python interface and reinstall to enable the newly added extension.
```
$ python setup.py build
$ sudo python setup.py install
```

Test the new Python API get_queue_info_by_name().

$ cat test2.py
from pythonlsf import lsf

strArr = lsf.new_stringArray(2)                             # Create a new stringArray, length 2
lsf.stringArray_setitem(strArr, 0, "normal")                # Set the 1st queue name
lsf.stringArray_setitem(strArr, 1, "public")                # Set the 2nd queue name

if lsf.lsb_init("test") > 0:                                # LSB initialization
  exit(-1);
                                       
for queueInfoEnt in lsf.get_queue_info_by_name(strArr, 2):  # Query the queue info in batch
  if queueInfo == None:
    exit(0);
  print 'queue name = %s' % queueInfoEnt.queue;
  print 'queue description = %s\n' % queueInfoEnt.description;

$ python test2.py
queue name = normal
queue description = For normal low priority jobs, running only if hosts are lightly loaded.

queue name = public
queue description = public queue

When you query a list of queues with one or more non-existent queue names, the Python API get queue info by name() returns the [None] value. You might expect the entities of existing queue names be returned, but this is the designed behavior. The following example shows how to test this:

$ cat test4.py
from pythonlsf import lsf

strArr = lsf.new_stringArray(2)
lsf.stringArray_setitem(strArr, 0, "normal")
lsf.stringArray_setitem(strArr, 1, "doesnotexist")

if lsf.lsb_init("test") > 0:
  exit(-1);

print lsf.get_queue_info_by_name(strArr, 2)

$ python test4.py
[None]

The low-level LSF C function lsb_queueinfo() returns NULL in case of any errors in user-specified arguments in queue, host, or user names. You can verify this with the LSF bqueues command, which internally calls lsb_queueinfo():

The following bqueues command shows normal output for two queues that exist:

$ bqueues normal public
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP
normal           30  Open:Active       -    -    -    -     0     0     0     0
public           35  Open:Active       -    -    -    -     0     0     0

If you specify a queue name that does not exist, an error message is displayed and no other output is shown:

$ bqueues normal doesnotexist
doesnotexist: No such queue

This is consistent with the Python API test.

Querying all entities returned from the LSF APIs

About this task

By specifying the requested resource, LSF APIs supports all entities of the specified resource to be returned, and the LSF Python wrapper needs similar APIs for Python, for example, to return all queueinfo entities from lsf.lsb queueinfo(). You need to generate the required Python interface in accordance with LSF C APIs and SWIG. The following example uses the LSF C API lsb queueinfo() to demonstrate how to extend the Python API.

Procedure

Add the following code into the SWIG interface file <python-wrapper-install-dir>/pythonlsf/lsf.i.

PyObject * get_queue_info_all() {
    struct queueInfoEnt *queueinfo;
    char *resreq;
    int numqueues = 0;
    int options = 0;

    resreq="";

    queueinfo = lsb_queueinfo(resreq,             // Return queries as C queueInfoEnt*
                  &numqueues, NULL, 0, options);

    PyObject *result = PyList_New(numqueues);     // Create PyObject * to get C returns
    int i;
    for (i = 0; i < numqueues; i++) {             // Save queries in a loop to result
        PyObject *o = SWIG_NewPointerObj(SWIG_as_voidptr(&queueinfo[i]),
                                         SWIGTYPE_p_queueInfoEnt, 0 | 0 );
        PyList_SetItem(result,i,o);
    }

    return result;
}

In <python-wrapper-install-dir>/, rebuild the Python interface and reinstall to enable the newly added extension.
```
$ python setup.py build
$ sudo python setup.py install
```

Test the new Python API get_queue_info_all().

$ cat test3.py
from pythonlsf import lsf

if lsf.lsb_init("test") > 0:                     # LSB initialization
  exit(-1);

for queueInfoEnt in lsf.get_queue_info_all():    # Query all queues info in batch
  if queueInfo == None:
    exit(0);
  print 'queue name = %s' % queueInfoEnt.queue;
  print 'queue description = %s\n' % queueInfoEnt.description;

$ python test3.py
queue name = owners
queue description = For owners of some machines. Only users listed in the HOSTSsection can submit jobs to this queue.

...        ...       ...

queue name = interactive
queue description = For interactive jobs only

queue name = idle
queue description = Running only if the machine is idle and very lightly loaded.