Level: Intermediate Scott Morris, Senior Software Engineer, IBM
03 May 2004 In the world of software, faster means better. This article describes ways you can improve the performance of the Lotus Connector for SAP.
The Lotus Connector for SAP allows access to SAP's Remote Function Calls (RFCs), Business Object Application Programmers Interfaces (BAPIs), and transactions via batch input mode. You can use the SAP Connector in scripted applications together with the Lotus Connectors LotusScript Extensions (LSX LC). More easily, you can use the SAP Connector with the Lotus Enterprise Integrator (LEI) and Domino Enterprise Connection Services (DECS).
This article describes ways you can improve the performance of the Lotus Connector for SAP when used with LEI, although many of the concepts that will be shown may also apply to DECS and/or the LSX LC. We assume that you're familiar with LEI and SAP.
SAP Connector architecture
The architecture of the SAP Connector is different from all other Lotus RDBMS connectors because the SAP Connector does not require the SAPGUI client to be locally available. Instead, the SAP Connector connects via the Remote Function Call Software Development Kit (RFCSDK). This is a single DLL on Windows 32; a single shared library on AIX, Sun Sparc, and Linux; and a single service program on AS400. When the SAP Connector asks SAP to execute an RFC, SAP does so and then delivers the entire result set to the SAP Connector.
This is very different from, for example, the DB2 or Oracle connectors. These RDBMS connectors require the corresponding RDBMS client software to be available locally, and it is this client that sends the SQL commands to the database and then fetches the result sets. There is a good amount of performance logic in this client/server interaction. Not all data is immediately returned to the client; it must be fetched. This means that at any one instance in time, the RDBMS result set occupies less space in memory because it is not the entire result set -- it is only what is currently fetched (and perhaps a read ahead number of records cached in the database client).
When you are dealing with a small number of records, this is not a big deal. But when result sets grow large, the SAP Connector needs more available system resources to hold the entire result set in memory. There is nothing that you can do about this -- it is a fact of the architecture of how SAP allows external applications to execute RFCs. There are a number of situations that arise in LEI that can double or even triple the amount of memory needed by the SAP Connector, and it is here that a bit of creativity can be of help.
Controlling sort order
For instance, the RFCs and BAPIs of SAP produce result sets that are almost never in a nice order. RFC_CUSTOMER_GET is a good example of this. This RFC returns a table CUSTOMER_T with names, customer numbers, and address data. Generally, either the customer name or the customer order is sorted in Domino. To see this in action, create a Direct Transfer in LEI to read from RFC_CUSTOMER_GET and write the data to a Domino database.
In our example database, all of the field names in the Domino form match the field names returned by the SAP Connector. To set up this example, we must define a connection to SAP, a connection to the Domino database, and a connection to the Direct Transfer activity.
SAP connection document
The SAP connection document contains all the information needed to connect to SAP. This includes the SAP application server name, the destination or instance identifier, the system number needed to connect, the SAP client number, the user name and password, and finally which language should be used to connect. You must also specify the RFC or BAPI that you will call, which in this case is RFC_CUSTOMER_GET. Use the pop-up button next to the word Table to select the metadata from RFC_CUSTOMER_GET that you want to work with.
The table CUSTOMER_T returns the desired metadata. After you select the table name, the fields of the table and their data types are displayed:
Figure 1. SAP connection document
Notes connection document
Next, define your Notes connection document to identify the Domino database with which you want to work. As Figure 2 shows, the only information that you need to specify for this example is the name of the Domino server where the database resides and the file name of the database:
Figure 2. Notes connection document
Direct Transfer activity
Finally, create the actual Direct Transfer activity. Here your SAP connection is the source and your Notes connection the target. You will read from CUSTOMER_T and write to a form named Customer in your Domino database. Field mapping is automatically defined by name because both sides have the same field names. The SAP select statement is KUNNR="*", where KUNNR is a German abbreviation for Customer Number and the asterisk (*) means you want them all:
Figure 3. Data Transfer activity
Limit the number of records allowed to be transferred to 10, which you can see in Figure 3 under Direct Transfer Options Performance Options in the field "Maximum Number of Records to Transfer."
Comparing results
Now you can run the activity and get the results. Open the Domino database and look at the customer numbers. Our example looks like this in which the customer numbers are 1, 2, 99, 110, 999, 1000, 1001, 1002, 1003, and 1004:
Figure 4. Customer data
Now delete all of the Domino documents. Edit your LEI activity and remove the restriction on the maximum number of records that are allowed to be transferred. Then re-run the LEI Direct Transfer activity. Open the Domino database and view the customer data. It is likely this information appears different from how it was displayed previously. For example, on an SAP 4.6C IDES system, our sample database shows the first 10 customer numbers as 1, 2, 3, 7, 51, 99, 110, 999, 1000, and 1001:
Figure 5. Re-sorted customer data
With an LEI Replication activity, this can cause problems. The sort order of the data from both connections used in the activity must be the same, or you will get excessive inserts and deletes.
Order metaconnector
The common LEI solution to this problem is to introduce the Order metaconnector. The Order metaconnector sorts the data you define and then presents it to the Replication activity in our case. The following is an example of this in which the Order metaconnector uses the SAP connection document and then sorts its result set in an ascending fashion based on the field CUSTOMER_TKUNNR from the SAP result set:
Figure 6. Order metaconnector
The Order metaconnector does this by fetching all the data from the underlying connector and sorting it. When that connector is the SAP Connector, it means you went from having a single result set in memory to having two complete copies of the result set in memory: the one that SAP delivered and the sorted result set being held by the Order metaconnector. If the SAP result set is large, this consumes more time and memory.
The only place to address this problem is inside SAP. Because SAP sits on top of an RDBMS, it can return the data sorted in any order. The recommended strategy here is to have your SAP ABAP staff create a wrapper RFC, which has the same interface as the original one. The only difference is that it would make certain that the data is sorted in the proper way to make your Replication activity function correctly without the need for the Order metaconnector. This reduces the amount of time - and memory - required to run your activity.
Working with RFC_READ_TABLE
There is a feature in the SAP Connector to make working with RFC_READ_TABLE easier. This RFC basically reads requested fields from tables that exist in the RDBMS under SAP. It takes these fields and turns them all to text. If you ask for five fields, those five fields are printed to a single string and placed in the DATA table (returned by RFC_READ_TABLE) as one column and one record.
Here is the ABAP header of RFC_READ_TABLE from SAP:
FUNCTION RFC_READ_TABLE.
*"----------------------------------------------------------------------
*"*"Lokale Schnittstelle:
*" IMPORTING
*" VALUE(QUERY_TABLE) LIKE DD02L-TABNAME
*" VALUE(DELIMITER) LIKE SONV-FLAG DEFAULT SPACE
*" VALUE(NO_DATA) LIKE SONV-FLAG DEFAULT SPACE
*" VALUE(ROWSKIPS) LIKE SOID-ACCNT DEFAULT 0
*" VALUE(ROWCOUNT) LIKE SOID-ACCNT DEFAULT 0
*" TABLES
*" OPTIONS STRUCTURE RFC_DB_OPT
*" FIELDS STRUCTURE RFC_DB_FLD
*" DATA STRUCTURE TAB512 - Data that we care about is returned here
*" EXCEPTIONS
*" TABLE_NOT_AVAILABLE
*" TABLE_WITHOUT_DATA
*" OPTION_NOT_VALID
*" FIELD_NOT_VALID
*" NOT_AUTHORIZED
*" DATA_BUFFER_EXCEEDED
*"----------------------------------------------------------------------
|
The SAP Connector offers a service to parse this long string and to convert the data in the string back to their original data types, which are the actual data types of the table in the RDBMS under SAP. The result set is the RDBMS table, not the table DATA filled with strings and returned from RFC_READ_TABLE. This parsing service is one of the most widely used features in the SAP Connector.
Figure 7 shows the metadata in your SAP connection document. In this example, table KNA1 (which is the RDBMS table under SAP that we really want to read) appears as part of the metadata of RFC_READ_TABLE, when in fact it really isn't, as the ABAP interface above shows:
Figure 7. Result set for RFC_READ_TABLE
One issue to consider in this example is that RFC_READ_TABLE returned table DATA holding all of the data in a bad format. (It is "bad" because it is all text in one long string, instead of in nice clean fields with the RDBMS column data types.) The SAP Connector processed this into a usable result set. Now you have the data twice in memory in the SAP Connector. The data conversion also takes time. When you have a lot of records to convert, this can add up.
Reducing memory usage
You now have the data in a format that you can use, ready to be delivered to the LEI Replication activity. Unfortunately, you find that the data is not sorted as you would like, and you are forced once again to introduce the Order metaconnector. By the time its work is done, you have the SAP result set stored three times in memory. You have converted all of the data from strings back to the correct data types, and it has been sorted.
The best way to address the memory usage and extra time needed for reading a table from under SAP is to get in touch with your SAP ABAP team again. Ask them to create a new RFC that simply returns the table that you want from the RDBMS under SAP, and don't forget to tell them how you want it sorted. This removes the need to use RFC_READ_TABLE, which eliminates its overhead, and also removes the need for the Order metaconnector. The result is a faster Replication activity that requires far less memory.
Final thoughts
You now have an understanding of the SAP Connector's requirements for memory. Even if everything is perfect (no Order metaconnector, no RFC_READ_TABLE), it still loads the entire result set into memory. You can definitely have result sets that require more RAM for your process than a 32-bit operating system can provide. The only way to deal with this situation is to once again contact your SAP ABAP team and request that they split up the result set returned by their RFC into two or more result sets. New RFCs (for example, RFC_WHAT_I_WANT_1 thru RFC_WHAT_I_WANT_3) would each return a part of the result set. In this case, you would now have three LEI activities in which each activity calls one of these new RFCs. Even if you are not running out of memory, this is worth considering. Functions such as LEI replication run much faster when the result sets to compare are smaller.
We hope these tips help you to improve the performance of your SAP activities running under LEI.
Resources
About the author  | | | Scott Morris is a Senior Software Engineer with IBM. He joined Lotus/IBM in 1990. Since 1997, he has worked on developing products that are offered by the Domino Enterprise Integration group. He was previously a member of the Notes API team, the Notes (before there was a Domino) server team, and somewhere in the deep dark past the 1-2-3 for Macintosh team. Scott has an MS in Computer Science from Boston University and a BS in Mathematics from Carnegie Mellon University. |
Rate this page
| 
