RPG and large message data
The RPG language is at the heart of IBM i. There can hardly be any IBM i application developer who has never used it for building their applications. Accessing data through data description specifications (DDS) file, such as a physical file, is an efficient way to get data easily from a programming point of view. This is a concept that is well considered for developing business applications quickly.
For a long time, user data that RPG has handled were not long because of the DDS I/O size limitation (for example, the length of a physical file record must be 32766 bytes or smaller).
In V5R4 RPG, the maximum message length for a fixed-length character field is 65535, but beyond V6R1, the size changed from 65535 to 16773104. Because the I/O size is smaller than RPG limitation, we need not care about the character length issue in our programming cycle.
On the other hand, when we look into our business transactions today, there are many long message lengths data, such as XML. We see business data over 64 KB message length, being exchanged and processed. In the real world, RPG does not ignore or isolate this game change. Even though today's RPG has already extended their capacity for large data, there are much larger data existing on the customer's production environment. For example, the IBM WebSphere MQ product has been used for exchanging information between systems. RPG can also handle these data through MQ common APIs, such as MQGET. You can increase the size of the
MaxMsgLength attribute up to 100 MB on all WebSphere MQ systems at V6 or later.
How can we handle these large message length data within RPG? The most common way is to use application segmentation. Figure 1 shows how to do that. In this article, we will discuss pointer in ILE RPG. Then, we will talk about pointer operation, overlaying data with a pointer-based field, and finally the data segmentation method with ILE RPG.
Figure 1. Data segmentation overview
Using the application segmentation for large message length data
This section explains segmentation and the concept of pointer that is used for dividing data easily.
The definition for the word segmentation itself, is the act of dividing something into smaller parts, or the state of being divided. In this article, segmentation means the program logic of dividing large message length data into chunks of smaller message length data so that they can be handled by a program variable. We will divide large data into 64 KB data chunks and process with a normal RPG code logic. To do this, we use the pointer operation. What is a pointer? Which operation code should we specify in RPG and how do we write a code using it?
The pointer in ILE RPG
The concept of pointer is not a state-of-the-art technology. Many computer languages (such as C) have the same concept for handling data efficiently. Generally, a pointer is a useful piece of advice or information that helps you do or understand something. In computer terminology, a pointer is a field that contains a memory address.
Multiple occurrence data structure is a good sample of pointer. The occurrence of the database is based on an offset of the address in the data structure.
Let's take a look at the pointers in ILE RPG.
First, you define pointer data types '*' in D spec. You also need to define fields, arrays or data structures whose positioning in memory is based on the value of the pointer with the Based keyword. See Figure 2.
Figure 2. Pointer definition in D spec
With this definition, we can map the data to the location that the pointer variable addresses. This means that any message length of data can be divided into smaller chunks. Figure 3 illustrates the concept of pointer address operation.
Figure 3. Moving the pointer address
A pointer can show any address of the data. A field that is based on this pointer can refer or overlay the data. We can change the value of a pointer by adding or subtracting that value. With these operations, you will see overlaying data.
Now, we understand what a pointer is in ILE RPG programming. Next, we must understand how to get the pointer value within an ILE RPG code.
Built-in function %ADDR
In ILE RPG, the %ADDR built-in function is used for getting the address of a pointer. From an ILE programming model point of view, variable fields are referenced for processing data. So a variable's pointer value is needed when coding a pointer operation. Figure 4 shows you how to use this built-in-function.
Figure 4. Usage of the built-in function, %ADDR
In this code snippet, variable
p is defined as a type pointer. The data structure,
TestDS, is also declared. It contains two text fields. The first one is initialized as static text '
1234' and second one as '
5678'. The variable field,
p, is initialized as the result of
%Addr(Text1). This means that the field
p retrieves the pointer value of the
Text1 field, which is '
1234'. This is because the variable field is based on the pointer variable
p. Next, the
Eval operation '
p= %Addr(Text2)' changes the values of pointer
p. Now the value of pointer
p is equal to the value of the
Text2 pointer which is '
So far we have gone through ILE RPG pointer concept, definition procedure itself and its operand in ILE RPG codes. To put these pieces together more clearly, I will give you a practical sample program code using the MQ API, MQGET, that can handle pointers.
Sample code MQGET with pointer
The MQGET function call is used for retrieving data from the message queue. From an ILE RPG point of view, MQGET syntax is as follows.
(HCONN, HOBJ, MSGDSC, GMO, BUFLEN, BUFFER, DATLEN,
And, its prototype definition for the call is:
C*..1....:....2....:....3....:....4....:....5....:....6....:....7.. C CALLP MQGET(HCONN : HOBJ : MSGDSC : MO : C BUFLEN : BUFFER : DATLEN : C CMPCOD : REASON) D*..1....:....2....:....3....:....4....:....5....:....6....:....7.. DMQGET PR EXTPROC('MQGET') D* Connection handle D HCONN 10I 0 VALUE D* Object handle D HOBJ 10I 0 VALUE D* Message descriptor D MSGDSC 364A D* Options that control the action of MQGET D GMO 112A D* Length in bytes of the BUFFER area D BUFLEN 10I 0 VALUE D* Area to contain the message data D BUFFER * VALUE D* Length of the message D DATLEN 10I 0 D* Completion code D CMPCOD 10I 0 D* Reason code qualifying CMPCOD D REASON 10I 0
You can see a parameter BUFFER that contains the whole message defined as type pointer. Also, the IBM i MQ product provides a sample code for the ILE language. AMQ3GET4 is a sample code for the MQGET function call with ILE RPG. This code is a good sample for understanding the ILE RPG pointer operation.
In the original code, the BUFFER variable is defined as a 60-byte message length field as follows.
D BUFFER S 60
In this article, we extend that message length from 60 bytes to 32752 bytes.
D BUFFER S 32752
And, we add the following definitions to operate pointer values and addresses.
* Buffer pointer D BUFPTR S * INZ (%ADDR(BUFFER)) * *****ADD FOR TEST***** D BUFF S 32752A BASED(P_BUFF) D ARRAY S 32752A DIM(32) D X S 10I 0 D Y S 10I 0 *****ADD FOR TEST*****
In the above code, the pointer variable
BUFPTR is declared and initialized with a message length of 32752 byte. And, the variable field,
ARRAY is defined for dividing into small bytes within the ILE RPG logic.
Here is an initialization logic for allocated pointer area.
***ADD FOR TEST*** /free buflen = 32752* 32; bufptr = %alloc(buflen); for x = 0 to 31; p_buff = bufptr + (x * %size(buff)); buff = *blanks; endfor; /end-free ***ADD FOR TEST***
The variable field
buflen is 32752 bytes multiplied by 32, which calculates to a message length of 1048064 bytes.
%alloc is used for allocating memory area. Thus, this code allocates 5244288 bytes long memory area and returns its pointer to the
bufptr variable field. The
%alloc function does not initialize its allocated area. So, the user should take care of its initialization (the following line shows how to do that). Each field, with a message length of 32752 bytes, is initialized as blank, and therefore, the pointer moves to the next 32752 bytes, initializing again.
By 32 times of loop processing, the whole allocated memory area is initialized as *BLANK.
The following code provides an example for retrieving data from the received message queue.
***ADD FOR TEST*** /free for x = 0 to 31; p_buff = bufptr + (x * %size(buff)); y = x + 1; ARRAY(y) = buff; endfor; /end-free ***ADD FOR TEST***
As stated above, the MQGET function returns the message's pointer value. In our sample code, it is represented by the variable field,
BUFFER. So, with the above code, the message is retrieved in chunks with 32752 bytes each. And then each message stores into the ARRAY field. With this ILE RPG code, we can handle messages of over 1048064 bytes long. Here is the entire code for our customization for the MQ sample AMQ3GET4.
This article described ILE RPG capability for large size data by using pointers.