Migrating InfoSphere Streams SPADE applications to Streams Processing Language, Part 1: Migrate basic SPADE applications

An introductory guide by example

The most significant new feature of Version 2.0 of the IBM InfoSphere® Streams product is the programming language model transformation from Streams Processing Application Declarative Engine (SPADE) to Streams Processing Language (SPL). Users with SPADE applications from previous versions will need to migrate and port their applications to SPL when upgrading their installations to Version 2.0. This tutorial is Part 1 of a 5-part series that uses actual SPADE samples to demonstrate a series of step-by-step procedures for migrating and porting different types of SPADE application content. Part 1 demonstrates the migration of basic SPADE applications.

Share:

Kevin Erickson (kjerick@us.ibm.com), Senior Software Engineer, I.B.M.

Photo of Kevin EricksonKevin Erickson joined IBM in Rochester, MN, in 1981 as an electrical engineer working on development and test of hard disk drives. He has contributed to a wide range of IBM products and technologies, from disk drive actuator and spindle servo control systems, to AS/400 and successor operating system kernel components. More recently Kevin has worked on performance testing of the Roadrunner supercomputer, the first computer to break the 1 petaflop barrier. In 2009, he transferred to Software Group to work with the InfoSphere Streams team, primarily in a variety of test-related roles. Over his IBM career, Kevin has been issued 14 U.S. patents, plus several in other countries. He holds a B.S. degree in Electrical Engineering from the University of Minnesota.



Richard P. King (rpk@us.ibm.com), Senior Programmer, IBM

Richarc King's photoRichard King has been a part of the InfoSphere Streams team for a number of years. When he joined IBM in 1977, it was as an associate programmer working on the development of System/38. Since transferring to IBM's Research Division at IBM Thomas J. Watson Research Center in Yorktown in 1981, he has worked on a wide variety of projects, including the design and development of what became the IBM Sysplex Coupling Facility. He has a bacholor's degree in industrial engineering and operations research from Cornell University and a master's in operations research and industrial engineering from Northwestern University.



19 May 2011

Before you start

Introduction

More in the series

Look for more in the Migrating InfoSphere Streams SPADE applications to SPL series:

Each part is self-contained and does not rely on the previous example. Therefore, you can learn how to migrate these types of SPADE applications in any order. However, the examples are arranged in what might be considered easiest to most difficult, so completing each part in order might be beneficial.

The "Before you start" section is repeated in each tutorial so that you can complete each part of the series independently.

IBM InfoSphere Streams is a high-performance computing system designed for streaming applications that span many different kinds of business, scientific, engineering, and environmental disciplines. InfoSphere Streams is a highly scalable and powerful analytics platform that can be used to rapidly analyze and solve many real-time problems, in real time, using a wide variety of data streams from thousands of live data sources, as well as reference data from data warehouses. InfoSphere Streams provides an application development and execution environment that is tailored to users developing specific applications that ingest, filter, analyze, correlate, classify, transform, and otherwise process large volumes of continuous data streams. These powerful applications result in faster and more intelligent decision-making outcomes within diverse business sectors such as healthcare, transportation, financial, manufacturing, communications, energy, and security.

Version 2.0 of the IBM InfoSphere Streams product introduces a variety of new features and enhancements to the previous 1.x versions, the most significant of which is the programming language model transformation from Streams Processing Application Declarative Engine (SPADE) to Streams Processing Language (SPL). The primary reasons for this change are to make the language simpler, more intuitive, and more uniform. The most visible change with SPL is the syntax, which has a structure and feel that is more in the realm of C or Java™, thus making it easier for programmers to read, understand, and be more productive when writing SPL applications.

Because the programming language model was transformed from SPADE to SPL in InfoSphere Streams Version 2.0, users with SPADE applications from the previous versions need to migrate and port their applications to SPL when upgrading their installation. Initial information to help users do this is made available as part of the shipped product, which is provided primarily in the IBM Streams Processing Language SPADE Application Migration and Interoperability Guide. This guide is located in the doc subdirectory below the directory where InfoSphere Streams is installed or in the IBM InfoSphere Streams Information Center (see Resources).

The IBM Streams Processing Language SPADE Application Migration and Interoperability Guide contains basic information and instructions for migrating SPADE applications, but the information is presented at a fairly high level without specific porting examples. This tutorial complements the high-level guide by providing examples of SPADE-to-SPL application migration. You will use a SPADE sample application that is shipped as part of a SPADE installation as the basis for the migration example in this tutorial. However, it is recommended that you understand the overall migration procedures contained in the high-level guide before tackling the more detailed step-by-step procedures outlined in this tutorial.

Finally, you should have at least a fundamental knowledge of, and experience with, the SPADE programming language prior to proceeding with this tutorial. You should also have access to a version of InfoSphere Streams prior to Version 2.0 (generally referred to as Version 1.2.x herein, because that is the version used to obtain and work with the sample examples) as well as a Version 2.0 installation in order to perform the exercises in this tutorial. You also need at least a basic understanding of SPL, which you can acquire by reading the IBM Streams Processing Language Introductory Tutorial and trying out the various examples given there. The SPL introductory tutorial, as well as several other documents helpful in learning about more advanced aspects of SPL, operator models, operator toolkits, and so on, can be found in the doc subdirectory below the directory where InfoSphere Streams is installed or in the IBM InfoSphere Streams Information Center (see Resources).

Setting up

Before proceeding with any of the migration and porting examples, there are some general prerequisites that must be satisfied. These prerequisites should be set up as follows:

  1. Ensure an InfoSphere Streams Version 1.2.x installation is available with the original (unaltered) sample code located under the installation directory within the samples subdirectory.
  2. Ensure an InfoSphere Streams Version 2.0 installation is available.
  3. Ensure availability of at least one terminal access point connection for each installation version with the appropriate environment set up to work with each version.
  4. Create a base working directory, located off your home directory, in which to place and work with the migration samples. The suggested name for this directory, which is used throughout this example, is migrationsamples. If not already created, issue the following commands to create this directory:
    cd ~
    mkdir migrationsamples

Understanding general concepts for the examples

There are a number of common items that pertain to each of the migration and porting examples discussed in this tutorial. Some of these items are environmental, while others are suggested methods and techniques for addressing particular steps of the porting processes.

Dual installation

Note that both InfoSphere Streams Version 1.2.x and Version 2.0 are used in the examples throughout this tutorial. When working with the various elements of SPADE applications, the use of Version 1.2.x is required, whereas when working with the various aspects of migration procedures and SPL applications, the use of Version 2.0 is required. The spade-to-spl command, which is the primary command used to translate SPADE code to SPL, is available in Version 2.0 installations. Thus, when you reach the point in the migration process where the spade-to-SPL command is issued, you will switch from using a SPADE installation to using an SPL installation.

Order of post-migration fix-ups and porting

One of the main steps of the migration process is the execution of the spade-to-spl translation command. After this translation, a list of errors, warnings, and informational messages could result. The items in this list can be addressed in any order, but it is generally easier to resolve the simpler issues first because they often don't require any explicit changes, although in some cases changes might be made based on user preference. You can continue to evaluate the issues and work on resolving them in order of increasing difficulty. This is the philosophy and methodology used in this tutorial, which means particular translation issues might be presented out of numerical order from that reported by the translation process.

Compilation choices for SPADE and SPL applications

Throughout this tutorial, the approach for the compilation of applications is different for SPADE and SPL. For both types of applications, either the distributed or standalone methods of compilation can be used. The approaches used in this tutorial were selected to reduce the overall number of steps and commands that would have to be executed to demonstrate certain aspects of the migration processes.

For the SPADE application samples, using the makefile process to compile and generate easy-to-use execution scripts makes running and validating the samples relatively straightforward, even though they use the distributed application environment. For the SPL applications being migrated from the SPADE application samples (and in general when initially developing any new SPL applications), it is usually quicker and easier to initially employ the standalone application method of compilation. The distributed method is best for final production tuning when the application workload needs to be distributed across multiple cluster nodes to produce the best performance characteristics. Therefore, to keep the content of this tutorial as simple as possible, the standalone approach is used for SPL applications.


Configuring and testing the basic SPADE application

This tutorial describes the process of migrating and porting the most basic type of SPADE application, which is one that consists of only a main application file (<name>.dps) without any mixed-mode Perl, user-defined function or user-defined operator content. For this tutorial, the SPADE Volume-Weighted Average Price (VWAP) application (which is provided as a sample application shipped with Version 1.2.x of InfoSphere Streams) is used to demonstrate the step-by-step migration and porting procedures.

The initial step of the migration process is to ensure that the SPADE application being migrated is working correctly. If the SPADE application doesn't even compile or produce the correct results, a migration to SPL would likely fail during the translation. Or the resulting SPL would likely fail to compile or produce the correct results. The following procedure outlines the steps used to ensure that the SPADE application is suitably prepared for the migration and porting process.

  1. Change directories to the base working directory.
    cd ~/migrationsamples
  2. Create a copy of the VWAP sample code subdirectory from the InfoSphere Streams Version 1.2.x installation into a vwap subdirectory under the current working directory, where <V1.2.x_Installation> indicates the base file system location path name for the installation.
    cp -r ~/<V1.2.x_Installation>/samples/apps/vwap .
  3. Using your favorite text editor (called my_editor in the example), edit the vwap.dps file to change the sink operator code to generate output results to files instead of just producing performance metrics.
    1. Enter
      cd vwap
      my_editor vwap.dps &
    2. Change Nil := Sink(BargainIndex%@i)["perfcounter:///"]{} to
      Nil := Sink(BargainIndex%@i)["file:///tuples%@i.dat", csvFormat, noDelays]{}
    3. Save and exit.
  4. Compile the application using the make command.
  5. Run the compiler-generated script start_streams_vwap.sh to make and start the default instance named spade@your_userid.
    ./start_streams_vwap.sh
  6. Run the compiler-generated script submitjob_vwap.sh to submit the vwap job for execution.
    ./submitjob_vwap.sh
  7. Monitor the output files (which by default are written to the data subdirectory) to verify that the application is writing data to the files by checking that their sizes are greater than 0 bytes and potentially growing as you continue to look at their sizes using the following command. Note that the makefile for this application specifies three command-line arguments (EMC, IBM, and HPQ). Also note in the application program that each of these arguments is used to generate a separate stream processing flow section with results sent to different output files. Hence, there will be three output files produced in the data directory.
    ls -l data/tuples*.dat
  8. Optionally, you can see the actual contents of the output files.
    cat data/tuples*.dat  [or]
    my_editor data/tuples*.dat &
  9. Run the compiler-generated script canceljob_vwap.sh to cancel and end the VWAP job.
    ./canceljob_vwap.sh
  10. Run the compiler-generated script stop_streams_vwap.sh to stop and remove the default instance named spade@your_userid.
    ./stop_streams_vwap.sh

Translating the basic SPADE application to SPL

The next step of the migration is to set up for and execute the translation script. Perform the following steps by using a Version 2.0 installation.

  1. Create an SPL working directory, located off the VWAP application subdirectory, in which to place and work with the migrated code. The name chosen for the directory in this tutorial is myspl.
    mkdir myspl
  2. Convert the SPADE source to SPL using the spade-to-spl translation command. Note that the original command-line arguments (EMC, IBM, and HPQ) that were passed to the SPADE compiler in the makefile for the SPADE application are also passed as arguments to the SPL translation command.
    spade-to-spl -t myspl -f vwap.dps EMC IBM HPQ

After the translation, a list of errors, warnings, and informational messages similar to the following could result:

Listing 1. Informational message
vwap.dps:53:5: CDISP0688I Identifier 'timestamp' was 
renamed to 'timestamp0' due to a conflict with an SPL or 
C++ keyword, or a type defined by a typedef in the SPADE 
program.
Listing 2. Informational message
vwap.dps:53:5: CDISP0688I Identifier 'timestamp' was 
renamed to 'timestamp0' due to a conflict with an SPL or 
C++ keyword, or a type defined by a typedef in the SPADE 
program.
Listing 3. Informational message
vwap.dps:53:5: CDISP0688I Identifier 'timestamp' was 
renamed to 'timestamp0' due to a conflict with an SPL or 
C++ keyword, or a type defined by a typedef in the SPADE 
program.
Listing 4. Informational message
vwap.dps:45:8: CDISP0647I A 'Functor' operator will be 
inserted after this 'Source' either because the 'Source' 
has multiple output ports, or uses column selectors.
Listing 5. Warning message
vwap.dps:45:8: CDISP0703W WARNING: SPL does not support 
column selectors.  Ensure that the source's output schema 
accounts for all attributes in the input data, otherwise a 
runtime exception may occur.
Listing 6. Informational message
vwap.dps:29:8: CDISP0452I The output stream 'TradeQuote' 
was renamed to 'TradeQuote_' due to the insertion of a 
'Functor' operator.
Listing 7. Informational message
vwap.dps:61:13: CDISP0480I The 'timeStampToMicroseconds' 
function is not provided by the SPL run-time library.  An 
equivalent function has been synthesized.
Listing 8. Informational message
vwap.dps:96:13: CDISP0676I The 'select' function has been 
translated to the '? :' operator.
Listing 9. Warning message
vwap.dps:99:10: CDISP0724W WARNING: Parameter 
'writePunctuations' with a value of 'true' has been 
specified in this sink because 'dropPunct' was not 
specified in the corresponding SPADE code.  This may not be 
semantically equivalent if the SPADE code was compiled with 
'-U' or '--preserve-punctuations'.  You may wish to remove 
the 'writePunctuations' parameter or give it a value of 
'false'.
Listing 10. Informational message
vwap.dps:61:13: CDISP0480I The 'timeStampToMicroseconds' 
function is not provided by the SPL run-time library.  An 
equivalent function has been synthesized.
Listing 11. Informational message
vwap.dps:96:13: CDISP0676I The 'select' function has been 
translated to the '? :' operator.
Listing 12. Warning message
vwap.dps:99:10: CDISP0724W WARNING: Parameter 
'writePunctuations' with a value of 'true' has been 
specified in this sink because 'dropPunct' was not 
specified in the corresponding SPADE code.  This may not be 
semantically equivalent if the SPADE code was compiled with 
'-U' or '--preserve-punctuations'.  You may wish to remove 
the 'writePunctuations' parameter or give it a value of 
'false'.
Listing 13. Informational message
vwap.dps:61:13: CDISP0480I The 'timeStampToMicroseconds' 
function is not provided by the SPL run-time library.  An 
equivalent function has been synthesized.
Listing 14. Informational message
vwap.dps:96:13: CDISP0676I The 'select' function has been 
translated to the '? :' operator.
Listing 15. Warning message
vwap.dps:99:10: CDISP0724W WARNING: Parameter 
'writePunctuations' with a value of 'true' has been 
specified in this sink because 'dropPunct' was not specified 
in the corresponding SPADE code.  This may not be 
semantically equivalent if the SPADE code was compiled with 
'-U' or '--preserve-punctuations'.  You may wish to remove 
the 'writePunctuations' parameter or give it a value of 
'false'.

Each of these items must be addressed and evaluated to determine the course of action required to produce a comparable working application in SPL. The result of addressing these items might be a number of porting activities, such as changes to the migrated vwap.spl file and possibly other generated SPL files. There might also be a need for other implementation additions and changes to build up the required environment for an SPL application.


Migrating the basic SPADE application to SPL

The next step of the migration process is the post-translation fix-ups and porting of the migrated code and environment. As shown in the previous section, a number of errors, warnings, and informational messages could occur during migration. Addressing and evaluating these items, and fixing them where necessary, is the first major step in the porting process. Once these initial fix-ups are done, any other porting that was not automatically handled or identified as needed by the spade-to-spl translator will need to be performed.

The following steps describe how to perform the initial fix-up by addressing the messages from easiest to most difficult.

  1. Change directories to the SPL working directory.
    cd myspl
  2. Evaluate the identical informational messages in Listing 1, Listing 2, and Listing 3 from the list of migration messages generated by the spade-to-spl translation command. Note that the original SPADE application passed to the translator the EMC, IBM, and HPQ command-line arguments, each of which generated a separate, identical message.

    The IBM Streams Processing Language SPADE Application Migration and Interoperability Guide indicates that 'timestamp' is a reserved word in SPL. Therefore, to avoid compilation or other issues, the variable name was changed to 'timestamp0' because it is a similar name that does not conflict with a reserved word or variable name already used by the application. This change is benign, and you do not need to make any other changes because of it. However, if the choice for the variable name change is not acceptable, you can change it to something you prefer.

  3. To optionally change 'timestamp0' to 'someothername,' edit the migrated SPL application source and then save and exit the file.
    my_editor vwap.spl &
    Global change ‘timestamp0’ to ‘someothername’
  4. Evaluate the informational message in Listing 6.

    Because the translator found it necessary to break up the SPADE source operator into two separate operators in SPL (a FileSource and Functor combination), it had to create a new name for the intermediate stream to avoid name conflicts. Thus the intermediate stream name was given the original stream name with an underscore suffix. This addition is benign, and you do not need to make any other changes because of it. However, if the choice for the new stream name is not acceptable, you can change it to something you prefer.

  5. To optionally change 'TradeQuote_' to 'someothername,' edit the migrated SPL application source and then save and exit the file.
    my_editor vwap.spl &
    Global change ‘TradeQuote_’ to ‘someothername’
  6. Evaluate the identical informational messages Listing 7, Listing 10, and Listing 13. Note that the original SPADE application passed to the translator the EMC, IBM, and HPQ command-line arguments, each of which generated a separate, identical message.

    In SPADE, the 'timeStampToMicroseconds()' function was provided as a built-in function by the runtime library. This is not the case in SPL. Therefore, the spade-to-spl translator attempts to provide equivalent code by synthesizing functions within the SPL application to replace the nonexistent library function. In most cases the provided code is sufficient, and you do not need to make any other changes because of it. However, if the synthesized code is not acceptable, you can change it to something you prefer. This can be done by editing the migrated SPL application source.

  7. To optionally change synthesized function and callers, edit the migrated SPL application source and then save and exit the file.
    my_editor vwap.spl &
    Change synthesized function and callers
  8. Evaluate the identical informational messages Listing 8, Listing 11, and Listing 14. Note that the original SPADE application passed to the translator the EMC, IBM, and HPQ command-line arguments, each of which generated a separate, identical message.

    Unlike SPADE, the syntax and language structure in SPL is geared more toward the C language style. Therefore, the conditional operator in SPL is used to replace the 'select()' function in SPADE. This change is benign, and you do not need to make any other changes because of it.

  9. Evaluate the identical informational messages Listing 9, Listing 12, and Listing 15. Note that the original SPADE application passed to the translator the EMC, IBM, and HPQ command-line arguments, each of which generated a separate, identical message.

    In this case the translator chose to include the line ‘writePunctuations : true;’ as part of the parameter list of the FileSink operator because the original SPADE application did not specifically indicate that punctuation should be dropped. The original SPADE application did not include any punctuation in the resulting output file. The choice to include punctuation or not depends on how the output file will be parsed and used. For this tutorial, in order to more closely match the original SPADE application behavior, this punctuation line should be deleted or commented out in each of the places it occurs in the migrated SPL application source.

  10. To delete the punctuation line, edit the migrated SPL application source and then save and exit the file.
    my_editor vwap.spl &
    Delete the ‘writePunctuations : true;’ lines, or comment each
    out ‘// writePunctuations : true;’
  11. Evaluate the informational message Listing 4 and the related warning message, Listing 5.

    Because column selectors are not supported in SPL, the translator modifies the operator graph by adding a Functor after the FileSource to act as an interface that ingests all of the source data columns, but then filters by selecting and passing only the columns of interest, just as the original source did. However, when using the comma separated value (csv) format, the FileSource is expected to extract all the columns in each line precisely as formatted and to present each of these columns as an attribute in the output tuple.

    The translator makes a valiant effort to provide migration transformations that are as functionally equivalent as possible to the original application. For this example, all the translator knows (from the source operator in the original vwap.dps application) is that there are at least 16 columns in the csv source data, but it doesn't know if there are more than 16. For the columns between 1 and 16 that are not selected, the translator does well to provide dummy attributes to fill in the gaps. However, because the VWAP input data format provided in the source file actually has 30 columns, the output tuple for the FileSource operator generated by the translator does not match the input data format, which throws a runtime error. Therefore, you must edit the migrated SPL application source to fix this.

  12. Before you can fix the migrated SPL application source, determine what the data attributes should be for the missing columns. Because the input data is in a zipped format, the file must first be unzipped before it can be edited. The first line in the file is a comment that provides titles for each of the columns, from which you can surmise what the remaining column attributes might be. To examine the input data file:
    cd ~/migrationsamples/vwap/data
    gunzip -c TradesAndQuotes.csv.gz > temp.csv
    my_editor temp.csv &
  13. By inspection and intuitive mapping, you can determine the remaining tuple attributes missed in the migration to be:
     <… int32 numsellers, rstring qualifiers, int32 seqno,
     rstring exchtime, decimal64 blockTrd, decimal64 floorTrd,
     decimal64 PEratio, decimal64 yield, decimal64 newprice,
     decimal64 newvol, int32 newseqno, decimal64 bidimpvol,
     decimal64 askimpcol, decimal64 impvol>
  14. Exit and remove the temporary file.
    rm temp.csv
  15. Fix the migrated SPL application source so the FileSource operator has the correct output tuple format by using the following commands:
    cd ~/migrationsamples/vwap/myspl
    my_editor vwap.spl &
  16. Change the first line of the FileSource stream to be:
    stream<rstring ticker, rstring date, rstring time, rstring
    dummyAttrib4_, rstring ttype, rstring dummyAttrib6_, float64
    price, float64 volume, float64 vwap, rstring dummyAttrib10_,
    float64 bidprice, rstring dummyAttrib12_, rstring
    dummyAttrib13_, rstring dummyAttrib14_, float64 askprice,
    float64 asksize, int32 numsellers, rstring qualifiers, int32
    seqno, rstring exchtime, decimal64 blockTrd, decimal64
    floorTrd, decimal64 PEratio, decimal64 yield, decimal64
    newprice, decimal64 newvol, int32 newseqno, decimal64
    bidimpvol, decimal64 askimpcol, decimal64 impvol>
    TradeQuote_ = FileSource() {

Finishing and testing the basic SPL application

The final step of the migration process is the compilation and testing of the completed SPADE-to-SPL migration and porting solution. The following procedure outlines the steps to do this.

  1. Compile the SPL application. This compilation should succeed, because all of the migration issues were adequately addressed through fix-ups and porting.
    sc -T -M vwap

    Note that the compilation process automatically creates a data subdirectory for the SPL application under the directory where the migrated SPL source was placed.

  2. Copy the original compressed source input file TradesAndQuotes.csv.gz from the SPADE data subdirectory into the SPL data subdirectory.
    cp ~/migrationsamples/vwap/data/TradesAndQuotes.csv.gz data/
  3. Execute the standalone application.
    ./output/bin/standalone -d 0
  4. Check the contents of the data subdirectory to verify that the application generated the expected output files, and that their sizes are not 0.
    ls -l data/tuples*.dat
  5. Optionally, further inspect the contents of the files to verify that the application generated the expected output data. This should be similar to that of the original SPADE application, although possibly not identical, because the flow of the original SPADE application is nondeterministic from run to run, and it produces different output each time.
    my_editor data/tuples*.dat &

Going above and beyond

The procedures outlined above should result in a successful migration and generation of comparable output results. To better understand the capabilities and limitations of the migration translator, optionally go back to Migrating the basic SPADE application to SPL, Steps 9 through 16 and remove the fix-ups and porting changes from the migrated SPL application file to see what would happen if these changes were not made. Complete the following steps.

  1. Save a copy of the good working SPL application code for later restoration.
    cd ~/migrationsamples/vwap/myspl
    cp vwap.spl vwap.spl.save
  2. Undo the fix-ups and porting that you did in Migrating the basic SPADE application to SPL, Steps 9 through 16. The best way to do this is to re-run the translator.
    cd ..
    spade-to-spl -t myspl -f vwap.dps EMC IBM HPQ
    cd myspl
  3. Compile the SPL application. Although the FileSource is actually missing attributes in its output tuple compared to the format of the input csv data file, the compiler does not know this. Thus, the compilation should succeed.
    sc -T -M vwap
  4. Try to execute the standalone application.
    ./output/bin/standalone -d 0

    It should fail with an error that includes Received exception during execution of operator 'TradeQuote_'.
  5. Restore the SPL application source so the FileSource operator has the correct output tuple format. Use the procedure from Migrating the basic SPADE application to SPL, Steps 11 through 16.
  6. Try again to compile and execute the standalone application. This time it should succeed and produce output data.
    sc -T -M vwap
    ./output/bin/standalone -d 0
  7. Inspect the contents of the tuples*.dat files to verify that the application generated output data.
    my_editor data/tuples*.dat &

    Notice that this time it has punctuation markers for each element of data that is output to the files.
  8. Restore the SPL application source so that the punctuation is once again removed. Use the procedure from Migrating the basic SPADE application to SPL, Steps 9 and 10.
  9. Try again to compile and execute the standalone application. This time it should produce the correct output data.
    sc -T -M vwap
    ./output/bin/standalone -d 0
  10. Inspect the contents of the tuples*.dat files to verify that the application generated output data without the punctuation markers.
    my_editor data/tuples*.dat &
  11. When you are finished with the extra credit exercise, restore the good working SPL application code that you saved previously, although in theory the code resulting from this procedure should be that same code.
    cp vwap.spl.save vwap.spl

Conclusion

This tutorial presented migration strategies for a simple, pure SPADE program. You can continue your education with all 5 parts of this tutorial series. As you tackle progressively more-difficult parts of the series, your effort will go up from checking on a few minor language differences (like for punctuation) to dealing with ancillary objects, such as function models and C++ implementation files for UDFs, UDOPs, and UBOPs. Parts 2-5 of this tutorial series demonstrate progressively more-complex migration procedures, which range from mixed-mode applications and procedures with user-defined functions to procedures that incorporate different kinds of user-defined operators (UDOPs and UBOPs).

Keep in mind the fact that these tutorial examples can only serve as a general guide for how to approach the conversion of a SPADE program to SPL. In other words, each SPADE program is different, and what is encountered during the conversion process for each can vary widely. For example, in the course of converting many other SPADE programs, you might see any of the following:

  • Failures to produce any SPL source
  • Source that doesn't compile
  • Compiled programs that don't run properly
  • Running programs that don't produce the correct results

Of significant importance is the availability of known good results from the SPADE version of the program. Once the SPL program is running, even apparently successfully, be sure to compare the former results with the new. Unfortunately, the new results will not always be good. A divide-and-conquer approach works fairly well in these cases. Both SPADE and SPL programs are easily modified through the addition of temporary file sinks. In both the SPADE and SPL versions of the program, add a new file sink to some operator in the middle of the program to record its output stream. The new file sink produces either good or bad results, which narrows the area of your search to either the part of the program before or after this new file sink. This process, repeated just a few times, usually helps you focus in on the erroneous computation fairly quickly.

None of this should frighten you away from the translation tool. It is, by far, the easiest and fastest way for someone to go from having a SPADE program to having a remarkably similar program that's now expressed in the SPL idiom. Just stay flexible and be prepared to be an active part of the migration process.

Resources

Learn

Get products and technologies

  • Build your next development project with IBM trial software, available for download directly from developerWorks.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Big data and analytics on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Big data and analytics, Information Management
ArticleID=659548
ArticleTitle=Migrating InfoSphere Streams SPADE applications to Streams Processing Language, Part 1: Migrate basic SPADE applications
publish-date=05192011