IBM SPSS Analytic Server 1.0.1 FP1 Release Notes

Content

Contents
Cloudera 4.6 Configuration.
BigInsights 2.1.2 Configuration.

BigInsights 2.1.2 (HDFS).
BigInsights 2.1.2 (GPFS).
BigInsights 2.1.2 with Platform Symphony (HDFS).
Analytic Server is installed on the BigInsights name node.
Analytic Server is installed elsewhere.
BigInsights 2.1.2 with Platform Symphony (GPFS).

HCatalog configuration.

HCatalog configuration for BigInsights 2.1.2.

Accumulo configuration.
Migrating settings from an existing Analytic Server installation.

New installation is on a different machine.

Improved ensemble modeling for very large datasets.
Reading empty/null strings in SPSS Modeler and Analytic Server.
Handling control characters in strings.

Cloudera 4.6 Configuration

The following JAR file should be copied from the Cloudera installation to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
commons-collections-3.2.1.jar

BigInsights 2.1.2 (all configurations)

There are some manual steps required to configure Analytic Server for use with BigInsights 2.1.2.

1. Run the Analytic Server 1.0.1 installer as the BigInsights administrator.  For GPFS, run the installer on one of the GPFS nodes.

a. At the step where you identify the Hadoop Namenode Server, enter the IP or name of the primary cluster system
b. The installer will fail to detect the Hadoop version, but this will correctly set most of the settings in config.properties.

2. Edit {AS_SERVER_ROOT}/configuration/config.properties

a. Change hdfs.namenode.url to point to your analytic root; for example, hdfs.namenode.url=hdfs://<ip>:<port>/user/biadmin/analytic-root for HDFS or hdfs.namenode.url=gpfs:///<ip>/user/biadmin/analytic-root for GPFS.
b. Make sure mapred.job.tracker points to the namenode job tracker. This can be found in the BigInsights admin console.

3. Follow the steps below for your particular Hadoop configuration
4. Install Analytic Server 1.0.1.1
5. Run {AS_ROOT}/bin/hdfsUpdate.sh.
6. Run {AS_ROOT}/bin/start.sh.

Note:

Analytic Server uses the 8080 port for HTTP by default; so does BigInsights. Follow these steps to change the port number for Analytic Server.

1. Stop the Analytic Server by executing the following command.
{AS_ROOT}/bin/stop.sh
2. Edit the following line of the {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/server.xml file to reference the ports you want to use.
<httpEndpoint host="*" id="ae_server" httpPort="8080" httpsPort="8443"/>
3. Restart the Analytic Server by executing the following command.
{AS_ROOT}/bin/start.sh

BigInsights 2.1.2 (HDFS)

Assuming you have followed the instructions to this point for all BigInsights configurations:

1. Copy the following JAR files from $BIGINSIGHTS_HOME/IHC/ to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
hadoop-core-2.2.0-mr1.jar
ibm-compression.jar
2. Copy the following JAR files to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
$BIGINSIGHTS_HOME/IHC/share/hadoop/common/lib/avro-1.7.4.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/commons-cli-1.2.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/common/lib/commons-configuration-1.6.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/commons-io-2.1.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/commons-lang-2.5.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/guava-11.0.2.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/jersey-core-1.9.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/jersey-server-1.9.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/common/lib/slf4j-api-1.7.5.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar
3. Copy the following JAR files from {AS_ROOT}\cloudera\cloudera4 to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
accumulo-hive-storage-handler_2.0-1.6.0.jar
hadoopmapreduce_1.1-1.0.1.0.jar
hcatalogstoragehandler_1.1-1.0.1.0.jar
4. Delete the following JAR files from {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
hadoopmapreduce-1.0.1.0.jar
accumulo-hive-storage-handler-1.6.0.jar
hcatalogstoragehandler-1.0.1.0.jar
commons-io-1.4.jar
5. Return to the instructions for all BigInsights configurations.

BigInsights 2.1.2 (GPFS)

Assuming you have followed the instructions to this point for all BigInsights configurations:

1. Add the following to log4j.xml
<!--biginsights related logs-->
<logger name="com.ibm.biginsights">
<level value="info" />
<appender-ref ref="as_trace_file" />
</logger>
2. Create a hadoop-conf directory in the {AS_SERVER_ROOT}/configuration directory and copy $BIGINSIGHTS_HOME/hadoop-conf/core-site.xml to it.
3. Copy the following JAR file from /usr/lpp/mmfs/fpo/hadoop-1.1.1/ to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
hadoop-1.1.1-gpfs.jar
4. Copy $BIGINSIGHTS_HOME/IHC/lib/native/Linux-amd64-64/libgpfs* to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/configuration/lib_64
5. Copy the following JAR files from $BIGINSIGHTS_HOME/IHC/lib/ to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
biginsights-gpfs-2.2.0.jar
ibm-compression.jar
6. Copy the following JAR files to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
$BIGINSIGHTS_HOME/IHC/share/hadoop/common/lib/avro-1.7.4.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/commons-cli-1.2.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/guava-*.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar
$BIGINSIGHTS_HOME/IHC/share/hadoop/common/lib/slf4j-api-1.7.5.jar
7. The Analytic Server installer performs the following steps automatically; however, if you chose a manual installation, you will need to follow these steps.

a. Copy the following JAR files from $BIGINSIGHTS_HOME/IHC/share/hadoop/common/lib/ to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
commons-configuration-1.6.jar
commons-io-2.1.jar
commons-lang-2.4.jar
jackson-core-asl-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jersey-core-1.9.jar
jersey-server-1.9.jar
jetty-6.1.26.jar
jetty-util-6.1.26.jar
servlet-api-2.5-20081211.jar
b. Copy the following JAR file from $BIGINSIGHTS_HOME/IHC/ to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEBINF/lib/
hadoop-core-2.2.0-mr1.jar

8. Copy the following JAR files from {AS_ROOT}\cloudera\cloudera4 to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
accumulo-hive-storage-handler_2.0-1.6.0.jar
hadoopmapreduce_1.1-1.0.1.0.jar
hcatalogstoragehandler_1.1-1.0.1.0.jar
6. Delete the following JAR files from {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
hadoopmapreduce-1.0.1.0.jar
accumulo-hive-storage-handler-1.6.0.jar
hcatalogstoragehandler-1.0.1.0.jar
commons-io-1.4.jar
7. Return to the instructions for all BigInsights configurations.

BigInsights 2.1.2 with Platform Symphony (HDFS)

Assuming you have followed the instructions to this point for all BigInsights configurations:

Analytic Server is installed on the BigInsights name node
1. Follow the steps for BigInsights 2.1.2 (HDFS)
2. Edit the file the file {AS_ROOT}/bin/hadoop_setup/symphony_6.1.x_pmr-site.xml so that the line <value>MapReduce6.1</value> reads <value>MapReduce6.1.1</value>. Then create the directory {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/configuration/hadoop-conf and copy the file {AS_ROOT}/bin/hadoop_setup/symphony_6.1.x_pmr-site.xml into it.
3. Return to the instructions for all BigInsights configurations.

Analytic Server is installed elsewhere
1. Follow the steps for BigInsights 2.1.2 (HDFS)
2. Copy the following JAR files to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib/
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib/activation.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib/axis-1.4.2.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib/commons-discovery.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib/commons-ego.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib/commons-soam.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib/geronimo-jaxrpc_1.1_spec-2.1.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib/wsdl4j.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/integration/IBM_BigInsights_2.1.2/IBM-pmr-hadoop-mr1-2.2.0.jar
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib/JavaSoamApi.jar
4. Copy the following library files to {AS_SERVER_ROOT}/configuration/lib_64
$BIGINSIGHTS_HOME/HAManager/data/1.2.8/linux2.6-glibc2.3-x86_64/lib/libvem.so
$BIGINSIGHTS_HOME/HAManager/data/1.2.8/linux2.6-glibc2.3-x86_64/lib/sec_ego_default.so
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libcom_platform_ACE_544_03.so.5.4.4
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libcom_platform_ACE_544_03_SSL.so.5.4.4
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libcom_platform_log4cxx_097_3.so.9
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libicudata.so.32
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libicui18n.so.32
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libicuuc.so.32
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libsec.so
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libsoam_resources.so
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libsoambase.so
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libz.so.1.old
$BIGINSIGHTS_HOME/HAManager/data/soam/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libjnativesoamapi_6.1.1_64.so
$BIGINSIGHTS_HOME/HAManager/data/soam/mapreduce/6.1.1/linux2.6-glibc2.3-x86_64/lib64/libVEMApiCommon.so
5. Copy the directory $BIGINSIGHTS_HOME/HAManager/data/kernel/conf to {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/configuration/
6. Edit the file the file {AS_ROOT}/bin/hadoop_setup/symphony_6.1.x_pmr-site.xml so that the line <value>MapReduce6.1</value> reads <value>MapReduce6.1.1</value>. Then create the directory {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/configuration/hadoop-conf and copy the file {AS_ROOT}/bin/hadoop_setup/symphony_6.1.x_pmr-site.xml into it.
7. Modify the {AS_ROOT}/bin/start.sh file to add the line
export EGO_CONFDIR=$AE_BASE/ae_wlpserver/usr/servers/aeserver/configuration/conf
following the line
export AE_BASE=`pwd`.
8. Return to the instructions for all BigInsights configurations.


BigInsights 2.1.2 with Platform Symphony (GPFS)

Follow the instructions for Platform Symphony (HDFS) and then the instructions for BigInsights 2.1.2 (GPFS) .

HCatalog configuration

When following the HCatalog configuration steps in the Administrator’s guide, in the step where you edit core-site.xml, remove all the whitespace from the <value> element so that it looks like:

<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value>

HCatalog configuration for BigInsights 2.1.2

In order to configure Analytic Server for use with HCatalog databases after a successful Analytic Server

installation:

1. Stop the Analytic Server; that is, run {AS_ROOT}/bin/stop.sh.
2. Add a hcataloginput@remote entry to the list of Analytic Server modules in the {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/configuration/config.properties file. For example:
ae.modules=restframework@local,\
objectstore,\
jndidb,\
securityprovidermanager@local,\
componentframework@remote,\
...
hcataloginput@remote
3. Uncomment or add the following lines to config.properties
hive.metastore.local=false
hive.metastore.uris=thrift://hostname:portnum
where
hostname
The name of the machine that hosts the Thrift server
portnum
The port number that is used in the HCatalog installation script
4. Make sure that the following files are available in the {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/apps/AE_BOOT.war/WEB-INF/lib directory.
The following JAR files are for HCatalog  0.12.0 and Hive 0.12.0. You must harvest the corresponding HCatalog, Hive, and dependent JAR files as appropriate for other versions.
· The following file can be copied from the HCatalog server installation.
hcatalog-core-0.12.0.jar
· The following files can be copied from the Hive server installation.
{BI_HOME}/hive/lib/hive-exec-0.12.0.jar
{BI_HOME}/hive/lib/hive-metastore-0.12.0.jar
{BI_HOME}/hive/hcatalog/share/hcatalog/hcatalog-core-0.12.0.jar
{BI_HOME}/hive/lib/guava-11.0.2.jar
{BI_HOME}/hive/lib/libfb303-0.9.0.jar
· The following files can be copied from the Hbase server installation.
{BI_HOME}/hbase/lib/netty-3.6.6.Final.jar
{BI_HOME}/hbase/lib/htrace-core-2.01.jar
{BI_HOME}/hbase/lib/jackson-jaxrs-1.8.8.jar
5. The HCatalog data sources based on the compressed files require Hadoop native libraries to be available on the Analytic Server machine. Copy the Hadoop native libraries found at:
{BI_HOME}/IHC/lib/native/Linux-amd64-64
to a directory on the Analytic Server machine and edit the LIB_PATH variable in {AS_ROOT}/bin/start.sh to include the path to that directory. For example:
export LIB_PATH=$AE_BASE/ae_wlpserver/usr/servers/aeserver/configuration/lib_32:
$AE_BASE/ae_wlpserver/usr/servers/aeserver/configuration/lib_64:
<hadoop_native_libraries_directory>
6. Run {AS_ROOT}/bin/hdfsUpdate.sh.
7. Add the property "hive.metastore.uris" with value "thrift://<HOST>:<PORT>" (the default port is 9083) into  {BI_HOME}/hive/conf/hive-site.xml
8. Grant execution permissions to "hduser" on {BI_HOME}/hive/hcatalog/sbin/hcat_server.sh
9. Run the command "hcat_server.sh start" to start the HCatalog server
10. Restart the Hive service in the BigInsights Admin Console.
11. Run {AS_ROOT}/bin/start.sh.

Accumulo configuration

The Apache Accumulo storage handler can be found at https://github.com/dvasilen/accumulo-hive-storage-manager/tree/master.  This is the master branch; from here you should switch to the branch for your version of Accumulo and Hadoop.  The JAR file can be found in the /release directory and has a name of the form:

accumulo-hive-storage-handler_<hadoopVersion>-<accumuloVersion>.jar

Migrating settings from an existing Analytic Server installation

When you install Analytic Server, you can migrate settings from an existing installation to the new installation.

New installation is on a different machine

If you install Analytic Server on a new machine, you can migrate settings to the new machine.

Steps to migrate:

1. Install Analytic Server on the new machine. If the installation on the old machine was configured to use SSL, HCatalog, Kerberos, and so on, those configuration steps should be repeated on the new machine.

Important: If the new installation uses the same Hadoop cluster as the old installation, the new installation must be given a different analytic root folder during the installation process.

2. Copy the {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/configuration/config.properties file from the original server to the new server.
3. Copy the {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/aedb directory and all of its contents from
the original server to the new server.
4. Copy the {AS_ROOT}/ae_wlpserver/usr/servers/aeserver/analytic_server.keystore file from the original server to the new server.

Note: If you are changing the authentication service; for example, moving from a Basic registry to LDAP, the migrated principals should still be valid so long as the usernames and groupnames match.

Improved ensemble modeling for very large datasets

The algorithm for building ensemble models for Tree, Linear, and Neural Net models is improved to better handle data that are not randomly distributed across uniformly sized blocks.

Reading empty/null strings in SPSS Modeler and Analytic Server

There is a minor difference in how SPSS Modeler reads a delimited file through the Var File node and how it is read when defined as an Analytic Server data source.

Modeler interprets two successive delimiters in a delimited file as the empty string. Analytic Server interprets this as a null value; an empty string in an Analytic Server delimited data source is defined by two quotes (an empty string) between the delimiters.

To force the Modeler interpretation on an Analytic Server data source, you can convert the nulls to the empty string using a Filler node with condition @NULL(fieldname) and replacement value "".  It is not possible for Modeler to read null string values from a delimited file through the Var File node.

Handling control characters in strings

Analytic Server follows the XML 1.0 standard, which has limitations on the use of control characters (0x00-0x1F) in strings, with the exception of tabs, carriage returns, and newlines. How control characters are handled is controlled by the ae.controlchars setting in the config.properties file. This setting can have one of the following values; the default is error.

error
The existence of a string value with control characters causes an error with the message "Unsupported control character found". In order to use data with control characters in strings, set ae.controlchars to setnull or strip.
Note: Defining an HCatalog- or file-based data source through the Console tries to read some records, and such attempts will fail. File-based data sources will fail with an "Error fetching record data" while HCatalog-based data sources will fail with "An error occurred loading the HCatalog schema."
setnull
Any string value containing one or more control characters is treated a null value.
Note: The null value is not the same as the empty string.
strip
Control characters are stripped (deleted) from string values. For example, the string “AB\01C\05” is converted to “ABC” (using Java string literals).