When your license limits are exceeded, IBM Big Replicate will operate in a limited manner, but allows you to apply a new license to bring the system back to full operation. Once a license is no longer valid:

Each different type of license has different limits.

Unless there’s a problem that stops you from reaching the IBM Big Replicate UI, the correct way to upgrade a node license is through the License panel, under the Settings tab.

You deploy IBM Big Replicate/IHC server nodes in proportion to the data traffic between clusters; the more data traffic you need to handle, the more resources you need to put into the IBM Big Replicate server software.

If you plan to locate both the IBM Big Replicate and IHC servers on the same machine then check the collocated Server requirements:

Before beginning installation you must have sufficient ports reserved. Below are the default, and recommended, ports.

DConE replication port: 6444
DConE port handles all coordination traffic that manages replication. It needs to be open between all IBM Big Replicate nodes. Nodes that are situated in zones that are external to the data center’s network will require unidirectional access through the firewall.

Big Replicate HTTP Server Port: 8082
The HTTP Server Port or Application/REST API is used by the IBM Big Replicate application for configuration and reporting, both internally and via REST API. The port needs to be open between all IBM Big Replicate nodes and any systems or scripts that interface with IBM Big Replicate through the REST API.

Big Replicate HTTPS Server Port: 8084
If SSL is enabled, this port is used for application for configuration and reporting, both internally and via REST API. The port needs to be open between all IBM Big Replicate nodes and any systems or scripts that interface with IBM Big Replicate through the REST API.

Big Replicate Request port: 8023
Port used by IBM Big Replicate server to communicate with HCFS/HDFS clients. The port is generally only open to the local IBM Big Replicate server, however you must make sure that it is open to edge nodes.

Big Replicate Server listening port: 8024
Port used by IBM Big Replicate server to listen for connections from remote IHC servers. It is only used in unidirectional mode, but it’s always opened for listening. Remote IHCs connect to this port if the connection can’t be made in the other direction because of a firewall. The SSL configuration for this port is controlled by the same ihc.ssl.enabled property that is used for IHC connections performed from the other side. See Enable SSL for IBM Big Replicate.

IHC ports: 7000-range or 9000-range
7000 range, (the exact port is determined at installation time based on what ports are available), it is used for data transfer between Big Replicate Server and IHC servers. It must be accessible from all IBM Big Replicate nodes in the replicated system.
9000 range, (the exact port is determined at installation time based on available ports), it is used for an HTTP Server that exposes JMX metrics from the IHC server.

The Hadoop distributions and versions supported are:

Supported Big Data applications my be noted here, as we complete testing:

We’ll now look at what you should know and do as you begin the installation.

If you are running Kerberos on your cluster you should consider the following requirements:

Before running the installer on a platform that is secured by Kerberos, you’ll need to run through the following steps: Setting up Kerberos.

For example, if you connect a Zone that runs CDH, which has superuser 'hdfs" with a zone running MapR, which has superuser 'mapr', you would need to create the user 'hdfs' on the MapR zone and 'mapr' on the CDH zone.

Before you start the installation you must ensure that there are no existing IBM Big Replicate installations or IBM Big Replicate components installed on your elected machines. If you are about to upgrade to a new version of IBM Big Replicate you must first see the Uninstall chapter.

then HADOOP_HOME must be set to

You need to match the IBM Big Replicate installer file to each data center’s version of Hadoop. Installing the wrong version of IBM Big Replicate will result in the IHC servers being misconfigured.

After completing an evaluation deployment, you will need to contact IBM about getting a license file for moving your deployment into production.

These tags are appended to /var/ibm/slmtags every 60 minutes, but the frequency of generation can be modified through the ibm.slm.tag.freq configuration property.

An example of the generated data:

IBM Big Replicate uses specific SubType metric entries that should be interpreted as follows:

Use the following steps to complete an installation using the installer file. This requires an administrator to enter details throughout the procedure. Once the initial settings are entered through the terminal session, the installation is then completed through a browser or alternatively, using a Silent Installation option to handle configuration programmatically.
Note: The screenshots shown in this section are from a Cloudera installation so there may be slight differences to your set up.

Follow this section to complete the installation by configuring IBM Big Replicate using a browser-based graphical user interface.

The silent installation process supports two levels: Unattended installation handles just the command line steps of the installation, leaving the web UI-based configuration steps in the hands of an administrator. See unattended installation.

Fully Automated also includes the steps to handle the configuration without the need for user interaction.

Use the following command for an unattended installation where an administrator will complete the configuration steps using the browser UI.

There are a number of properties that need to be set up before the installer can be run:

This mode only automates the initial command line installation step, the configuration steps still need to be handled manually in the browser steps.

This mode is closer to a full "Silent" installation as it handles the configuration steps as well as the installation.

Properties that need to be set:

Cluster Manager Variables are deprecated
The cluster manager variables are mostly redundant as they generally get set in different processes though they currently remain in the installer code.

If this part of the installation fails it is possible to re-run the silent_installer part of the installation by running:

This procedure is useful for UI-only installations:

See the Uninstall Script Usage Section for information on removing Big Replicate.

For every package of IBM Big Replicate there’s both an env.sh and a .properties file. The env.sh sets environment variables that complete the initial command step of an installation. The env.sh also points to a properties file that is used to automate the browser-based portion of the installer. The properties files for the different installation types are provided below:

Be sure to point your replicated directory to your NFS mount, either directly or using a a symlink.

The following procedure covers the installation and setup of IBM Big Replicate deployed over the LocalFileSystem. This requires an administrator to enter details throughout the procedure. Once the initial settings are entered through the terminal session, the deployment to the LocalFileSystem is then completed through a browser.

Follow the first few steps given in the On-premises installation guide. Make sure that you use the LocalFileSystem installer, for example fusion-ui-server-localfs_rpm_installer.sh.

Once the fusion-ui-server has started, follow the steps below to configure IBM Big Replicate with Swift through the browser.

The following requirements come into play if you have deployed IBM Big Replicate using with its native fusion:/// URI.

In order to store a Hive table in IBM Big Replicate you specify a IBM Big Replicate URI when creating a table. E.g. consider creating a table called log that will be stored in a replicated directory.

CREATE TABLE log(requestline string) stored as textfile location 'fusion:///repl1/hive/log';. Note: Replicating table storage without sharing the Hive metadata will create a logical discrepancy in the Hive catalog. For example, consider a case where a table is defined on one cluster and replicated on the HCFS to another cluster. A Hive user on the other cluster would need to define the table locally in order to make use of it.

Hive from CDH 5.3/5.4 does not work with IBM Big Replicate, (because of HIVE-9991). To get it working with CDH 5.3 and 5.4. you need to modify the default Hive file system setting. In Cloudera Manager, add the following property to hive-site.xml:

This property should be added in 3 areas:

It’s possible to configure Hive to use IBM Big Replicate URIs as output paths for storing data, to do this you must specify a Big Replicate URI when writing data back to the underlying Hadoop-compatible file system (HCFS). For example, consider writing data out from a table called log to a file stored in a replicated directory:

In this section we’ll describe how to configure Hive to use fusion URIs as input paths for loading data.

It is not common to load data into a Hive table from a file using the fusion URI. When loading data into Hive from files the core-site.xml setting fs.default.name must also be set to fusion, which may not be desirable. It is much more common to load data from a local file using the LOCAL keyword:

If you do wish to use a fusion URI as a load path, you must change the fs.defaultFS setting to use IBM Big Replicate, as noted in a previous section. Then you may run:

Advanced configuration - please contact IBM before attempting
In this section we’ll describe how to share the Hive metastore between two clusters. Since IBM Big Replicate can replicate the file system that contains the Hive data storage, sharing the metadata presents a single logical view of Hive to users on both clusters.

When sharing the Hive metastore, note that Hive users on all clusters will know about all tables. If a table is not actually replicated, Hive users on other clusters will experience errors if they try to access that table.

There are two options available.

In this configuration, the Hive metastore is configured normally on one cluster. On other clusters, the metastore process points to a read-only copy of the metastore database. MySQL can be used in master-slave replication mode to provide the metastore.

In this configuration, the Hive metastore is writable on all clusters.

Standard

For Cloudera CDH: See Hive Metastore High Availability.

For Hortonworks/Ambari: High Availability for Hive Metastore.

Manual Replication

In order to manually replicate metastore data ensure that the DDLs are placed on two clusters, and perform a partitions rescan.

If you plan to use IBM Big Replicate’s own fusion:/// URI, then you will need to use the provided parcel (see the screenshot, below for link in the Client Download section of the Settings screen):

Follow the same steps described for installing the IBM Big Replicate client, downloading the parcel and SHA file, i.e.:

In order for Impala to load the Big Replicate Client jars, the user needs to make a small configuration change in their Impala service, through Cloudera Manager. In Cloudera Manager, the user needs to add an environment variable in the section Impala Service Environment Advanced Configuration Snippet (Safety Valve).

The following command gives an example of how to do this. Note that you need to replace the versions given with your Big Replicate and Cloudera versions.

Presto is an open source distributed SQL query engine for running interactive analytic queries. It can query and interact with multiple data sources, and can be extended with plugins.

Presto requires the use of Java 8 and has internal dependencies on Java library versions that may conflict with those of the Hadoop distribution with which it communicates when using the “hive-hadoop2” plugin. For example, Presto makes use of guava-21.0.jar, while HDP 2.4 uses guava-11.0.2.jar.

IBM Big Replicate leverages a replacement client library when overriding the hdfs:// scheme for access to the cluster file system in order to coordinate that access among multiple clusters. This replacement library is provided in a collection of jar files in the /opt/wandisco/fusion/client/lib directory for a standard installation. These jar files need to be available to any process that accesses the file system using the com.wandisco.fs.client.FusionHdfs implementation of the Apache Hadoop FileSystem API.

Because Presto requires these classes to be available to the hive-hadoop2 plugin, they must reside in the plugin/hive-hadoop2 directory of the Presto installation.

It is also important to confirm that the Presto configuration includes the necessary properties to function correctly with the hive-hadoop2 plugin.

The specific values below will need to be adjusted for the actual environment, including references to the WANdisco replicated metastore, the HDP cluster configuration that includes Big Replicate configuration, and Kerberos-specific information to allow Presto to interoperate with a secured cluster.

Keytabs and principals will need to be configured correctly, and as the hive-hadoop2 Presto plugin uses YARN for operation, the /user/yarn directory must exist and be writable by the yarn user in all clusters in which Big Replicate operates.

Presto embeds Hadoop configuration defaults into the hive-hadoop2 plugin, including a core-default.xml file that specifies the following property entry:

Although Presto allows the hive-hadoop2 plugin to use additional configuration properties by adding entries like the following in a .properties file in the etc/catalog directory:

This entry allows extra configuration properties to be loaded from a standard Hadoop configuration file, but those entries cannot override settings that are embedded in the core-default.xml that ships with the Presto hive-hadoop2 plugin.

In a kerberized implementation the Big Replicate client library relies on the ability to read the hadoop.security.authentication configuration property to determine if it should perform a secure handshake with the Big Replicate server. Without that property defined, the client and server will fail to perform their security handshake, and Presto queries will not succeed.

The solution to this issue is to update the core-default.xml file contained in the hive-hadoop2 plugin:

Edit the core-default.xml file to update the hadoop.security.authentication property so that its value is “kerberos”

Distribute the hadoop-apache2-0.10.jar to all Presto nodes, and restart the Presto coordinator.

Something to be aware of in Hyper Scale-Out Platform (HSP) installations - when you install the client stack, the fusion-client RPM creates symlinks in /usr/hdp/current/oozie-server/libext for the client jars. However, these get left behind if the client stack/RPM are removed.

If a new version of fusion-client is installed, Oozie server will refuse to start because of the broken symlinks.

Oracle:BDA is built on top of Cloudera’s Hadoop and requires some extra steps to support multi-homed network environment.

There’s a known issue concerning configuration and the Cloudera Navigator Metadata Server classpath.

Error message:

There’s no clear way to override the fs.hdfs.impl setting just for the Navigator Metadata server, as is required for running with IBM Big Replicate.

Use the following fix script to overcome the problem:

The environment variables are provided here - navigator_env.txt

You need to put this in the configuration for the Cloudera Management Service under "Navigator Metadata Server Environment Advanced Configuration Snippet (Safety Valve)". This modification needs to be done any time the Cloudera version is changed (upgrade or downgrade).

In order to make Hive with Tez work, you need to append the Big Replicate jar files in tez.cluster.additional.classpath.prefix under the Advanced tez-site section:

e.g. IBM Big Replicate tree

Running Hortonworks Data Platform, the tez.lib.uris parameter defaults to /hdp/apps/${hdp.version}/tez/tez.tar.gz. So, to add Big Replicate libs, there are three options.

Option 1: Delete the tez.lib.uris path, e.g. "/hdp/apps/${hdp.version}/tez/tez.tar.gz". Instead, use a list including the path where the tez.tar.gz file will unpack, and the path where Big Replicate libs are located.

Option 2: Or unpack tez.tar.gz, repack with IBM Big Replicate libs and re-upload to HDFS.

Option 3: Alternatively, you may set the tez.lib.uris property with the path to the IBM Big Replicate client jar files, e.g.

The following configuration changes are needed when running Tez with Low Latency Analytical Processing functionality.

These instructions will ensure that you get properly configured for running Apache Sentry with IBM Big Replicate.

Requirements:

Steps:

The hive-site.xml & sentry-site.xml from the prior step should now be uploaded to the WD Hive Metastore node.

They should be placed in the following path and should be owned by hive:hadoop with permissions 644.

If the file /etc/wandisco/hive/hive.keytab exists, thus should be deleted.

Leave all other files and symlinks in the path "as is".

Within Cloudera Manager, you should now restart the WD Hive Metastore service. This step is required here to load the updated configuration, but additionally allow generation of a new file for the next phase.

Example command

The wd-hive-site.xml from the prior step should now be uploaded to the Big Replicate Server node.

Addiitonally, the files hive-site.xml & sentry-site.xml captured 4 steps before should also be uploaded to the Big Replicate Server node.

They should be placed in the following path and files should be owned by hive:hadoop with permissions 644.

If the file /etc/wandisco/hive/hive.keytab exists, thus should be deleted.

Leave all other files and symlinks in the path "as is".

You should now restart the Big Replicate Server.

Getting set up with the default URI is simple, Solr just needs to be able to find the Big Replicate client jar files that contain the FusionHdfs class.

This is a minimal working solution with Solr on top of fusion.

If you want to set up Flume through Cloudera Manager follow these steps:

There is a known issue where Spark is not picking up hive-site.xml, So that Hadoop configuration is not localised when submitting job in yarn-cluster mode (Fixed in version Spark 1.4).

You need to manually add it in by either:

Spark applications are run on a cluster as independent sets of processes, coordinated by the SparkContext object in the driver program. To run on a cluster, the SparkContext can connect to several types of cluster managers (either Spark’s own standalone cluster manager, Mesos or YARN), which allocate resources across applications. Once connected, Spark acquires executors on nodes in the cluster, which are processes that run computations and store data for your application. Next, it sends your application code (defined by JAR or Python files passed to SparkContext) to the executors. Finally, SparkContext sends tasks to the executors to run.

IBM Big Replicate uses a replacement client library when overriding the hdfs:// scheme for access to the cluster file system in order to coordinate that access among multiple clusters. This replacement library is provided in a collection of jar files in the /opt/wandisco/fusion/client/lib directory for a standard installation. These jar files need to be available to any process that accesses the file system using the com.wandisco.fs.client.FusionHdfs implementation of the Apache Hadoop File System API.

Because Spark does not provide a configurable mechanism for making the Big Replicate classes available to the Spark history server, the Spark Executor or Spark Driver programs, IBM Big Replicate client library classes need to be made available in the existing Spark assembly jar that holds the classes used by these Spark components. This requires updating that assembly jar to incorporate the Big Replicate client library classes.

This is one of a number of methods that may be employed to provide Big Replicate-Spark integration. We hope to cover some alternate methods at a later date.

The following example covers how you may upgrade the Spark Assembly as part of a Big Replicate upgrade. This example uses CDH 5.11, although it can be applied generically:

The resulting spark-assembly is now purged and requires one of two actions:

If Spark2 is installed after IBM Big Replicate there will be a need to manually symlink the IBM Big Replicate clien t libraries.

Create a symlink like this:

For HBase to run with IBM Big Replicate, the following directories need to be created and permissioned, as shown below:

The steps below provide a method of handling a recovery using a cold back-up. Note that multiple HMaster/region servers restarts might be needed for certain steps, since hbck command generally requires master to be up, which may require fixing filesystem-level inconsistencies first.

Running service check on Slider in Ambari results in a ClassNotFoundException error. To fix this, add the following line into 'Advanced slider-env.sh' configuration in Ambari:

To configure Big Replicate for access to encrypted zones, two aspects need to be considered:

The following items need to be considered within KMS configuration to ensure Big Replicate has access:

The kms-site configuration (such as Advanced kms-site in Ambari) contains its own auth_to_local type parameter called “hadoop.kms.authentication.kerberos.name.rules”

Ensure that any auth_to_local mapping used for the Big Replicate principal is also contained here. This can be most easily achieved via simple copy/paste from core-site.xml.

The kms-site configuration (such as Custom kms-site in Ambari) contains proxyuser paramaters such as

Entries should be created for the local Big Replicate user (after auth_to_local translation) to allow Big Replicate to proxy / impersonate other users requests. This could be as simple as.

In the dbks-site configuration, the parameter hadoop.kms.blacklist.DECRYPT_EEK exists. Ensure this does not contain the username that Big Replicate uses (after auth_to_local translation).

In the KMS ACLs, such as using Ranger KMS, ensure that the Big Replicate user (after auth_to_local translation) has "Get Metadata" and "Decrypt EEK" permissions to keys.

This could be granted access to all keys. This will avoid a need to review rules when new keys are added. However, Big Replicate will only need these permissions to keys that apply to zones that fall within a replicated path. Consideration is needed here based on the user that Big Replicate has been configured as - either "HDFS" will need access to EEKs, OR the Big Replicate user will need access, OR the supergroup could be given access to EEKs (it is enabled by default on Ambari but disabled on CDH), and then make the Big Replicate user a member of the supergroup.

If you do not perform the correct configuration, both local operations (as performed by a client) and / or the replicated actions may fail when the Big Replicate client is invoked. This should only apply to replicated paths.

So to troubleshoot:

You can also perform an installation to a custom directory using the following alternative:

Inject the environmental variable:

Run the installer as per the usual method, i.e.:

The installer will use the provided path for the installation, as described in the main procedure.

When installing to a customer location, you will need to ensure that your clients are configured with the matching location. You should be able to correctly install clients using the normal procedure, outlined above. See Client Installation, immediately below.

The IBM Big Replicate installer doesn’t currently handle the installation of the client to the rest of the nodes in the cluster. You need to go through the following procedure:

In the Client Installation section of the installer you will see line "Download a list of your client nodes" along with links to the client RPM packages.

RPM package location
If you need to find the packages after leaving the installer page with the link, you can find them in your installation directory, here:

If you are installing the RPMs, download and install the package on each of the nodes that appear on the list from step 1.

Installing the client RPM is done in the usual way:

HDP2.1/Ambari 1.6: Start services after installation
When installing clients via RPM into HDP2.1/Ambari 1.6., ensure that you restart services in Ambari before continuing to the next step.

Debian not supported
Although Ubuntu uses Debian’s packaging system, currently Debian itself is not supported. Note: Hortonworks HDP does not support Debian.

If you are running with an Ubuntu Linux distribution, you need to go through the following procedure for installing the clients using Debian’s DEB package:

For deployments into Cloudera clusters, clients can be installed using Cloudera’s own packaging format: Parcels.

For deployments into Hortonworks HDP/Ambari/IBM BigInsights cluster, version 1.7 or later. Clients can be installed using Hortonwork’s own packaging format: HDP Stack.

Ambari 1.6 and earlier
If you are deploying with Ambari 1.6 or earlier, don’t use the provided Stacks, instead use the generic RPMs.

Ambari 1.7
If you are deploying with Ambari 1.7, take note of the requirement to perform some necessary restarts on Ambari before completing an installation.

Ambari 2.0
When adding a stack to Ambari 2.0 (any stack, not just IBM Big Replicate client) there is a bug which causes the YARN parameter yarn.nodemanager.resource.memory-mb to reset to a default value for the YARN stack. This may result in the Java heap dropping from a manually-defined value, back to a low default value (2Gb). Note that this issue is fixed from Ambari 2.1.

Upgrading Ambari
When running Ambari prior to 2.0.1, we recommend that you remove and then reinstall the IBM Big Replicate stack if you perform an update of Ambari. Prior to version 2.0.1, an upgraded Ambari refuses to restart the IBM Big Replicate stack because the upgrade may wipe out the added services directory on the stack.

If you perform an Ambari upgrade and the Ambari server fails to restart, the workaround is to copy the IBM Big Replicate service directory from the old to the new directory, so that it is picked up by the new version of Ambari, e.g.:

Again, this issue doesn’t occur once Ambari 2.0.1 is installed.

HDP 2.3/Ambari 2.1.1 install
There’s currently a problem that can block the installation of the IBM Big Replicate client stack. If the installation of the client service gets stuck at the "Customize Service" step, you may need to use a workaround:

Make sure the service components are created and the configurations attached by making a GET call, e.g.

1. Add the service

2. Add the component

3. Get a list of the hosts

4. For each of the hosts in the list, add the FUSION_CLIENT component

5. Install the FUSION_CLIENT component

On MapR clusters, you need to copy IBM Big Replicate configuration onto all other nodes in the cluster:

When installing IBM Big Replicate on a system that already has an earlier version of IBM Big Replicate installed, you need to first ensure that components and configuration for an earlier installation have been removed. Go through the steps in the Uninstall chapter, Clean IBM Big Replicate HD, before installing a new version of IBM Big Replicate.

There’s an uninstaller script, so that you don’t need to clean up your deployment manually. If you used the unified installer then use the uninstall script, following the steps described in Uninstall.

For a complete uninstallation, remove the following properties from the core-site.xml:

Reinstalling fusion server only
If you reinstall the fusion-server without also reinstalling the fusion-ui-server, then you should restart the fusion-ui-server service to ensure the correct function of some parts of the UI. If the service is not restarted then you may find that the dashboard graphs stop working properly, along with the UI’s Stop/start controls. e.g. run:

This section runs through the installation of IBM Big Replicate into an Openstack environment using Swift storage. We use Bluemix, the IBM cloud managed Swift solution, as an example but other implementations are available.
Currently this deployment is limited to an active-passive configuration that would be used to ingest data from your on-premises cluster to your Swift storage.

Before you begin an installation you need to have a Bluemix (or equivalent) account with container(s) set up. This guide runs through installing IBM Big Replicate and using it with Bluemix, but not how to set up Bluemix.

Make sure that you have the following directories created and suitably permissioned. Examples:

The installation process runs through the following steps:

Follow the first few steps given in the On-premises installation guide. Make sure that you use a Swift specific installer, for example fusion-ui-server-swt_rpm_installer.sh.

Once the fusion-ui-server has started, follow the steps below to configure IBM Big Replicate with Swift through the browser.

Follow this section to complete the installation by configuring IBM Big Replicate on a server that will place data that is replicated from your on-premises cluster to your Bluemix/OpenStack Swift storage. This second node can also be on-premises or co-located with your OpenStack platform.

Open a web browser and point it at the provided URL. e.g

Once you have entered this information click Validate.
The following Swift properties are validated:

The installer checks that the Swift account details are valid for accessing the segment container. If the validation fails, you should recheck your Swift account credentials.

If everything is successfully validated, click Next step.

It’s now time to demonstrate data replication from the on-premises cluster to the Swift storage. First we need to perform a synchronization to ensure that the data stored in both zones is in exactly the same state.

You should complete any necessary repairs to file consistency between your two clusters. Follow the steps provided in the User Guide:

You can complete a Swift installation using the Silent Installation procedure, putting the necessary configuration in the swift_silent_installer.properties and swift_silent_installer_env.sh as described in the section that covers Silent Installation.

In the next step you should complete any necessary repairs to file consistency between your two clusters. Follow the steps provided in the following sections:

Swift containers can appear to have a file-size discrepancy, looking smaller than the sum of their stored files. The reason for this apparent discrepancy is given below.

Files that are smaller than the segment size of a container are, predictably, stored directly to the container - as would be expected. However, large files that are bigger than the container’s segment size are actually stored in a companion container that is used for segments and has the same name as the parent container with the suffix "_segments". Segmented files appear in the main container, although these are empty manifest objects that symlink to the segments that correspond to the file in question.

So, for measuring the actual volume of data stored in a Swift container, you must also take into account the size of the corresponding segment’s container.

To start IBM Big Replicate UI:

To shut down:

Shutdowns take some time

The shutdown script attempts to stop processes in order before completing, as a result you may find that (from IBM Big Replicate 2.1.3) shutdowns may take up to a minute to complete.

The start-up script for persistent running of IBM Big Replicate is in the /etc/init.d directory. Run the script with the help command to list the available commands:

Check the running status (with current process ID):

Check the version:

IBM Big Replicate’s replication system is deeply tied to the cluster’s file system (HDFS). If HDFS is shut down, the IBM Big Replicate server will no longer be able to write to HDFS, stopping replication even if the cluster is brought back up.

To avoid replication problems:

Currently you need to use the same username and password that are required for your platform manager, e.g. Cloudera Manager or Ambari. In a future release we will separate IBM Big Replicate UI from the manager and use a new set of credentials.

You can change the user/password credentials using the following procedure:

Using IBM Big Replicate UI’s log viewer (View Logs):

Unless configured differently, IBM Big Replicate logs default to the following locations:

Logs use UTC timezone by default but this can be manually altered through log4j configuration if required.

To alter the timezone the xxx.layout.ConversionPattern property needs to be overwritten.

{UTC} can be replaced with, for example {GMT} or {ITC+1:30}. If offsetting from a timezone, + or - can be used, hours must be between 0 and 23, and minutes must be between 00 and 59.

This property is located in several properties files. For an example set up these are listed below, but the exact paths may differ for your set up:

After updating all the relevant files, IBM Big Replicate needs to be restarted for the changes to take effect.

At startup the default log location is /dev/null. If there’s a problem before log4j has initialised this will result in important logs getting lost. You can set the log location to a filespace that preserve early logging.

Edit fusion_env.sh adding paths to the following properties:

The induction process performs some validation before running. If this validation failures you will quickly see a warning messages appear.

These instructions cover the setting up of Big Replicate’s Authz function, that works through Kerberos.

You now need to configure users in the core-site file, through your Hadoop manager: