xStats Generic CSV Adapter Quick Start Guide

ABOUT

This document describes the installation of the xStats Generic CSV adapter using SevOne NMS. The same procedure should work with prior releases but it has not be tested. The following topics are discussed in this document.

  • Install xStats package for the Generic CSV adapter
  • Start / Stop the simulator for each adapter
  • Configure the collector for each adapter
Important: The ownership of /opt/sevone-xstats is 1003:1003 (UID:GID).

PREREQUISITES

Prior to installing your xStats adapters, you will need the following.

Requirement Description
Snapshot of the vPAS If you are using a virtualized appliance, create a snapshot of your vPAS before getting started. This will be useful if you need to start over with a clean slate.
IP address or hostname of the PAS Make sure to have the IP address or hostname of the PAS so you can connect to it via SSH.
IP address of the hot standby appliance (HSA) pair This applies only if you have an HSA pair.
Support access to the PAS You will need the support username and password for your SSH session.
Important: NMS Container
To run the commands in this guide, you must be in the NMS container. To enter the NMS container, execute the following command.

podman exec -it nms-nms-nms /bin/bash
An SSH client To SSH into the PAS, you will need an SSH client such as PuTTY.
PAS version It is a good idea to know which version of SevOne NMS is installed on the PAS.
GenericCSVTransform-<version>.phar.spk

Installation package for your xStats Generic CSV adapter. For example, IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-v7.1.0.phar.spk.

The latest .phar.spk file can be downloaded IBM Passport Advantage (https://www.ibm.com/software/passportadvantage/pao_download_software.html) via Passport Advantage Online. However, if you are on a legacy / flexible SevOne contract and do not have access to IBM Passport Advantage but have an active Support contract, please contact SevOne Support Team for the file.


Once you have downloaded the package, unpack the fix / upgrade packs to obtain the required files.

Note: Key xStats Terms
Key xStats Terms Description
Collector A script responsible for collecting data files from a remote or foreign host.
File Collector Aggregator Daemon (fcad) A daemon responsible for collecting data files and processing them into a format that allows the data to be inserted into the SevOne data database.
Package A component that is installed on a customer's PAS and contains the xStats adapter along with xStats configuration information, including the parser source code and device-object-indicator settings for all objects and indicators.
Note: If you have a cluster and you are installing the xStats adapters on only one of the peers, SevOne recommends installing on the least utilized peer. You can also install adapters on ➤   Multiple Peers Installation.

DOWNLOAD GENERIC CSV ADAPTER

This section describes how to download the Generic CSV adapter obtained from IBM Passport Advantage via Passport Advantage Onlineor SevOne Support Team..

  1. Using ssh, log into SevOne NMS appliance as support.
    ssh support@<NMS appliance>
    Important: Run the commands below from the NMS container.
  2. You will require the latest Generic CSV adapter package (.phar.spk). Download the latest .phar.spk file IBM Passport Advantage via Passport Advantage Onlineor SevOne Support Team directly to your SevOne NMS appliance using wget to the /var/support directory.
    wget <location to download .phar.spk files from> -P /var/support

    For example, IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-v7.1.0.phar.spk file.

  3. Ensure the .phar.spk files have been downloaded successfully without any errors.
    
    cd /var/support
    
    ls -al
    

GENERIC CSV ADAPTER

Using ssh, log into SevOne NMS appliance as support.


ssh support@<NMS appliance>
Important: Installation commands must run from the NMS container.

➤   Install Generic CSV Adapter

  1. Extract the CSV package.
    Example
    
    cd /var/support
    
    
    php IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-v7.1.0.phar.spk
    
    
    Output:
    #!/usr/bin/php
    The ADK shared library was updated!
    Package source moved to: /opt/sevone-xstats/GenericCSVTransform
    Sync ADK shared directory. [ OKAY ]
    Sync adapter directory. [ OKAY ]
    
  2. Change directory to where the adapter has extracted the files.
    Example
    cd /opt/sevone-xstats/GenericCSVTransform
  3. Check to ensure the parser source can be installed. Option -c only performs the check.
    Example
    
    php bin/console.php install -c
    
    
    Output:
    Adapter version requirements are compatible with the NMS Version
    Adapter sources can be inserted into database.
    Object types can be inserted into database.
    
  4. Install the parser stub.
    Example
    php bin/console.php install
    
    
    Output:
    Adapter version requirements are compatible with the NMS Version
    Adapter sources can be inserted into database.
    Object types can be inserted into database.
    Adapter sources were installed successfully.
    Object were inserted into database.
    Note: To view the available commands for the console.php script, enter the following command. codeblock outputclass="language-bourne">php bin/console.php
  5. Optional (informational only) - At this point, the xStats collection source for the CSV adapter has not been created in SevOne NMS. However, the GenericCSVTransform source type is available in the add/edit source pop-up. Perform the following actions to view.
    1. From a web browser of your choice, log into your SevOne NMS.
    2. Go to Administration > Monitoring Configuration > xStats Source Manager. You will notice that there is no xStats collection source for the CSV adapter.
      xStatsCSVNoStats
    3. Click Add Source button to display the Add Source pop-up.
      xStatsCSVAddSource
    4. Click the Source Type drop-down to view the available source types. You should only see one source type GenericCSVTransform.
      xStatsCSVAddSourceType Generic CSV
    5. Click Cancel to close.
  6. This step applies only if you are using a jump box. You have GenericCSVTransform source type installed for the local file system, which appears as GenericCSVTransform in the pop-up screenshot above. If you plan to use the local file system and not SFTP, you can skip this step. However, if you plan to use the SFTP source type, you will need to install it too.

    Enter the following command to install the SFTP source type.
    
    php /opt/sevone-xstats/GenericCSVTransform/bin/console.php install-sftp
    
    
    Output:
    Parser source code has been inserted
    Object types have been inserted/updated
    

    After you execute this command, the SFTP source type, GenericCSVTransform SFTP will appear in the pop-up when you click the Source Type drop-down. You may need to click Refresh in the upper-right corner to see GenericCSVTransform SFTP from the drop-down list.
    xStatsCSVAddSourceType Generic CSV with SFTP

➤   Run Simulator & Add Sources

For each source type you plan to use for the adapter, three .json files are required with the following information.

  1. collector configuration settings for the source
  2. parser configuration settings for the source
  3. Object type and Indicator type information for the source

Please refer to ➤   Prepare .json Configuration Files on how to prepare the .json files.

Start Simulator

Important: The simulator is only used for testing purposes. In the real environment, you must stop the simulator to start the collection of actual data. For details on how to stop the simulator, please refer to ➤   Stop / Delete Simulator Data
From your ssh client, enter the following command to start the simulator.

php /opt/sevone-xstats/GenericCSVTransform/bin/console.php start-simulator

Output:
Setting simulator configuration!
Adding simulator cron job!
Adding ssh keys for localhost!
The simulator is configured and started!
To configure new sources for the simulator data you can use the following commands:

-----------------------------
for Profilel:
php bin/console.php configure --sourceName GenericCSVSimulator.
Profilel --file simulator/config/simulator.collector.sample.lfs.profilel.json--parserConfigs
imulator/config/output-dir-1.config.parser.json --objectIndicatorTypes simulator/config/output-dir-1.object_indicator_type_info.json

-----------------------------
for Profile2:
php bin/console.php configure --sourceName GenericCSVSimulator Profile2 --file simulator/config/simulator.collector.sample.lfs.profile2.json --parserConfig s imulator/config/output-dir-2.config.parser.json --objectIndicatorTypes simulator/config/output-dir-2.object _indicator.
_type_info.json

-----------------------------
for Profile3:
php bin/console.php configure --sourceName GenericCSVSimulator_Profile3 --file simulator/config/simulator.collector.sample.lfs.profile3.json --parserConfig s imulator/config/output-dir-3.config-parser.json --objectIndicatorTypes simulator/config/output-dir-3.object
_indicator_ type_info.json
Note: When you start the simulator by running the command above, the output contains commands for adding the three sources. These sources are used specifically for the simulator.

You are now ready to add the sources.

Add Sources

To add the sources, perform the actions below.

  1. Add the first source.
    
    php /opt/sevone-xstats/GenericCSVTransform/bin/console.php \
    configure --sourceName GenericCSVSimulator_Profile1 \
    --file /opt/sevone-xstats/GenericCSVTransform/simulator/config/simulator.collector.sample.lfs.profile1.json \
    --parserConfig /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-1.config.parser.json \
    --objectIndicatorTypes /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-1.object_indicator_type_info.json
    
    
    Output:
    objectIndicatorTypes is validated!!!
    parserConfig is a valid parser configuration!!!
    Source name can be created.
    Configuration file seems OK.
    Cron entries can be created.
    Logs and rotation can be created.
    Adding source to database.
    Source has been inserted into database.
    Collector configuration saved to: /opt/sevone-xstats/GenericCSVTransform/sources/source_1/config/collector.json.
    Collector crons have been configured.
    Log dir and rotations have been configured.
    Configuration was successful.
    objectIndicatorTypes are installed!!!
    parserConfig is installed!!!
    
    Note: Explanation of the command:
    • php /opt/sevone-xstats/GenericCSVTransform/bin/console.php configure - this is the command to add the source, before the necessary parameters are added.
    • --sourceName GenericCSVSimulator_Profile1 - --sourceName is the first parameter (you may also use -s) which specifies the source name. In this case, the source is named GenericCSVSimulator_Profile1.
    • --file /opt/sevone-xstats/GenericCSVTransform/simulator/config/simulator.collector.sample.lfs.profile1.json - this is the second parameter. Parameter, --file (you may also use -f), points to the .json file used to configure the collector for this source. The parameter is followed by the path to the file and the filename, /opt/sevone-xstats/GenericCSVTransform/simulator/config/simulator.collector.sample.lfs.profile1.json. You must replace the path with the path to the .json file containing the collector configuration for the source you are adding.
    • --parserConfig /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-1.config.parser.json - this is the third parameter. Parameter, --parserConfig, points to the .json file used to configure the parser for this source. The parameter is followed by the path to the file and the filename, /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-1.config.parser.json. You must replace the path with the path to the .json file containing the parser configuration for the source you are adding.
    • --objectIndicatorTypes /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-1.object_indicator_type_info.json - this is the fourth parameter. Parameter, --objectIndicatorTypes, points to the .json file that contains the object type and indicator type information for this source. The parameter is followed by the path to the file and the filename, /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-1.object_indicator_type_info.json. You must replace the path to the .json file containing the object type and indicator type information for the source you are adding.
  2. Add the second source.
    
    php /opt/sevone-xstats/GenericCSVTransform/bin/console.php \
    configure --sourceName GenericCSVSimulator_Profile2 \
    --file /opt/sevone-xstats/GenericCSVTransform/simulator/config/simulator.collector.sample.lfs.profile2.json \
    --parserConfig /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-2.config.parser.json \
    --objectIndicatorTypes /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-2.object_indicator_type_info.json
    
  3. Add the third source.
    
    php /opt/sevone-xstats/GenericCSVTransform/bin/console.php \
    configure --sourceName GenericCSVSimulator_Profile3 \
    --file /opt/sevone-xstats/GenericCSVTransform/simulator/config/simulator.collector.sample.lfs.profile3.json \
    --parserConfig /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-3.config.parser.json \
    --objectIndicatorTypes /opt/sevone-xstats/GenericCSVTransform/simulator/config/output-dir-3.object_indicator_type_info.json
    

Return to your SevOne NMS appliance where the sources have been added. From Administration > Monitoring Configuration > xStats Source Manager, you will see that the xStats collection sources have been created for each source added via the ssh client.
xStatsCSVSourcesAdded

View Cron File, Run Collector Manually, & Check Data

  1. From your ssh client, enter the following command to confirm that the cron file for the CSV adapter has been created.
    
    ls -al /etc/cron.d
    
    
    Output:
    total 8
    drwxr-xr-x.  2 root sevone  113 Aug 19 19:14 .
    drwxr-xr-x. 80 root root     30 Aug 19 18:49 ..
    lrwxrwxrwx.  1 root sevone   29 Aug 19 14:19 mode -> /config/cron-available.d/mode
    lrwxrwxrwx.  1 root sevone   35 Aug 19 14:19 sevone-nms -> /config/cron-available.d/sevone-nms
    -rw-r--r--.  1 root root   1045 Aug 19 19:14 xstats-GenericCSVTransform
    -rw-r--r--.  1 root root    436 Aug 19 19:12 xstats-GenericCSVTransformSimulator
    
  2. To view the contents of the cron file, enter the following command.
    
    cat /etc/cron.d/xstats-GenericCSVTransform
    
    
    Output:
    # SevOne xstats adapter crontab file.
    # Global variables
    SHELL=/bin/bash
    PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
    MAILTO=""
    HOME=/
    # crontab format:
    # <minute> <hour> <day of the month> <month> <day of the week> <user> <what to do>
    
    
    */5 * * * * ingestion php /opt/sevone-xstats/GenericCSVTransform/bin/console.php acquire-data --source 1 &>> /var/log/xstats/GenericCSVTransform/adapter.log
    #-------------------------------------------
    
    
    0 0 * * * ingestion php /opt/sevone-xstats/GenericCSVTransform/bin/console.php cleanup --source 1 &>> /var/log/xstats/GenericCSVTransform/adapter.log
    #-------------------------------------------
    
    
    
    */5 * * * * ingestion php /opt/sevone-xstats/GenericCSVTransform/bin/console.php acquire-data --source 2 &>> /var/log/xstats/GenericCSVTransform/adapter.log
    #-------------------------------------------
    
    
    0 0 * * * ingestion php /opt/sevone-xstats/GenericCSVTransform/bin/console.php cleanup --source 2 &>> /var/log/xstats/GenericCSVTransform/adapter.log
    #-------------------------------------------
    
    
    */5 * * * * ingestion php /opt/sevone-xstats/GenericCSVTransform/bin/console.php acquire-data --source 3 &>> /var/log/xstats/GenericCSVTransform/adapter.log
    #-------------------------------------------
    
    
    0 0 * * * ingestion php /opt/sevone-xstats/GenericCSVTransform/bin/console.php cleanup --source 3 &>> /var/log/xstats/GenericCSVTransform/adapter.log
    #-------------------------------------------
    
    The cron job for each source is scheduled to run every 5 minutes. You will start to see files in /opt/sevone-xstats/GenericCSVTransform/sources/source_<id>/data/. If you do not want to wait for the cron job to run, you may run the collector manually for each source by entering the following commands.

    Example: Source 1
    
    php /opt/sevone-xstats/GenericCSVTransform/bin/console.php acquire-data --source 1
    

    Run the following command to check that there are files in the data directory for Source 1.
    
    ls -la /opt/sevone-xstats/GenericCSVTransform/sources/source_1/data/
    


    Example: Source 2
    
    php /opt/sevone-xstats/GenericCSVTransform/bin/console.php acquire-data --source 2 
    

    Run the following command to check that there are files in the data directory for Source 2.
    
    ls -la /opt/sevone-xstats/GenericCSVTransform/sources/source_2/data/
    


    Example: Source 3
    
    php /opt/sevone-xstats/GenericCSVTransform/bin/console.php acquire-data --source 3 
    

    Run the following command to check that there are files in the data directory for Source 3.
    
    ls -la /opt/sevone-xstats/GenericCSVTransform/sources/source_3/data/
    
  3. Using a web browser of your choice, log into SevOne NMS.
  4. For the three sources added, you will see the devices from Administration > Monitoring Configuration > xStats Source Manager.
  5. To view the objects for the CSV adapter, from Devices, select Object Manager.
  6. On the Filter Options pop-up, click the Plugin drop-down and select xStats.
  7. Click Object Type drop-down and select all CSV object types or you may select only the object types for a specific source you want to view.
    xStatsCSVObjectManagerFilterOptions
  8. Click Apply to view the list of CSV objects.
    xStatsCSVObjectManager

Note: To view the cronfile for the simulator, enter the following command.

cat /etc/cron.d/xstats-GenericCSVTransformSimulator

Output:
# Sevone xstats adapter crontab file.
# Global variables
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
MAILTO=""
HOME=/
# crontab format:
# <minute> <hour> <day of the month> <month> <day of the week> <user> <what to do>
*/5 * * * * root php /opt/sevone-xstats/GenericCSVTransform/simulator/simulator.php >> /var/10g/xstats/GenericCSVTransform/simulator.109 2>&1
#end of simulator cron[root@quality532blank:/]

➤   Stop / Delete Simulator Data

From your ssh client, enter the following command to delete the simulator cron files.

rm /etc/cron.d/xstats-GenericCSVTransformSimulator

To delete the .json configuration files and data files for each source, run the following command. This includes the .json files created when each source was added.

Example: Source 1

rm -r /opt/sevone-xstats/GenericCSVTransform/sources/source_1/


Example: Source 2

rm -r /opt/sevone-xstats/GenericCSVTransform/sources/source_2/


Example: Source 3

rm -r /opt/sevone-xstats/GenericCSVTransform/sources/source_3/


Enter the following command to delete the simulator data output files. This applies to all three sources that were added for the simulator.

Example

rm -r /opt/sevone-xstats/GenericCSVTransform/simulator/simulatorData/

Perform the following actions in SevOne NMS to delete the devices.
  1. From the navigation bar, Administration > Monitoring Configuration > select xStats Source Manager, you will see the list of devices associated with each source added using xStats Source Manager. Locate the xStats collection source for each source and click the link under column Devices.
  2. From navigation bar, Devices > select Device Manager.
  3. Select the check box for each device associated with the source(s) you want to delete.
  4. At the top of the page, click Move selected to Deletion Queue to move the device to deletion queue.
    xStatsCSVDeviceManagerDeletion

➤   Prepare .json Configuration Files

Create directory where you want to save the .json files. For example,

cd /opt/sevone-xstats/GenericCsvTransform

mkdir prepared_json_files

cd prepared_json_files

mkdir source_1

cd source_1
You are now in directory /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1
  1. Prepare collector configuration file.
    1. Copy /opt/sevone-xstats/GenericCSVTransform/config/collector.sample.json file to directory /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1.
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/collector.sample.json \
      /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1/<filename>.json
      
      For example,
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/collector.sample.json \
      /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/collector_source1.json
      
  2. Prepare parser configuration file.
    1. Copy /opt/sevone-xstats/GenericCSVTransform/config/parser.sample.json file to directory /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1.
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/parser.sample.json \
      /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1/<filename>.json
      
      For example,
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/parser.sample.json \
      /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/parser_source1.json
      
  3. Prepare object types and indicator types configuration file.
    1. Copy /opt/sevone-xstats/GenericCSVTransform/config/object_types.sample.json file to directory /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1.
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/object_types.sample.json \
      /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1/<filename>.json
      
      For example,
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/object_types.sample.json \
      /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/object_types_source1.json
      
  4. Prepare metadata configuration file.
    1. Copy /opt/sevone-xstats/GenericCSVTransform/config/metadata.sample.json file to directory /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1.
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/parser.sample.json \
      /opt/sevone-xstats/GenericCsvTransform/prepared_json_files/source_1/<filename>.json
      
      For example,
      
      cp /opt/sevone-xstats/GenericCSVTransform/config/metadata.sample.json \
      /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/metadata_source1.json
      

Using a text editor of your choice, edit each .json file to modify the settings based on your configuration. The details on the configuration variables for each .json file can be found in the following tables under section APPENDIX.
  1. Collector - ➤   Collector .json File
  2. Parser - ➤   Parser .json File
  3. Object Types & Indicator Types - ➤   Object Type & Indicator Type .json File
  4. Metadata - ➤   Metadata Configuration .json File
After updating the .json file, save it.

Add a Source using prepared .json Configuration Files

When your three .json files are prepared, you are now ready to add your new source. The following command to add a source has four main components.
  • The name of the source.
  • A .json file with settings for configuring the collector.
  • A .json file with settings for configuring the parser.
  • A .json file with settings for configuring the object types and indicator types.

php /opt/sevone-xstats/GenericCSVTransform/bin/console.php configure \
--sourceName GenericCSV_Source1 \
--file /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/collector_source1.json \
--parserConfig /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/parser_source1.json \
--objectIndicatorTypes \
/opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/object_types_source1.json

Output:
objectIndicatorTypes is validated!!!
parserConfig is a valid parser configuration!!!
Source name can be created.
Configuration file seems OK.
Crontab for collection can be created.
Logs and rotation can be created.
Adding source to database.
Source has been inserted into database.
Collector configuration saved to: /opt/sevone-xstats/GenericCSVTransform/sources/source_1/config/collector.json.
Collector crons have been configured.
Log dir and rotations have been configured.
Configuration was successful. objectIndicatorTypes are installed!!! parserConfig is installed!!!
From Administration > Monitoring Configuration > xStats Source Manager, you will see that the new source has been added successfully.

➤   Summary on how to Install CSV Adapter

The details in this section show how to install the adapter using the simulator and without using the simulator.

with Simulator

If you plan to run the simulator before using the CSV adapter in a production environment, follow the steps below.

  1. Upload the .spk package to your PAS.
  2. Extract files from the .spk package.
  3. Install the adapter.
  4. (optional) Add the SFTP source type. This applies only if you plan to use SFTP (as opposed to the local file system).
  5. Start the simulator.
  6. Add the sources for the simulator.
  7. In SevOne NMS, from Administration > Monitoring Configuration > xStats Source Manager, confirm that the sources have been added.
  8. Verify that the cron file for the adapter has been created and, optionally, view contents of the cron file.
  9. (optional) Run the collector for each source manually. This is useful if you do not want to wait for the cron job to run on its own.
  10. In SevOne NMS, from Administration > Monitoring Configuration > xStats Source Manager, confirm that devices are showing up for the sources you have added.
  11. In SevOne NMS, from Devices > Object Manager, confirm that objects exist.
  12. Stop the simulator and delete simulator data.
  13. Add one or more sources for use in your production environment. You will need to have the three preconfigured .json files for each source you add i.e., for collector, parser, and object/indicator types.
  14. In SevOne NMS, from Administration > Monitoring Configuration > xStats Source Manager, confirm that the sources have been added.
  15. Configure the metadata.json file for each source you have added and copy it to /sources/source_<id>/.
  16. (optional) Run the collector for a source manually. This is useful if you do not want to wait for the cron job to run on its own.
  17. In SevOne NMS, from Administration > Monitoring Configuration > xStats Source Manager, confirm that devices are showing up for the sources you have added.
  18. In SevOne NMS, from Devices > Object Manager to confirm that objects exist.

without Simulator

If you plan to use the adapter in production environment without first running the simulator, follow the steps below.

  1. Upload the .spk package to your PAS.
  2. Extract files from the .spk package.
  3. Install the adapter.
  4. (optional) Add the SFTP source type. This applies only if you plan to use SFTP (as opposed to the local file system).
  5. Add one or more sources for use in a production environment. You will need to have the three preconfigured .json files for each source you add i.e., for collector, parser, and object/indicator types.
  6. In SevOne NMS, from Administration > Monitoring Configuration > xStats Source Manager, confirm that the sources have been added.
  7. Configure the metadata.json file for each source you have added and copy it to /sources/source_<id>/.
  8. Verify that the cron file for the adapter has been created and, optionally, view contents of the cron file.
  9. (optional) Run the collector for a source manually. This is useful if you do not want to wait for the cron job to run on its own.
  10. In SevOne NMS, from Administration > Monitoring Configuration > xStats Source Manager, confirm that devices are showing up for the sources you have added.
  11. In SevOne NMS, from Devices > Object Manager, confirm that objects exist.

FAQs

Important: Run the commands below from the NMS container.

➤   How can I stop the simulator?

Using ssh, log into SevOne NMS appliance as support.


ssh support@<NMS appliance>
  1. Get a process ID (PID) for the simulator.
    ps -ef | grep simulator
    
    
    
    Output:
    root 5993 4748 0 06:29 pts/5 00:00:00 grep --color=auto simulator
    root 19058 1 0 Aug20 ? 00:00:00 SCREEN -dm php -S 0.0.0.0:8100 simulator.php
    root 19059 19058 0 Aug20 pts/0 00:00:43 php -S 0.0.0.0:8100 simulator.php
    root 24890 1 0 Aug20 ? 00:00:00 SCREEN -dm php -S 127.0.0.1:8081 /opt/sevone-xstats/GenericCSVTransform/simulator/simulator.php
    root 24891 24890 0 Aug20 pts/2 00:00:13 php -S 127.0.0.1:8081 /opt/sevone-xstats/GenericCSVTransform/simulator/simulator.php
    
  2. Choose a PID to use. In the output for the command above, there are two listings for /opt/sevone-xstats/GenericCSVTransform/simulator/simulator.php. You may use either PID.
  3. Enter the following command to stop the simulator.
    kill <PID>

    Example: Kill PID

    kill 24891
  4. Confirm that the simulator has stopped running by executing the grep command again. You should no longer see the two entries for GenericCSVTransform simulator.php.
    ps -ef | grep simulator
    
    
    Output:
    root 5993 4748 0 06:29 pts/5 00:00:00 grep --color=auto simulator
    root 19058 1 0 Sep30 ? 00:00:00 SCREEN -dm php -S 0.0.0.0:8100 simulator.php
    root 19059 19058 0 Sep30 pts/0 00:00:43 php -S 0.0.0.0:8100 simulator.php

➤   How can I delete the xStats Collection Source?

Perform the following steps to delete the Generic CSV adapter xStats collection source from SevOne NMS. Make sure to perform the steps in the order listed below.

  1. Delete the cron file for the Generic CSV adapter.
    rm -r /config/cron.d/xstats-GenericCSVTransform
  2. Delete the xStats collection sourcein SevOne NMS. From the navigation bar, go to Administration > Monitoring Configuration > xStats Source Manager. Select the source you want to delete and click trashCanIcon to delete it.
    xStatsCSVDeleteSource

    You will get the following pop-up.
    xStatsCSVDeleteOneSource If you want to delete more than one source, you may select the check box of the sources you want to delete. Click gearIcon and select Delete Selected to delete all the selected sources you want deleted.

    You will get the following pop-up.
    xStatsCSVDeleteMultipleSources
    Note: You will need to delete the devices for this xStats collection source. To view the devices before deleting the xStats collection source, locate GenericCSVTransform in xStats Source Manager and click the link under Devices. This will bring up a list of the devices that need to be deleted.
  3. Delete the log files.
    rm -r /config/cron.d/xstats-GenericCSVTransform
  4. Delete the object types in SevOne NMS.
    1. From the navigation bar, go to Administration > Monitoring Configuration > Object Types.
    2. To delete the object types for the Generic CSV adapter, place the cursor on a Generic CSV adapter object type in the left-panel and click trashCanIcon to delete it.
      Note: To display a list of all the object types (and indicator types) associated with this xStats collection source, SSH into your PAS and enter the following command.
      more /opt/sevone-xstats/GenericCSVTransform/config/object_types.json
  5. Delete the devices in SevOne NMS. Please refer to section on ➤   How can I delete the device(s) added by the Generic xStats Adapter?.
    1. From the navigation bar, go to Devices > Device Manager.
    2. To delete the devices for the Generic CSV adapter, select one or more devices and click Move selected to Deletion Queue.
  6. Delete the /opt/sevone-xstats/GenericCSVTransform directory.
    rm -r /opt/sevone-xstats/GenericCSVTransform/

➤   How can I delete the device(s) added by the Generic xStats Adapter?

Perform the following actions in SevOne NMS to delete the devices.

  1. You can view a list of the devices associated with the simulator using the xStats Source Manager. From the navigation bar, go to Administration, select Monitoring Configuration > xStats Source Manager. Select the xStats source under which you want to delete the device(s) it has and click xStatsSourceDeleteDevices. Or, click the link under Devices for the xStats source under which you want to delete the device(s). This will bring up a list of the devices that need to be deleted..
  2. You will get a pop-up with the list of devices it contains. You may choose the device(s) to delete.
    xStatsCSVChooseDevicesToDelete.
  3. From the navigation bar, go to Devices and select Device Manager.
  4. Select the check box for each device associated with the simulator to delete and click Move selected to Deletion Queue.
    xStatsCSVMoveDeviceToDeletion

    Note: In addition to stopping the simulator, you will need to delete the collector profile for the simulator found in /opt/sevone-xstats/GenericCSVTransform/config.

➤   How can I check the xStats Adapter Version?

Execute the following command to verify the version of the Generic CSV adapter.

Example

php /opt/sevone-xstats/GenericCSVTransform/bin/console.php show-version --gitHash

Output:
Version: 7.1.0
Git hash: 2a6fb977f4c783f5be51e3df0363426ebdca6304

➤   How can I check the Collector and Parser status?

You can see the status of the collector or the parser by looking at the heartbeat file associated with it.

  1. To check the collector status, enter the following command to view the collector heartbeat file.
    cat /var/log/xstats/GenericCSVTransform/collector_heartbeat.csv
    
    
    Output:
    status,timestamp,collector,collection_source,transfer_time,data_volume
    200,2024-08-20T14:15:15+0000,GenericCSVTransform,2::127.0.0.1,1470,16440
    200,2024-08-20T14:15:15+0000,GenericCSVTransform,3::127.0.0.1,1847,36324
    200,2024-08-20T14:15:16+0000,GenericCSVTransform,1::127.0.0.1,1249,11352
    200,2024-08-20T14:20:16+0000,GenericCSVTransform,1::127.0.0.1,704,0
    200,2024-08-20T14:20:16+0000,GenericCSVTransform,2::127.0.0.1,486,0
    200,2024-08-20T14:20:16+0000,GenericCSVTransform,3::127.0.0.1,534,0
    200,2024-08-20T14:25:14+0000,GenericCSVTransform,3::127.0.0.1,1422,3026
    200,2024-08-20T14:25:14+0000,GenericCSVTransform,2::127.0.0.1,957,1368
    200,2024-08-20T14:25:14+0000,GenericCSVTransform,1::127.0.0.1,630,944
    200,2024-08-20T14:30:15+0000,GenericCSVTransform,2::127.0.0.1,2736,2736
    200,2024-08-20T14:30:17+0000,GenericCSVTransform,1::127.0.0.1,1861,1888
    200,2024-08-20T14:30:19+0000,GenericCSVTransform,3::127.0.0.1,982,6052
    200,2024-08-20T14:35:13+0000,GenericCSVTransform,2::127.0.0.1,730,0
    200,2024-08-20T14:35:13+0000,GenericCSVTransform,1::127.0.0.1,664,0
    200,2024-08-20T14:35:13+0000,GenericCSVTransform,3::127.0.0.1,599,0
    200,2024-08-20T14:40:20+0000,GenericCSVTransform,2::127.0.0.1,1249,1372
    200,2024-08-20T14:40:20+0000,GenericCSVTransform,1::127.0.0.1,1627,948
    ...
    ...

    The collector heartbeat file for each adapter provides the following information.

    • status - HTTP-like status code. 200 indicates that the request has succeeded, while 500 indicates an error.
    • timestamp - Timestamp marking the last time the collector was invoked.
    • package - Name of the collector.
    • collection_source - The name of the xStats collection source.
    • transfer_time - The amount of time that it took to retrieve files or objects from the xStats collection source.
    • data_volume - The amount of data transferred.
  2. To check the parser status, enter the following command to view the parser heartbeat file.
    cat /var/log/xstats/GenericCSVTransform/parser_heartbeat.csv
    
    
    Output:
    status,timestamp,package,parser,parse_time,total_parsed_entries,parsed_file
    200,2024-08-20T14:15:18+0000,GenericCSVTransform,GenericCSVSimulator_Profile3,14,192,Device_30__1724163187.csv
    200,2024-08-20T14:15:18+0000,GenericCSVTransform,GenericCSVSimulator_Profile2,11,240,Device2_0__1724163187.csv
    200,2024-08-20T14:15:18+0000,GenericCSVTransform,GenericCSVSimulator_Profile1,13,336,Device1_0__1724163187.csv
    200,2024-08-20T14:15:19+0000,GenericCSVTransform,GenericCSVSimulator_Profile3,13,192,Device_31__1724163187.csv
    200,2024-08-20T14:15:19+0000,GenericCSVTransform,GenericCSVSimulator_Profile1,10,336,Device1_1__1724163187.csv
    200,2024-08-20T14:15:19+0000,GenericCSVTransform,GenericCSVSimulator_Profile2,11,240,Device2_1__1724163187.csv
    200,2024-08-20T14:25:16+0000,GenericCSVTransform,GenericCSVSimulator_Profile3,19,16,Device_30__1724163785.csv
    200,2024-08-20T14:25:16+0000,GenericCSVTransform,GenericCSVSimulator_Profile1,7,28,Device1_0__1724163785.csv
    200,2024-08-20T14:25:16+0000,GenericCSVTransform,GenericCSVSimulator_Profile2,8,20,Device2_0__1724163785.csv
    200,2024-08-20T14:25:17+0000,GenericCSVTransform,GenericCSVSimulator_Profile2,6,20,Device2_1__1724163785.csv
    200,2024-08-20T14:25:17+0000,GenericCSVTransform,GenericCSVSimulator_Profile1,7,28,Device1_1__1724163785.csv
    200,2024-08-20T14:25:17+0000,GenericCSVTransform,GenericCSVSimulator_Profile3,7,16,Device_31__1724163785.csv
    200,2024-08-20T14:30:20+0000,GenericCSVTransform,GenericCSVSimulator_Profile1,5,56,Device1_0__1724164087.csv
    200,2024-08-20T14:30:21+0000,GenericCSVTransform,GenericCSVSimulator_Profile2,7,40,Device2_0__1724164087.csv
    200,2024-08-20T14:30:21+0000,GenericCSVTransform,GenericCSVSimulator_Profile2,10,40,Device2_1__1724164087.csv
    200,2024-08-20T14:30:21+0000,GenericCSVTransform,GenericCSVSimulator_Profile3,8,32,Device_30__1724164087.csv
    200,2024-08-20T14:30:22+0000,GenericCSVTransform,GenericCSVSimulator_Profile1,5,56,Device1_1__1724164087.csv
    200,2024-08-20T14:30:22+0000,GenericCSVTransform,GenericCSVSimulator_Profile3,9,32,Device_31__1724164087.csv
    200,2024-08-20T14:40:23+0000,GenericCSVTransform,GenericCSVSimulator_Profile2,6,20,Device2_0__1724164686.csv
    ...
    ...

    The parser heartbeat file for each adapter provides the following information

    • status - HTTP-like status code. 200 indicates that the request has succeeded, while 500 indicates an error.
    • timestamp - Timestamp when the parser was invoked.
    • package - Name of the xStats adapter.
    • parser - Name of the parser.
    • parse_time - Amount of time to parse.
    • total_parsed_entries - Number of times the indicator values were translated and sent for ingestion. This excludes ignored and unknown values.
    • parsed_file - Name of the parsed file.

➤   What if I am using a Jump Box?

If you are using a jump box, you will need to manually install the collector on the jump box and connect to the SevOne appliance via SFTP.

➤   HSA Installation

To install the Generic adapters on HSA pair, please follow the steps below.

  1. Copy the installation package (.phar.spk) for each adapter to active and passive appliances.
  2. Using ssh, log into the active and passive appliances as support.
  3. Execute the following commands on both appliances.
    php IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-<version>.phar.spk
    

    Example

    php IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-v7.1.0.phar.spk
    
  4. The adapters must be installed on the active appliance only. If you try to install the adapters on the passive appliance, you will receive an error message stating that the appliance is not active and therefore, the process cannot continue.
    Note: Even though you will not be installing the adapters on the passive appliance, the xStats collection sources for the adapters will appear in the xStats Source Manager (Administration > Monitoring Configuration > xStats Source Manager) on the passive appliance, just as they do on the active appliance, as a result of database replication.
  5. Configure the collector for each adapter on both the active and the passive appliances. Also, make sure to configure the collector exactly the same on both appliances.
    Note: For the Generic CSV adapter, using the collector is optional.
  6. For the Generic CSV adapter, you will need to configure a service, such as SFTP or SCP, to push files to the active appliance in an HSA pair.
    Note: The IP address must be the IP address of the HSA pair - not the IP address of an individual PAS.

    SCP command to copy the .csv files from the remote server to the /data directory on the active appliance, using the IP address of the HSA pair

    scp /<full_path_to_Generic_CSV_files>/*.csv \
    <user_name>@<IP_address_of_HSA_pair>:/opt/sevone-xstats/GenericCSVTransform/sources/source_<##>/data
    

    Example: SCP command

    scp /Generic/CSV-files/*.csv \
    root@192.168.136.224:/opt/sevone-xstats/GenericCSVTransform/sources/source_1/data

➤   Multiple Peers Installation

Installing the Generic adapter on multiple peers in a cluster does not require anything special. You will install the adapters on each relevant peer in the same way.

The adapters recognize the Peer ID and this ID is used to name the xStats collection sources. For example,

  • If you install your Generic CSV adapter package on a peer with an ID of 1, the xStats collection source for the Generic CSV adapter will be named as GenericCSVTransform_1.
  • If you install the Generic CSV adapter on another peer in the same cluster, and that peer has an ID of 5, the xStats collection source for the Generic CSV adapter will be GenericCSVTransform_5.

Example: Install Generic Adapter on Multiple Peers

In this example, Generic adapter is installed on two peers in a cluster and each peer has an HSA setup. Execute the steps below.

On Peer 1
For adapter, Generic CSV, on the 'active' appliance

  1. For each adapter, execute the following commands.
    php IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-<version>.phar.spk
  2. Install each adapter.
  3. Configure the collector for each adapter.
    Note: The collector is optional for the Generic CSV adapter.

For adapter, Generic CSV, on the 'passive' appliance

  1. For each adapter, execute the following commands.
    php IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-<version>.phar.spk
    
  2. Configure the collector for each adapter.
    Note:
    1. The collector is optional for the Generic CSV adapter.
    2. The exact same specifications used to configure the collector (for each adapter) on the active appliance must be used on the passive appliance.

On Peer 2
For adapter, Generic CSV, on the 'active' appliance

  1. For each adapter, execute the following commands.
    php IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-<version>.phar.spk
    
  2. Install each adapter.
  3. Configure the collector for each adapter.
    Note: The collector is optional for the Generic CSV adapter.

For adapter, Generic CSV, on the 'passive' appliance

  1. For each adapter, execute the following commands.
    php IBM-SevOne-NPM-Solution-xStats-GenericCSVTransform-<version>.phar.spk
    
  2. Configure the collector for each adapter.
    Note:
    1. The collector is optional for the Generic CSV adapter.
    2. The exact same specifications used to configure the collector (for each adapter) on the active appliance must be used on the passive appliance.

TROUBLESHOOTING

Important: Run the commands below from the NMS container.

➤   Collector Issues

Unable to connect to remote device - SFTP issue

Most collectors require SFTP to be enabled. If you are having trouble connecting to the remote device that you are collecting data files from, it is possible that SFTP needs to be enabled on that device. One way to test this is to simply try connecting to the remote device with SFTP. If you are unable to connect via SFTP, it could mean that the SFTP server is not running on the remote device. In this case, you will need to enable the SFTP server. If your collector requires SFTP to be enabled on the remote device, this will likely resolve the issue.

Unable to connect to remote device - Authentication issue

The collector needs to successfully authenticate to the remote device before it can collect data files. If you are having trouble authenticating, it could be a result of one of the following problems.

  • The username/password combination is incorrect.
  • The password on the remote device has expired.

➤   xStats Issues

Package does not install correctly

If you encounter errors during the package installation, please report them to Expert Labs.

xStats collection source has been created, but no devices are showing up

Devices will show up only after the collector has delivered new files to the retrieval directory. If your collector runs every thirty minutes, for example, you will have to wait a bit. If there are files already in the local directory, it is possible that fcad may not pick them up immediately. In this case, you can use the touch command to trick fcad into thinking that those files just arrived.

Go to the /data directory for the Generic CSV adapter and type touch [filename], then hit Enter. After that, refresh the xStats Source Manager page. One or more devices should show up then.

Note: Generic CSV xStats Adapter: Default retrieval directories
  • CSV adapter: /opt/sevone-xstats/GenericCSVTransform/sources/source_<##>/data

Devices still do not show up, even after you have touched the files

From the xStats Source Manager, edit the xStats collection source for the adapter that does not have devices showing up and make sure that the retrieval directory is correct. The default retrieval directory for the Generic xStats adapter is:

  • CSV adapter: /opt/sevone-xstats/GenericCSVTransform/sources/source_<##>/data

If everything checks out, there may be a problem with the package that you installed. Contact the Expert Labs for resolution.

Devices show up, but no objects appear for them

When objects fail to show up, it is often related to the discovery process. The following steps will walk you through the discovery process in SevOne NMS.

  1. Go to Devices > Device Manager.
    xStatsCSVDeviceDiscovery

  2. Select the device(s)in question. Once you have made your selection, click gearIcon and select Discovery. Then select Discover. Give Discovery a chance to run its course. Click refresh to refresh the page. Objects should start showing up.

Objects and indicators show up for a device, but no data appears in the graphs

This is likely a backfill problem, which can result from running an adapter for the first time with a large number of data files. Data should start showing up in a bit. Sometimes it can take up to an hour in this case.

➤   Simulator Issues

If you are not getting any data after running the simulator, please run the following command.

SevOne-act check xstats-permissions
This command checks for various permission issues with xStats on the local peer. If you encounter any permission issues, please contact IBM SevOne Support.

APPENDIX

Using a text editor of your choice, edit each .json file to modify the settings based on your configuration and then, save the .json file.

➤   Collector .json File

Example: Collector .json file


cat /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/collector_source1.json

Output:
{
    "adapterName": "GenericCSVTransform",
    "logFile": "/var/log/xstats/GenericCSVTransform/collector.log",
    "heartbeatFile": "/var/log/xstats/GenericCSVTransform/collector_heartbeat.csv",
    "lastPollRepository": "localfilesystem",
    "os": "ubuntu",
    "sourceType": "real",
    "debug": true,
    "username": "root",
    "password": "",
    "publicKey": "/opt/sevone-xstats/GenericCSVTransform/simulator/id_rsa.pub",
    "privateKey": "/opt/sevone-xstats/GenericCSVTransform/simulator/id_rsa",
    "passphrase": "",
    "remoteHostname": "127.0.0.1",
    "remoteDir": "/opt/sevone-xstats/GenericCSVTransform/simulator/simulatorData/output-dir-1",
    "port": 22,
    "thresholdDays": 2,
    "serviceType": "scp",
    "cutoff": "2015-01-01T00:00:00",
    "regex": "/\\.*/",
    "dateStringFormat": "Ymd",
    "dateTimezone": "UTC",
    "dateRegex": "",
    "run": true,
    "fullScan": true
}
Variable Description
lastPollRepository Specify one of the following.
  • localfilesystem - if the collector is installed on a jump box.
  • database - if the collector is installed on SevOne primary and secondary HSA appliances.
sourceType Specify one of the following.
  • real - if you are connecting directly to the remote server.
  • jumpbox - if you are connecting to the remote server via a jump box.
debug Set to true to store all debug information in the log file. Otherwise, set it to false.
os Specify the operating system of the remote server.
  • Ubuntu
  • Freebsd
  • SunOS
  • Solaris
  • Linux
  • CentOS
  • Gentoo
  • Other
thresholdDays The maximum number of days old that a file can be and still be eligible for download. Files older than the number of days specified in this field cannot be downloaded.
cutoff The cut-off date for files to be downloaded. Files older than this date will not be downloaded. Please use yyyy-mm-ddThh:mm:ss format.
remoteDir The directory on the remote device that will be used to collect the files from.
remoteHostname The hostname or IP address of the remote device.
serviceType The service used for downloading files from the server. i.e., scp, sftp, or ftp.
port The port to connect to on the remote device.
username The user name to authenticate to the remote device.
password The password to authenticate to the remote device.
publicKey The public key used to authenticate to the remote device. This is used if the password field is empty.
privateKey The private key used to authenticate to the remote device. This is used if the password field is empty.
passphrase The passphrase for the private key (if the paraphrase has been set up).
regex A regular expression that files need to match in order to be downloaded. The format should be /regex/. For example, /.csv/ specifies that .csv files should be downloaded.
fullscan Specify one of the following.
  • true - to collect files from the remote directory and its sub-directories.
  • false - to collect files only from the remote directory.
dateStringFormat The format of the timestamp in the file names.
dataTimezone The timezone that applies to the timestamp of the files names.
dateRegex The regular expression for the file's timestamp.
run Enter true to collect data else, enter false. If you enter false, you can stop collecting files for the source without having to remove the source configuration. Setting run to false, disables the source.
(optional) adapterName The name of the adapter. Default setting is recommended. Please contact Expert Labs to change the setting.
(optional) logFile The name of the adapter log file including the full path to the file. Default setting is recommended. Please contact Expert Labs to change the setting.
(optional) heartbeatFile The name of the collector heartbeat file including the full path to the file. Default setting is recommended. Please contact Expert Labs to change the setting.
If required, using a text editor of your choice, edit and save /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/collector_source1.json file.

Copy the file to /sources/source_<id>/config folder.

cp /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/collector_source1.json /opt/sevone-xstats/GenericCSVTransform/sources/source_1/config/

➤   Parser .json File

Example: Parser .json file


cat /opt/sevone-xstats/GeneicCSVTransform/prepared_json_files/source_1/parser_source1.json

Output:
{
    "delimiter": ",",
    "additionalDelimiter": "",
    "maxColumns": 60,
    "maxCharsOnLine": 2048,
    "stringEnclosure": "\"",
    "typeMatchString": "%Object Type%",
    "columns": {
        "Date": 3,
        "Device Name": 0,
        "Object Type": 2,
        "Object Name": 1
    },
    "fileNameRegex": "",
    "timestampRound": 0,
    "device": {
        "name": "%Device Name%",
        "type": "GenericCSVDevice_Test"
    },
    "object": {
        "name": "%Object Name%",
        "type": "%Object Type%",
        "class": "GenericCSV_Test"
    },
    "timestamp": "%Date%",
    "timestampFormat": "Y-m-d H:i:s",
    "timezone": "UTC",
    "timestampRound": 300,
    "metadata": {
        "device": {
            "attr1": "%0%",
            "attr2": "%1%"
        },
        "object": {
            "attr3": "%2%",
            "attr4": "%3%"
        },

    },
    "rowTypes": {
        "Profile1ObjectType_11": {
            "indicators": {
                "IndicatorEpoch1": 4,
                "IndicatorSecondOfMinute1": 5,
                "IndicatorMinuteOfHour1": 6,
                "IndicatorHourOfDay1": 7,
                "IndicatorUnitA1": 8,
                "IndicatorUnitB1": 9,
                "IndicatorUnitC1": 10,
                "IndicatorRoundedSine1": 11
            }
        },
        "Profile1ObjectType_12": {
            "indicators": {
                "IndicatorEpoch1": 4,
                "IndicatorSecondOfMinute1": 5,
                "IndicatorMinuteOfHour1": 6,
                "IndicatorHourOfDay1": 7
            }
        }
    }
}
Variable Description
delimiter The delimiter that is used to separate columns. The default value is ,.
additionalDelimiter An additional delimiter if there is one. Only one additional delimiter is supported.
maxColumns The maximum number of columns in a row. Rows with more columns than the specified maximum will not be parsed. The default value is 60.
maxCharsOnLine The maximum number of characters in a row. Rows with more characters than the specified maximum will not be parsed. The default value is 2048 characters.
typeMatchString The template that will be used to match the row type.
stringEnclosure The character that will be stripped from the columns. The default values is ".
fileNameRegex The regular expression used to parse the file name. Without a regex here, it is not possible to use the file name in the templates.
headers The number of headers in the file. The default value is 0. If you set it to 2, the first two rows in the file will be read as headers. The data in the headers is used as, %headers[<headerIndex>][<columnIndex>]%.
columns The list of columns with indexes that can be used in the templates.
devicePrefix The prefix for names of devices associated with the source that you are adding. If you do not provide input here, no prefix will be included in the device names.
  • device
    • name - The name of the device.
    • type - The device type.
  • object
    • name - The name of the object.
    • type - The object type.
    • class - The object class.
timestamp The template for the timestamp.
timestampFormat The format for the timestamp. If this is missing, a unix timestamp format will be used.
timezone The timezone that the data is in. If no input is provided, this setting will default to UTC. http://php.net/manual/en/timezones.php provides the list of timezones.
(optional) timestampRound This setting is used to align timestamps in the file. This is useful when there is little difference in timestamps; it allows minor differences to be ignored. However, using this option can result in values being overwritten. If you need all timestamps in SevOne NMS, do not use this setting.
metadata This is for metadata information which includes attributes for devices and objects. The setting allows you to specify which data you want to be injested as SevOne NMS metadata from your .csv files.
Any metadata information added here must correspond to the metadata information added in the metadata configuration file, metadata.json. Please refer to ➤   Metadata Configuration .json File.
  • device - For each device attribute that you want to include, provide either the column number or column name within quotes following attr1, etc. In the screenshot above, the value for attr1 is %0%. In this case, the column being referred to is Column 0. Columns start with 0, meaning that this would be the first column from the sample .csv file. Accordingly, the next column to the right would be Column 1, which would be indicated here by %1%. If you choose to use a column name, on the other hand, and the name of the column is Device Name, for example, then you would enter %Device Name%.
    • attr1 - The column number or name that you want to include.
    • attr2 - The column number or name that you want to include.
  • object - For each object attribute that you want to include, provide either the column number or column name within quotes following attr3, etc. In the screenshot above, the value for attr3 is %2%. In this case, the column being referred to is Column 2. Columns start with 0, meaning that %2% would be the third column from the sample .csv file. If you choose to use a column name, on the other hand, and the name of the column is Object Name, for example, then you would enter %Object Name%.
    • attr3 - The column number or name that you want to include.
    • attr4 - The column number or name that you want to include.
rowTypes Specify parsing configuration for each separate row that you want to support. f all rows are of the same type, you can use * to set the configuration for all row types. For each row type, you can specify the following variables.
  • columns - The list of columns with indexes that can be used in the templates. Each row type can have different indexes for the columns.
  • device
    • name - The name of the device.
    • type - The device type.
  • object
    • name - The name of the object.
    • type - The object type.
    • class - The object class.
  • timestamp - The template for the timestamp.
  • timestampFormat - The format for the timestamp. If this is missing, a unix timestamp format will be used.
  • metadata - Metadata information. This can include attributes for devices and objects. These settings allow you to specify which data from your .csv files that you want to be ingested as SevOne NMS metadata. See the previous metadata entry above for detailed information. Any metadata information that you enter here, under rowTypes, will apply only to the row type.
  • (optional)skip - Row types set to true or 1 will be skipped.
  • indicators - The indicators to use. This should be a list of indicators and columns in the row. In the screenshot above, the indicator name is provided first, followed by the value, which is the column index. The column index is 0-based. You can also specify the index from right to left by using negative values. For example, -1 indicates the first column from the right. The indicators setting, if configured, takes precedence over the repeatableIndicators setting.
  • repeatableIndicators - A setting used to specify repeatable groups of columns. This is useful if you have two or three columns for each indicator and if you have a different number of indicators for each row type. The following are the options that you can configure for this setting.
    • name - The template for building the indicator name.
    • value - The template for building the indicator value. This will be either %valueIdx% or %columnName%. Currently, formulas are not supported here.
    • valueIdx - The offset of the column that contains the indicator value. You can use %valueIdx% for special text.
    • startIdx - The first index indicating where the repeated columns start.
    • toupleLen - The number of columns in a group. For example, if we have the row "Device, Object, Date, Ind1, 1, Ind2, 2, Ind3, 3", the startIdx would be 3 and the toupleLen would be 2.
    • endIdx - The end index of the last column in the last group. If this is omitted, the parser will try to get all of the columns up to the end.
    • columns - A list of columns that you can use in the templates. In this case, the index is actually offset in the group. In the above example, the index for indicator name would be 0 and for indicator value would be 1.
    • namesToSkip - A list of any indicator names that you don't want to emit.
If required, using a text editor of your choice, edit and save /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/parser_source1.json file.

Copy the file to /sources/source_<id>/config folder.

cp /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/parser_source1.json /opt/sevone-xstats/GenericCSVTransform/sources/source_1/config/

➤   Object Type & Indicator Type .json File

Example: Object / Indicator Types .json file


cat /opt/sevone-xstats/GeneicCSVTransform/prepared_json_files/source_1/object_types_source1.json

Output:
{
    "Profile1ObjectType_11": {
        "field_identifiers": "Profile1ObjectType_11",
        "is_ignore": 0,
        "indicators": [
            {
                "indicator": "IndicatorEpoch1",
                "description": "IndicatorEpoch1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorSecondOfMinute1",
                "description": "IndicatorSecondOfMinute1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorMinuteOfHour1",
                "description": "IndicatorMinuteOfHour1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorHourOfDay1",
                "description": "IndicatorHourOfDay1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorUnitA1",
                "description": "IndicatorUnitA1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorUnitB1",
                "description": "IndicatorUnitB1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorUnitC1",
                "description": "IndicatorUnitC1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorRoundedSine1",
                "description": "IndicatorRoundedSine1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            }
        ]
    },
    "Profile1ObjectType_12": {
        "field_identifiers": "Profile1ObjectType_12",
        "is_ignore": 0,
        "indicators": [
            {
                "indicator": "IndicatorEpoch1",
                "description": "IndicatorEpoch1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorSecondOfMinute1",
                "description": "IndicatorSecondOfMinute1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorMinuteOfHour1",
                "description": "IndicatorMinuteOfHour1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            },
            {
                "indicator": "IndicatorHourOfDay1",
                "description": "IndicatorHourOfDay1",
                "type": "GAUGE",
                "percentable": "0",
                "units": "Number",
                "data_units": "Number",
                "is_default": "1",
                "is_ignore": "0",
                "default_value": "",
                "index_in_file": -1
            }
        ]
    }
}
Variable Description
field_indicators The object type.
is_ignore A setting to specify whether the object type should be ignored (hidden) in SevOne NMS.
indicators
  • indicator - The name of the indicator type.
  • description - The name to display.
  • type - The data type. Specify GAUGE, COUNTER32, or COUNTER64.
  • percentable - A setting to specify whether the values in the file represent percentages. 1 indicates that they do represent percentages. 0 indicates that they do not.
  • units - The data unit to display results in.
  • data_units - The data unit to measure in.
  • is_default - A setting to specify whether the indicator should be created for new devices even if it is not emitted.
  • is_ignore - A setting to specify whether the indicator should be ignored (hidden) in SevOne NMS.
  • default_value - The default value to be emitted in the event that the indicator for an object is not emitted.
  • (optional) index_in_file - A setting to display the index of a column (in the data file) that should be used as the indicator value. This setting applies only if you specified useObjectIndicatorTypes under indicators in the parser configuration .json file.
If required, using a text editor of your choice, edit and save /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/object_types_source1.json file.

Copy the file to /sources/source_<id>/config folder.

cp /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/object_types_source1.json /opt/sevone-xstats/GenericCSVTransform/sources/source_1/config/

➤   Metadata Configuration .json File

Example: Metadata .json file


cat /opt/sevone-xstats/GeneicCSVTransform/prepared_json_files/source_1/metadata_source1.json

Output:
{
    "path": "/opt/sevone-xstats/GenericCSVTransform/sources/",
    "namespace": "GenericCSVTransform",
    "device": {
        "filename": "deviceMetadata.json",
        "prefix": "",
        "attributes": {

        }
    },
    "object": {
        "filename": "objectMetadata.json",
        "prefix": "",
        "attributes": {

        }
    }
}
Variable Description
device For each device attribute that you added in the parser config .json file, you will need to add its corresponding attribute type in this section. Possible attribute types include string, ip, integer, mac, dateTime, regex, url, latLong.
  • (optional) prefix - Allows you to add a prefix. For example, if you add csv_ as a prefix here, you will see csv_attr1 and csv_attr2 in SevOne NMS. Prefixes are useful for preventing ambiguity in the case that another xStats adapter has a metadata attribute with the same name but different type.
  • attributes
    • attr1 - The type of the attribute that you added for attr1 in the parser config .json file. For example, if the attribute you added in the parser .json file was a string, then you would enter string within quotes here.
    • attr2 - The type of the attribute that you added for attr2 in the parser config .json file. For example, if the attribute you added in the parser .json file was an integer, then you would enter integer within quotes here.
object For each object attribute that you added in the parser config .json file, you will need to add its corresponding attribute type in this section. Possible attribute types include string, ip, integer, mac, dateTime, regex, url, latLong.
  • (optional) prefix - Allows you to add a prefix. For example, if you add csv_ as a prefix here, you will see csv_attr3 and csv_attr4 in SevOne NMS. Prefixes are useful for preventing ambiguity in the case that another xStats adapter has a metadata attribute with the same name but different type.
  • attributes
    • attr3 - The type of the attribute that you added for attr3 in the parser config .json file. For example, if the attribute you added in the parser .json file was an IP address, then you would enter ip within quotes here.
    • attr4 - The type of the attribute that you added for attr4 in the parser config .json file. For example, if the attribute you added in the parser .json file was for longitude and latitude, then you would enter latLong within quotes here.
The following variables can also be configured but keeping the default values is recommended. Please contact SevOne Support or Expert Labs if want to change the default settings for the following variables.
  • path
  • namespace
  • filename
If required, using a text editor of your choice, edit and save /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/metadata_source1.json file.

Copy the file to /sources/source_<id>/config folder.

cp /opt/sevone-xstats/GenericCSVTransform/prepared_json_files/source_1/metadata_source1.json /opt/sevone-xstats/GenericCSVTransform/sources/source_1/config/