Tutorial: Adding a custom function

In this tutorial, you are writing a custom function that you package and store in a GitHub repository. You are testing the function in your local environment before you register it with the catalog and use it in your calculations.

About this task

Note: To complete this tutorial, you use a sample device type template and data. Sample device types do not use the latest version of device types and do not support streaming data metrics. For more information, see Creating batch or streaming data metrics.

In this tutorial, you are writing a custom function, MultiplyByFactor, that multiples an input item by a factor value. As part of the function code, define how Maximo® Monitor controls for input and output items are represented. Later, specify the input item and the factor value.

Use the execute_local_test method to test the function. You use this method to test any function that is derived from base classes in IoT Functions. The method generates sample data for a function locally. It writes the results to a file named df_test_entitity_for_function_name in the working directory. If the function works, register it with Maximo Monitor.

Finally, create a sample robot device type in Maximo Monitor and apply the new MultiplyByFactor function to the device type.

This tutorial uses the Pycharm IDE to create the Python package for the custom function. If you use another IDE or Watson Studio, make sure to use Python 3.9.x with it.

Alternatively, separate from this tutorial, a custom function starter package is available in IoT functions to help you get started with building your own custom functions.

Before you begin

You can clone a starter package and use it as a sample package in your local environment. Cloning a starter package provides you with an easy way to install the required packages, including IoT Functions, for your custom function. The starter package provides an example of the required directory structure for your custom package.

  1. Install Python 3.9.x in your environment.
  2. Install GIT.
  3. Install pip3 package manager for Python.
  4. Install an editor, such as Xcode, to make code updates.
  5. Clone the HelloWorld starter package
    1. Open a terminal window.
    2. Make a new project folder for the starter package. For example, mkdir project.
    3. Change to the project directory. For example, cd project.
    4. Clone the starter package. Enter: git clone --branch starter_package https://github.com/ibm-watson-iot/functions.git
    5. Verify that you cloned the starter package to the project directory.
    6. Change to the functions directory in the starter package. Enter: cd functions
    7. Enter: git status
    8. Copy the working directory path for later use. Enter pwd and copy the path.
    9. Open the setup.py file. The setup.py file defines the package name and the list of packages that are required for the project.
    10. Set the name parameter to custom{your_initials}. Replace the {your_initials} variable with your initials or with another unique value.
    11. In the ibm-watson-iot/functions repository, verify that the iotfunction package is the correct version. For example, for Maximo Monitor 8.8, the package must be from the 8.8.X GitHub branch. Do not replace the x in the version with another value. The following text is an example setup.py file. The name parameter is set to customId and the 8.7.X branch is used.
#!/usr/bin/env python 
from setuptools import setup, find_packages
setup(name='customld', version='0.0.1', packages=find_packages(),
install_requires=['iotfunctions@git+https://github.com/ibm-watson-iot/functions.git@8.7.x'])
  1. Install the PyCharm community edition.
  2. In PyCharm, point your environment to the starter package you cloned. Click Open and browse to find the directory you cloned.
  3. Set up a new Pycharm virtual environment.
    1. Go to PyCharm > Preferences > Project <your_project> > Project Interpreter.
    2. If you are setting up a new virtual environment:
    3. Click the settings icon and click Add.
    4. Specify the location of your project folder.
    5. Set the base interpreter to your Python 3.9.x directory.
    6. If you have an existing virtual environment, set the project interpreter to Python 3.x.
  4. Run the following command to install the iotfunctions package into your Python environment. Ensure that you use the correct branch name for your version. 
        pip3 install git+https://github.com/ibm-watson-iot/functions.git@8.7.x --upgrade

Make sure that you are always using the latest version of IoT Functions. Check the version of each Python module that is supported by Maximo Monitor. If you do not use the exact version that is supported, your pipeline might break.

Steps

Complete these steps to apply a calculation to sample data by using a custom function.

Step 1: Define your calculation

A company that develops robotic arms is introducing a new robotic arm and is testing that it performs as well as the existing robotic. In early testing, the team discovered that for some of the robots, the distance traveled during testing was too high. After investigation, the team discovered that the tools used to do the testing were adding a delay to the speed and travel times of the robot. The operations manager wants to adjust both values by a factor of 2 but wants the flexibility to change that value later.

The analyst identifies that a similar calculation might be useful in other calculations and generalizes the function by writing a MultiplyByFactor class.

distance * factor

Step 2: Package your function and store it in GitHub.

  1. Create a new repository on GitHub for the starter package you cloned.
    1. The repository must have a URL that is accessible from a pip install.
    2. A best practice is to make the repository private. The repository can be accessed using a personal access token.
  2. Add a file to the repository to create the master branch. For example, add an empty file that is named test.md and click Commit new file.
  3. Open the project in PyCharm locally for editing.
  4. Verify that the project has the following directory structure and files: 
    functions
|
|__ setup.py
|
|__ scripts
  |
  |_ local_test_of_function.py
|
|__ custom
  |
  |_ functions.py
  |_  __init__.py
  5. Change the module name, custom, to a unique name, such as customyourinitials. For example, customld.
  6. Create a multiplybyfactor<yourinitials>.py function module in custom<yourinitials> and paste the following code. Update the variables in <> to match your environment. For example, if your code is in GitHub, you might set PACKAGE_URL to 'git+https://<XXXXXX>@github.com/github.com/jones/starter@starter_package'.

    In this function, you implement two methods. You add your calculation to the execute method. You set the inputs and outputs arguments for the function in the build_ui method. These arguments are configured through the configuration user interface. 

    import inspect
import logging
import datetime as dt
import math
from sqlalchemy.sql.sqltypes import TIMESTAMP,VARCHAR
import numpy as np
import pandas as pd

from iotfunctions.base import BaseTransformer
from iotfunctions import ui

logger = logging.getLogger(__name__)

# Specify the URL to your package here.
# This URL must be accessible via pip install.
# Example assumes the repository is private.
# Replace XXXXXX with your personal access token.
# After @ you must specify a branch.

PACKAGE_URL = 'git+https://XXXXXX@github.com/<user_id><path_to_repository>@starter_package'

class MultiplyByFactor<YourInitials>(BaseTransformer):

    def __init__(self, input_items, factor, output_items):

        self.input_items = input_items
        self.output_items = output_items
        self.factor = float(factor)
        super().__init__()
    def execute(self, df):
        df = df.copy()
        for i,input_item in enumerate(self.input_items):
            df[self.output_items[i]] = df[input_item] * self.factor
        return df

    @classmethod
    def build_ui(cls):
        #define arguments that behave as function inputs
        inputs = []
        inputs.append(ui.UIMultiItem(
                name = 'input_items',
                datatype=float,
                description = "Data items adjust",
                output_item = 'output_items',
                is_output_datatype_derived = True)
                      )        
        inputs.append(ui.UISingle(
                name = 'factor',
                datatype=float)
                      )
        outputs = []
        return (inputs,outputs)

Step 3: Save your credentials to a file

Set credentials to connect to Maximo Monitor.

  1. Download the credentials_as.json file.
  2. Replace the variables with your data and then save the file to your local machine.

The credentials file is used to run or test the function locally. Do not push this file to your external repository in step 4.

Step 4: Push your local changes to your GitHub repository

Verify that you completed these steps:

  • The files that you modified are still pointing to the GitHub repository that you cloned the code from. Change the remote repository to your GitHub repository.
  • Verify that your credentials file is not included with your function code before the commit the changes to your external directory.

Push your function code to your external repository in GitHub.

  1. Open a terminal window in the project folder.
  2. Enter: git remote -v. The remote repository is still pointing to https://github.com/ibm-watson-iot/functions
  3. Change the remote origin URL by using the following commands: 
    git remote set-url origin URL_To_YOUR_GITHUB_REPOSITORY
    Confirm that the Fetch and Push URLs are pointing to your repository.
  4. Add the upstream remote by using the following command. Enter: 
    git remote add upstream https://github.com/ibm-watson-iot/functions.git
  5. Add your files to GIT and commit them.
    1. In PyCharm, select the custom<your_initials> directory.
    2. Select Git > Add.
    3. Select Git > Commit.
    4. In the Commit Changes window, click Commit.
    5. Select Git > Repository > Push.
    6. Click Push.
  6. Verify that your module was pushed to a starter_package branch in your repository.

Step 5: Install the custom function in your local environment

Install the custom function in your local environment. Update the variables in <> to match your environment.

pip3 install git+https://<XXXXXX>@github.com/<user_id><path_to_repository>@starter_package --upgrade

For example:

pip3 install git+https://<XXXXXX>@github.com/jones/starter@starter_package --upgrade

Replace <xxxxxx> with your personal access token.</xxxxxx>

Step 6: Test your custom function locally.

  1. Create a script called test_my_custom_function.py in the scripts folder. In the script, import Python libraries and packages: 
    import datetime as dt
import json
import pandas as pd
import numpy as np
from sqlalchemy import Column, Integer, String, Float, DateTime, Boolean, func
from iotfunctions.base import BaseTransformer
from iotfunctions.metadata import EntityType
from iotfunctions.db import Database
from iotfunctions import ui
  2. Connect to Maximo Monitor. In the test_my_custom_function.py script, add: 
    with open('credentials_as.json', encoding='utf-8') as F:
  credentials = json.loads(F.read())
db_schema = None
db = Database(credentials=credentials)
  3. Import and instantiate the function. In the test_my_custom_function.py script, add the following code. Update the variables in <> to match your environment. 
    from custom<yourinitials>.multiplybyfactor<yourinitials> import MultiplyByFactor<YourInitials>
fn = MultiplyByFactor<YourInitials>(
    input_items = ['speed', 'travel_time'],
    factor = '2',
    output_items = ['adjusted_speed', 'adjusted_travel_time']
              )
df = fn.execute_local_test(db=db, db_schema=db_schema, generate_days=1,to_csv=True)
print(df)
  4. Run the script from the command line. The date frame results are saved to a .csv file. Look for df_test_entity_for_multiplybyfactor<your_initials>.csv in the scripts directory. Enter: 
    python3 test_my_custom_function.py

Step 7: Register your custom function.

  1. Create a script called register_my_custom_function.py in the scripts folder. In the script, import Python libraries and packages: 
    import datetime as dt
import json
import pandas as pd
import numpy as np
from sqlalchemy import Column, Integer, String, Float, DateTime, Boolean, func
from iotfunctions.base import BaseTransformer
from iotfunctions.metadata import EntityType
from iotfunctions.db import Database
from iotfunctions import ui
  2. Connect to Maximo Monitor. In the register_my_custom_function.py script, add: 
    with open('credentials_as.json', encoding='utf-8') as F:
  credentials = json.loads(F.read())
db_schema = None
db = Database(credentials=credentials)
``
  3. In the register_my_custom_function.py script, to register the function, add the following code. Update the variables in <> to match your environment. 
    from custom<yourinitials>.multiplybyfactor<yourinitials> import MultiplyByFactor<YourInitials>

db.register_functions([MultiplyByFactor<YourInitials>])
  4. Run the script from the command line. For example: 
    python3 register_my_custom_function.py

Step 8: Create a sample device type.

  1. On the Setup page, on the Devices tab, click the plus (+) icon to add a new device type.
  2. Select the Robot sample type template.
  3. Assign a name to the device type.
  4. Click Create.
  5. Select the sample device type, click Set up device type, and then click Data.

Metrics take up to 5 minutes to generate.

Step 9: Apply the custom function to the device type.

  1. If you have not created a distance metric, add it. Follow the steps in Tutorial: Adding expressions
  2. From the Data tab, click Create metric.
  3. Select the MultiplyByFactor<YourInitials> function from the catalog.
  4. Set the scope and then click Next.
  5. In the factor field, enter 2.
  6. In the input_item field, select distance.
  7. Click Next.
  8. Rename the output to 'adjusted_distance'.
  9. Click Create.
  10. In the data item list, select adjusted_distance. Wait up to 5 minutes for Maximo Monitor to evaluate the custom function against the sample data.