SSLSRPV_2.4.x - Documentation Index

Table of Contents

• Welcome
• Overview of IBM watsonx
  • What's new
    • Analytics Engine powered by Apache Spark
    • Common core services
    • Data Refinery
    • Decision Optimization
    • Execution Engine for Apache Hadoop
    • Orchestration Pipelines
    • RStudio Server Runtimes
    • SPSS Modeler
    • Synthetic Data Generator
    • Watson Machine Learning
    • Watson Studio
    • Watson Studio Runtimes
    • watsonx.ai
    • watsonx.governance
  • watsonx use cases
    • watsonx.ai use case
    • watsonx.governance use case
  • Platform architecture
    • Asset types and properties
      • Previews
      • Profiles
      • Activities
      • Visualizations
  • Security of foundation models
  • Known issues
• Planning an AI solution
  • Initial strategic decisions
  • Comparison of tools and code
  • Planning the workflow for gen AI
  • Planning the workflow for ML
  • Solution pattern for answering questions with RAG
• Installing and administering watsonx
  • Setting up the model gateway
    • Setting up the model gateway in the UI
    • Setting up the model gateway programmatically
  • Managing the model gateway
  • Managing governance activities across multiple clusters
    • Manage your models on multiple clusters with a direct connection
    • Manage your models on multiple clusters with a custom governance connector
• Getting started and tutorials
  • Roadmaps for getting started
  • Setting up your console and first steps
    • Setting your preferences
    • Finding services
    • Checking your permissions
  • Switching between experiences
  • Generating API keys
    • Generating ZenApiKey authorization tokens
    • Generating a bearer token
    • Generating an authorization token for the admin user
  • Available APIs
  • Quick start tutorials
    • Prepare data
      • Refine data
      • Generate synthetic tabular data
    • Analyze data
      • Analyze data in a Jupyter notebook
    • Build machine learning models
      • Build a model with AutoAI
      • Build a model in a Jupyter notebook
      • Build a model with SPSS Modeler
      • Build a Decision Optimization model
    • Work with generative AI
      • Prompt a foundation model with Prompt Lab
      • Prompt a foundation model by using RAG
      • Tune a foundation model
      • Try the watsonx.ai end-to-end use case
    • Govern AI
      • Evaluate and track a prompt template
      • Evaluate a deployment in spaces
      • Evaluate a machine learning model
  • AI solution accelerators
    • Q&A RAG accelerator
    • Building ML models in watsonx.ai and batch scoring with Spark in watsonx.data accelerator
  • Video library
• Gen AI solutions
  • Terms of use
  • Tokens
  • Supported foundation models
    • IBM foundation models
      • Prompting IBM Granite Code models
    • Third-party foundation models
    • Supported encoder models
      • IBM Granite embedding 107m multilingual model card
      • IBM Granite embedding 278m multilingual model card
      • IBM Granite embedding English reranker r2 model card
      • IBM Slate 30m v2 embedding model card
      • IBM Slate 125m v2 embedding model card
    • Model gateway
    • Choosing a model
      • Foundation model deployment methods
      • Foundation model benchmarks
      • Foundation model lifecycle
    • Get model information with code
      • Getting foundation model information (Python)
  • Building prompts
    • Prompt tips
      • Avoiding undesirable output
      • Generating accurate output
    • Model parameters for prompting
    • Filtering model content with AI guardrails
    • Sample prompts
    • Prompt Lab
      • Adding prompts
      • Saving prompts
      • Building reusable prompts
      • Chatting with documents and media files
    • Generating text with code
      • Text generation
        • Inferencing a foundation model (Python)
        • Inferencing with a prompt template (Python)
        • Inferencing gateway models
      • Chat
      • Agent-driven chat
  • Coding generative AI solutions
    • Developer resources
      • Credentials for programmatic access
      • Finding the project ID
      • Python library
      • Node.js SDK
      • REST API
    • Adding files to reference from the API
  • Working with the lightweight engine
  • Forecast future values
  • Retrieval-augmented generation
    • Text embeddings
    • Rerank passages
    • Understanding documents
      • Text processing parameters
        • Creating custom schemas
        • Invoice schema
        • Utility bill schema
        • Mortgage lending document schema
        • Bill of lading schema
        • Customs form schema
        • Delivery receipt schema
        • Expense report schema
        • Receipt schema
        • Purchase order schema
        • Tax form schema
        • Financial statement schema
        • Remittance or Payment Advice schema
        • Bank statement schema
        • Credit card statement schema
        • Driver's license statement schema
        • Passport schema
        • National ID card schema
        • W-4 form schema
        • I-9 form schema
        • Patient intake form schema
        • Insurance claim schema
        • Transcript schema
        • Diploma or certification schema
        • Life insurance standard disability claim form schema
        • Standard life insurance authorization form schema
        • ACORD standardized insurance form schema
        • Claimant's statement schema
        • Business license and permit schema
      • Text classification
        • Text classification parameters
        • Classifying documents
      • Generating schemas for text extraction
      • Text extraction
        • Text extraction parameters
        • Extracting text from documents
      • Adding extracted text to RAG solutions
        • Parsing extracted JSON output
    • Optimizing your RAG knowledge base
    • Automating a RAG pattern with AutoAI
      • Choosing a vector store for a RAG experiment
      • Creating a RAG experiment (fastpath)
      • Using knowledge bases in RAG experiments
        • Using vector knowledge bases in AutoAI RAG experiments
        • Using SQL knowledge bases in AutoAI RAG experiments
      • Customizing RAG experiment settings
      • Reviewing RAG pattern details
      • Saving a RAG pattern
      • Coding an AutoAI RAG experiment in Python
      • Coding an AutoAI RAG experiment with Chroma
      • Coding an AutoAI RAG experiment with text extraction
      • Coding an AutoAI RAG experiment with a custom foundation model
      • Using AutoAI RAG index to chat with documents
  • Tuning models
    • When to tune a model
    • Choosing a model to tune
    • Tuning methods
      • Full fine tuning
      • LoRA fine tuning
      • QLoRA fine tuning
    • Data formats for tuning
    • Tuning experiment parameters
    • Tuning Studio
    • Model tuning with code
    • Evaluating the tuning experiment
• Projects
  • Shared projects across experiences
  • Creating a project
    • Importing a project
    • Importing project assets
    • Accessing a Git repository
    • Creating access token for collaboration
    • Configuring custom encryption for access tokens
  • Integrating with Git repositories
    • Default Git integration
      • Git operations
      • Project terminal
      • Connecting to data sources
    • Deprecated Git integration
      • Syncing projects
      • Syncing when working remotely
  • Administering projects
    • Managing collaborators
    • Project collaborator roles
    • Marking a project as sensitive
    • Logging project activity
    • Exporting project assets
    • Enabling folders
  • Managing assets in projects
    • Organizing assets with folders
  • Environments
    • Notebook environments
    • Spark environments
    • Data Refinery environments
    • SPSS Modeler environments
    • RStudio environments
    • JupyterLab environments
    • GPU environments
    • Hadoop environments
    • Creating non-standard environment templates
    • Customizing environments
      • Customizing environment templates
        • Customizing environment templates with pip
        • Examples of customizations
    • Managing environments
    • Viewing active runtimes across projects
  • Creating and managing jobs
    • Creating jobs in the Notebook editor
    • Creating jobs in Data Refinery
    • Creating jobs in SPSS Modeler
    • Creating jobs for code files
    • Creating jobs for Pipelines
    • Viewing jobs across projects
  • Leaving a project
  • Markdown cheatsheet
• Deployment spaces
  • Creating deployment spaces
  • Deployment space collaborator roles
  • Deleting deployment spaces
  • Assets in deployment spaces
    • Promoting assets to a deployment space
    • Importing spaces and projects
    • Exporting space assets
    • Adding data assets to a deployment space
    • Importing models to a deployment space
    • Adding code packages to a deployment space
    • Accessing asset details
    • Managing asset versions
• Preparing data
  • Connectors
    • Amazon Aurora for MySQL
    • Amazon Aurora for PostgreSQL
    • Amazon DynamoDB
    • Amazon RDS for MySQL connection
    • Amazon RDS for Oracle connection
    • Amazon RDS for PostgreSQL connection
    • Amazon Redshift connection
    • Amazon S3 connection
      • Setting up temporary credentials or a Role ARN for Amazon S3
    • Apache Cassandra connection
    • Apache Derby connection
    • Apache HDFS connection
    • Apache Hive connection
    • Apache Hive Metastore connection
    • Apache Impala connection
    • Apache Spark SQL connection
    • AWS Databricks
    • Box connection
    • ClickHouse connection
    • Collibra connection
    • Confluence connection
    • DataStax Enterprise connection
    • DataStax HCD connection
    • Denodo connection
    • Dremio connection
    • Dropbox connection
    • Elasticsearch connection
    • Elastic Cloud connection
    • Exasol connection
    • FTP connection
    • Generic JDBC connection
    • Generic S3 connection
    • Google BigQuery connection
      • Workload identity federation examples
    • Google Cloud Storage connection
    • Google Drive connection
    • Google Looker connection
    • Greenplum connection
    • HDFS via Execution Engine for Hadoop connection
    • Hive via Execution Engine for Hadoop connection
    • HTTP connection
    • IBM Cloud Data Engine connection
    • IBM Cloud Databases for MongoDB connection
    • IBM Cloud Databases for MySQL
    • IBM Cloud Databases for PostgreSQL connection
    • IBM Cloud Object Storage connection
    • IBM Cloud Object Storage (infrastructure) connection
    • IBM Cloudant connection
    • IBM Cognos Analytics connection
    • IBM Data Virtualization Manager for z/OS connection
    • IBM Db2 connection
      • Setting up a Db2 connection that uses TLS and SSL
    • IBM Db2 Big SQL connection
    • IBM Db2 for i connection
    • IBM Db2 for z/OS connection
    • IBM Db2 on Cloud connection
    • IBM Db2 Warehouse connection
    • IBM FileNet P8 connection
    • IBM Informix connection
    • IBM Master Data Management connection
    • IBM Netezza Performance Server connection
    • IBM Netezza Performance Server for DataStage connection
    • IBM Planning Analytics connection
    • IBM Product Master connection
    • IBM SPSS Analytic Server connection
    • IBM watsonx.data Milvus connection
    • IBM watsonx.data Presto connection
    • IBM watsonx.data SharePoint connection
    • Iceberg metastore connection
    • Impala via Execution Engine for Hadoop connection
    • Informatica PowerCenter connection
    • MariaDB connection
    • Microsoft Azure Blob Storage connection
    • Microsoft Azure Cosmos DB connection
    • Microsoft Azure Data Lake Storage connection
    • Microsoft Azure Databricks connection
    • Microsoft Azure Fabric Warehouse connection
    • Microsoft Azure File Storage connection
    • Microsoft Azure PostgreSQL connection
    • Microsoft Azure SQL Database connection
    • Microsoft Azure Synapse Analytics connection
    • Microsoft Exchange connection
    • Microsoft OneDrive connection
    • Microsoft SharePoint Files connection
    • Microsoft SharePoint connection
    • Microsoft SQL Server connection
    • Milvus connection
    • MinIO connection
    • MongoDB connection
    • MySQL connection
    • OData connection
    • OpenSearch connection
    • OpenSearch IBM Cloud connection
    • Oracle connection
    • PostgreSQL connection
    • Presto connection
    • Salesforce.com connection
    • SAP ASE connection
    • SAP BW Extract connection
      • Creating a connection for SAP BW Extract with SNC encryption
    • SAP HANA connection
    • SAP IQ connection
    • SAP OData connection
    • SingleStoreDB connection
    • Slack connection
    • Snowflake connection
    • Storage volume connection
    • Tableau connection
    • Teradata connection
    • Trino connection
    • Vertica connection
  • Adding data to a project
    • Adding connections to projects
      • Using secrets from vaults in connections
    • Adding integrated service connectors
    • Adding data from a connection
    • Adding a connected folder asset from a connection
    • Accessing data sources with Flight service
      • Flight data requests
      • Supported connectors in Flight service
      • Flight service in Python notebooks
        • Examples of Flight service data requests with Python
        • Connecting to Flight service manually in Python
        • Using itc_utils with your own code
        • Examples of Flight service with itc_utils and pandas
        • Resolving timeouts in Flight service
        • Using Flight with Apache Spark
      • Flight service in R notebooks
        • Examples of Flight service data requests with R
        • Connecting to Flight service manually in R
      • Best practices for loading large volumes of data from a file or connection
    • Downloading data assets
    • Adding catalog assets to a project
    • Publishing assets to a catalog
  • Adding platform connections
    • Managing collaborators on platform connections
  • Building custom connectors with Connector forge
  • Parametrized connections
  • Refining data
    • Adding data to Data Refinery
    • Validating your data
    • Visualizing your data
    • Managing Data Refinery flows
    • Target connection options
    • Refining data on the Hadoop cluster
      • Refining HDFS data
      • Refining data stored in tables in a Hive warehouse
      • Refining data stored in tables in Impala
    • GUI operations
    • Interactive code templates
    • Supported data sources for Data Refinery
  • Generating synthetic data
    • Generating synthetic structured data
      • Generating structured synthetic data from sample data
        • Using differential privacy
        • Evaluating synthetic data
        • Exporting synthetic data
        • Viewing synthetic data
      • Generating synthetic data from a custom data schema
      • Data sources for Synthetic Data Generator
      • Referential integrity and multi-table nodes
      • Nodes in Synthetic Data Generator
        • Anonymize node
        • Evaluate node
        • Export node
        • Generate node
        • Import node
        • Mimic node
        • Multi-export node
        • Multi-gen node
        • Multi-import node
        • Multi-mimic node
      • Scripting flows in Synthetic Data Generator
        • Running scripts
        • Properties for nodes and flows
        • Available methods for scripting
        • Parameters in Synthetic Data Generator
    • Generating synthetic unstructured data
      • Data builders and seed data
        • Tool calling data pipeline
        • Text to SQL data pipeline
        • Knowledge data pipeline
      • Creating jobs to generate unstructured synthetic data
      • Creating jobs programmatically to generate unstructured synthetic data
      • Configuration for creating jobs
  • Adding vectorized documents
    • Vector index settings
      • Setting up an Elasticsearch vector store
      • Setting up a Milvus vector store
    • Creating a vector index from the UI
    • Creating a vector index with code
  • Managing feature groups
• Data science solutions
  • Notebooks and scripts
    • Planning your notebooks and scripts experience
    • Jupyter Notebook editor
      • Creating and managing notebooks
      • Parts of a notebook
      • Jupyter kernels and notebook environments
      • Coding and running notebooks
      • Markdown cheatsheet
    • JupyterLab
      • JupyterLab with default Git integration
        • Organizing your code files
      • JupyterLab with deprecated Git integration
    • Visual Studio Code
    • RStudio
      • Working in RStudio with default Git integration
      • Working in RStudio with deprecated Git integration
      • Working in RStudio with no Git integration
      • Using Spark in RStudio
      • Using libs from Anaconda Repository
      • Accessing data in MySQL databases by using the RMariaDB library
      • Adding custom R packages
    • Libraries and scripts
      • Installing custom libraries
      • Importing scripts into a notebook
    • Watson Natural Language Processing
      • Working with pre-trained models
      • Library task catalog
        • Language detection
        • Syntax analysis
        • Noun phrase extraction
        • Keyword extraction and ranking
        • Entity extraction
        • Embeddings
        • HAP detection
        • Sentiment extraction
        • Tone classification
        • Emotion classification
        • Relations extraction
        • Hierarchical categorization
          • Category types
      • Creating your own models
        • Detecting entities with a custom dictionary
        • Detecting entities with regular expressions
        • Detecting entities with a custom transformer model
        • Classifying text with a custom classification model
        • Extracting sentiment with a custom transformer model
        • Extracting targets sentiment with a custom transformer model
      • Usage samples
    • Geospatial data analysis
    • Data skipping for Spark SQL
    • Parquet encryption
      • Key management by application
      • Key management by KMS
    • Time series analysis
      • Using the time series library
      • Time series key functionality
      • Time series functions
      • Time series lazy evaluation
      • Time reference system
    • SPSS predictive analytics algorithms
      • Data preparation
      • Classification and regression
      • Clustering
      • Forecasting
      • Survival analysis
      • Score
    • Loading and accessing data in a notebook
      • Loading data through generated code snippets
      • Adding data from a Planning Analytics connection
      • Accessing project assets with ibm-watson-studio-lib
        • ibm-watson-studio-lib for Python
        • ibm-watson-studio-lib for R
        • Migrating from project-lib for Python to ibm-watson-studio-lib
        • Migrating from project-lib for R to ibm-watson-studio-lib
      • Accessing project assets with project-lib
        • project-lib for Python (deprecated)
        • project-lib for R (deprecated)
      • Managing feature groups with assetframe-lib for Python
    • Managing the notebooks and scripts lifecycle
      • Automating the lifecycle of notebooks and scripts
      • Sharing notebooks
        • Hiding code in a notebook
  • AutoAI for machine learning
    • AutoAI glossary
    • Automating a machine learning experiment
      • Configuring experiment settings
      • Configure a text analysis experiment
    • Automating a time series forecast experiment
      • Tutorial: AutoAI univariate time series experiment
      • Tutorial: AutoAI supporting features time series experiment
      • Time series experiment implementation details
      • Scoring a time series model
    • Saving an AutoAI generated notebook
    • Running an AutoAI notebook on IBM Z and IBM LinuxONE
    • Selecting an AutoAI model
    • AutoAI library
    • AutoAI implementation details
    • Configuring data imputation
      • Imputation details for time series experiments
    • Using incremental learning to train pipelines
      • Incremental learning details
    • Evaluating AutoAI experiments for fairness
    • Troubleshooting AutoAI experiments
  • Deep learning
    • Deep learning experiment tutorial using MNIST data set
    • Deep learning experiment tutorial using CIFAR data set
  • Decision Optimization
    • Ways to use Decision Optimization
    • Supported data sources in Decision Optimization
    • Sample models and notebooks
    • Decision Optimization notebooks
    • Decision Optimization experiments
      • Creating a Decision Optimization experiment
      • Scenarios
        • Opening the scenario pane
        • Creating a scenario from scenario pane
        • Creating a scenario from Overview
        • Importing a scenario
        • Exporting a scenario
        • Viewing all scenarios
        • Generating a notebook from a scenario
        • Comparing scenarios
        • Comparing scenario tables
        • Comparing scenario models and log files
        • Saving scenarios for deployment
      • Configuring a Decision Optimization experiment
        • Changing the deployment space for your experiment
        • Changing default environments and adding Python libraries
      • Preparing input data
        • Importing data into a scenario
        • Editing input data
        • Table search and filtering
        • Exporting data to a project
      • Building a model
        • Code snippets for building models
        • Referencing imported data and defining output
        • Multiple model files
        • Importing a model
        • Configuring engine settings in a model
      • Solving a model
        • Intermediate solutions
        • Configuring the run parameters for a scenario
        • Selecting a different run environment for a scenario
      • Exploring the solution
      • Visualization view
        • Visualization widgets
        • Displaying multiple scenarios
        • Visualization widget syntax
          • Note widgets
          • Table widgets
          • Chart widgets
          • Vega Chart widgets
          • Gantt widgets
          • Pivot table widgets
      • Python DOcplex models
        • Input and output data
        • Solving and analyzing a model: the diet problem
          • Changing the scenario data
        • Working with multiple scenarios
          • Generating multiple scenarios
        • Engine settings
      • Modeling Assistant models
        • Selecting a Decision domain in the Modeling Assistant
        • Formulating and running a model: house construction scheduling
        • Adding multi-concept constraints and custom decisions: shift assignment
        • Creating advanced custom constraints with Python
      • OPL models
    • Decision Optimization Java models
  • SPSS Modeler
    • Getting started in SPSS Modeler
      • Mining data in SPSS Modeler
      • Mining text data with Text Analytics
        • About text mining
        • How extraction works
        • How categorization works
    • Tutorials
      • Introduction to modeling
      • Automate modeling for a flag target
      • Automate modeling for a continuous target
      • Automate data preparation
      • Explore graphs for drug treatment
      • Screen predictors
      • Reduce input data string length
      • Classify telecommunications customers
      • Predict telecommunications churn
      • Forecast bandwidth utilization
      • Forecast catalog sales
      • Make offers to customers (self-learning)
      • Predict retail sales promotions
      • Condition Monitoring
      • Analyze text for hotel satisfaction
    • Understanding and preparing data
      • Importing an SPSS Modeler stream
      • SQL optimization
        • How does SQL pushback work?
        • Tips for maximizing SQL pushback
        • Nodes supporting SQL pushback
        • CLEM expressions and operators supporting SQL pushback
        • Generating SQL from model nuggets
      • Missing data values
        • Handling missing values
          • Handling records with missing values
          • Handling fields with missing values
          • Handling records with system missing values
        • Functions available for missing values
      • Temporary storage
    • Supported data sources for SPSS Modeler
    • Building flows and models
      • Disabling or caching nodes in a flow
        • Disabling nodes in a flow
        • Caching options for nodes
      • Adding comments and annotations
      • Setting properties for flows
      • Creating and scheduling jobs
      • Chart builder
      • Flow scripting
        • Flow scripting example
        • Flow and SuperNode parameters
    • Nodes palette
      • Import
        • Data Asset node
        • Extension Import node
        • Sim Gen node
        • User Input node
      • Record Operations
        • Aggregate node
        • Append node
        • Balance node
        • CPLEX Optimization node
        • Distinct node
        • Extension Transform node
        • Merge node
        • RFM Aggregate node
        • Sample node
        • Select node
        • SMOTE node
        • Space-Time-Boxes node
        • Sort node
        • Streaming TCM node
        • Streaming Time Series node
      • Field Operations
        • Anonymize node
        • Auto Data Prep node
        • Binning node
        • Derive node
        • Ensemble node
        • Field Reorder node
        • Filler node
        • Filter node
        • History node
        • Partition node
        • Reclassify node
        • Restructure node
        • RFM Analysis node
        • Set to Flag node
        • Time Intervals node
        • Transpose node
        • Type node
          • Viewing and setting information about types
          • Measurement levels
            • Geospatial measurement sublevels
          • Converting continuous data
          • What is instantiation?
          • Data values
            • Setting options for values
            • Specifying values and labels for continuous data
            • Specifying values and labels for nominal and ordinal data
            • Specifying values for a flag
            • Specifying values for collection data
            • Specifying values for geospatial data
          • Defining missing values
          • Checking type values
          • Setting the field role
          • Setting field format options
      • Graphs
        • Charts node
        • Collection node
        • Distribution node
        • Evaluation node
        • Histogram node
        • Plot node
        • Multiplot node
        • Time Plot node
        • Web node
      • Modeling
        • Anomaly node
        • Apriori node
        • Association Rules node
        • Auto Classifier node
          • Continuous machine learning
        • Auto Numeric node
        • Auto Cluster node
        • Bayes Net node
        • C5.0 node
        • C&R Tree node
        • CARMA node
        • CHAID node
        • Cox node
        • Decision List node
        • Discriminant node
        • Extension Model node
          • Extension model nugget
        • Feature Selection node
        • Gaussian Mixture node
        • GenLin node
        • GLE node
        • GLMM node
        • HDBSCAN node
        • Isotonic-AS node
        • KDE node
        • K-Means node
        • K-Means-AS node
        • KNN node
        • Kohonen node
        • Linear node
        • Linear-AS node
        • Logistic node
        • LSVM node
        • Neural Net node
        • MultiLayerPerceptron-AS node
        • One-Class SVM node
        • PCA/Factor node
        • QUEST node
        • Random Trees node
        • Random Forest node
        • Regression node
        • Sequence node
        • SLRM node
        • SVM node
        • TCM node
        • Time Series node
        • Tree-AS node
        • TwoStep cluster node
        • TwoStep-AS cluster node
        • XGBoost-AS node
        • XGBoost Linear node
        • XGBoost Tree node
      • Text Analytics
        • Language Identifier node
        • Text Link Analysis node
          • Expert options
          • TLA node output
        • Text Mining node
          • Text Mining model nuggets
          • Text Analytics Workbench
            • Concepts tab
            • Text links tab
            • Categories tab
            • Resource editor tab
              • Rule lab
              • Creating TLA rules with the Rule lab
            • Setting options
              • Advanced linguistic settings
              • Advanced frequency settings
            • Generating a model nugget
        • Linguistic resources
          • Reusing custom linguistic resources
            • Reusing custom category sets
            • Reusing a library
            • Reusing a template
            • Reusing a text analysis package (TAP)
      • Outputs
        • Analysis node
        • Data Audit node
        • Extension Output node
        • KDE Simulation node
        • Matrix node
        • Means node
        • Report node
        • Set Globals node
        • Sim Fit node
        • Sim Eval node
        • Statistics node
        • Table node
        • Transform node
      • Export
        • Data Asset Export node
        • Extension Export node
      • Extension nodes
        • Native Python APIs
        • Python for Spark scripts
          • Scripting with Python for Spark
            • Analytic Server Context
            • Data metadata
            • Date, time, timestamp
            • Exceptions
            • Examples
        • R scripts
      • Decision tree nodes
        • The interactive tree builder
          • Growing and pruning the tree
          • Defining custom splits
          • Saving tree models and results
        • Building a tree model directly
        • Decision tree nodes
      • SuperNodes
    • Promoting SPSS Modeler flows and models
    • Scripting and automation
      • Scripting overview
        • Types of scripts
        • Flow scripts
          • Flow script example: Training a neural net
          • Jython code size limits
        • Running and interrupting scripts
      • The scripting language
        • Python and Jython
        • Python scripting
          • Operations
          • Lists
          • Strings
          • Remarks
          • Statement syntax
          • Identifiers
          • Blocks of code
          • Passing arguments to a script
          • Examples
          • Mathematical methods
          • Using non-ASCII characters
        • Object-oriented programming
          • Defining a class
          • Creating a class instance
          • Adding attributes to a class instance
          • Defining class attributes and methods
          • Hidden variables
          • Inheritance
      • Scripting in SPSS Modeler
        • Flows, SuperNode streams, and diagrams
          • Flows
          • SuperNode flows
          • Diagrams
        • Running a flow
        • The scripting context
        • Referencing existing nodes
          • Finding nodes
          • Setting properties
        • Creating nodes and modifying flows
          • Creating nodes
          • Linking and unlinking nodes
          • Importing, replacing, and deleting nodes
          • Traversing through nodes in a flow
        • Getting information about nodes
      • The scripting API
        • Example: Searching for nodes using a custom filter
        • Metadata: Information about data
        • Parameters
        • Global values
        • Error reporting for flows
      • Scripting tips
        • Looping through nodes
        • Accessing flow run results
          • Table content model
          • XML content model
          • JSON content model
          • Column Statistics content model and Pairwise Statistics content model
      • Properties reference overview
        • Syntax for properties
          • Structured properties
          • Abbreviations
        • Node and flow property examples
        • Node properties overview
          • Common node properties
      • Flow properties
      • Data Asset Import node properties
        • dataassetimport properties
        • extensionimportnode properties
        • simgennode properties
        • userinputnode properties
      • Record Operations node properties
        • appendnode properties
        • aggregatenode properties
        • balancenode properties
        • cplexoptnode properties
        • derive_stbnode properties
        • distinctnode properties
        • extensionprocessnode properties
        • mergenode properties
        • rfmaggregatenode properties
        • samplenode properties
        • selectnode properties
        • sortnode properties
        • streamingtimeseries properties
      • Field Operations node properties
        • anonymizenode properties
        • autodataprepnode properties
        • astimeintervalsnode properties
        • binningnode properties
        • derivenode properties
        • ensemblenode properties
        • fillernode properties
        • filternode properties
        • historynode properties
        • partitionnode properties
        • reclassifynode properties
        • reordernode properties
        • restructurenode properties
        • rfmanalysisnode properties
        • settoflagnode properties
        • transposenode properties
        • typenode properties
      • Modeling node properties
        • Common modeling node properties
        • anomalydetectionnode properties
        • apriorinode properties
        • associationrulesnode properties
        • autoclassifiernode properties
          • Setting algorithm properties
        • autoclusternode properties
        • autonumericnode properties
        • bayesnetnode properties
        • c50node properties
        • carmanode properties
        • cartnode properties
        • chaidnode properties
        • coxregnode properties
        • decisionlistnode properties
        • discriminantnode properties
        • extensionmodelnode properties
        • factornode properties
        • featureselectionnode properties
        • genlinnode properties
        • glmmnode properties
        • gle properties
        • kmeansnode properties
        • kmeansasnode properties
        • knnnode properties
        • kohonennode properties
        • linearnode properties
        • linearasnode properties
        • logregnode properties
        • lsvmnode properties
        • neuralnetworknode properties
        • questnode properties
        • randomtrees properties
        • regressionnode properties
        • sequencenode properties
        • slrmnode properties
        • svmnode properties
        • tcmnode properties
        • ts properties
        • treeas properties
        • twostepnode properties
        • twostepAS properties
      • Model nugget node properties
        • applyanomalydetectionnode properties
        • applyapriorinode properties
        • applyassociationrulesnode properties
        • applyautoclassifiernode properties
        • applyautoclusternode properties
        • applyautonumericnode properties
        • applybayesnetnode properties
        • applyc50node properties
        • applycarmanode properties
        • applycartnode properties
        • applychaidnode properties
        • applycoxregnode properties
        • applydecisionlistnode properties
        • applydiscriminantnode properties
        • applyextension properties
        • applyfactornode properties
        • applyfeatureselectionnode properties
        • applygeneralizedlinearnode properties
        • applyglmmnode properties
        • applygle properties
        • applygmm properties
        • applykmeansnode properties
        • applyknnnode properties
        • applykohonennode properties
        • applylinearnode properties
        • applylinearasnode properties
        • applylogregnode properties
        • applylsvmnode properties
        • applyneuralnetworknode properties
        • applyocsvmnode properties
        • applyquestnode properties
        • applyrandomtrees properties
        • applyregressionnode properties
        • applyselflearningnode properties
        • applysequencenode properties
        • applysvmnode properties
        • applytcmnode properties
        • applyts properties
        • applytreeas properties
        • applytwostepnode properties
        • applytwostepAS properties
        • applyxgboosttreenode properties
        • applyxgboostlinearnode properties
        • hdbscannugget properties
        • kdeapply properties
      • Graph node properties
        • collectionnode properties
        • distributionnode properties
        • dvcharts properties
        • evaluationnode properties
        • histogramnode properties
        • multiplotnode properties
        • plotnode properties
        • timeplotnode properties
        • webnode properties
      • Output node properties
        • analysisnode properties
        • dataauditnode properties
        • extensionoutputnode properties
        • kdeexport properties
        • matrixnode properties
        • meansnode properties
        • reportnode properties
        • setglobalsnode properties
        • simfitnode properties
        • statisticsnode properties
        • tablenode properties
        • transformnode properties
      • Export node properties
        • dataassetexport properties
        • extensionexportnode properties
      • Python node properties
        • gmm properties
        • hdbscannode properties
        • kdemodel properties
        • kdeexport properties
        • ocsvmnode properties
        • rfnode properties
        • smotenode properties
        • xgboostlinearnode properties
        • xgboosttreenode properties
      • Spark node properties
        • isotonicasnode properties
        • kmeansasnode properties
        • multilayerperceptronnode properties
        • xgboostasnode properties
      • SuperNode properties
    • Expression Builder
      • Selecting functions
    • Reference information
      • Tips and shortcuts
      • CLEM (legacy) language reference
        • Building CLEM (legacy) expressions
          • About CLEM
          • CLEM examples
          • Values and data types
          • Expressions and conditions
          • Working with strings
          • Handling blanks and missing values
          • Working with numbers
          • Working with times and dates
          • Summarizing multiple fields
          • Working with multiple-response data
          • The Expression Builder
            • Accessing the Expression Builder
            • Creating expressions
            • Selecting functions
              • Database functions
            • Selecting fields
            • Viewing or selecting values
            • Checking CLEM expressions
          • Find
        • CLEM datatypes
          • Integers
          • Reals
          • Characters
          • Strings
          • Lists
          • Fields
          • Dates
          • Time
        • CLEM operators
        • Functions reference
          • Conventions in function descriptions
          • Bitwise integer operations
          • Comparison functions
          • Conversion functions
          • Date and time functions
            • Converting date and time values
          • Functions handling blanks and null values
          • Global functions
          • Information functions
          • Logical functions
          • Numeric functions
          • Probability functions
          • Random functions
          • Sequence functions
          • SoundEx functions
          • Spatial functions
          • Special fields
          • String functions
          • Trigonometric functions
      • SPSS algorithms
  • Visualizing your data
    • Chart types
      • 3D charts
      • Bar charts
      • Box plots
      • Bubble charts
      • Candlestick charts
      • Circle packing charts
      • Custom charts
      • Dendrogram charts
      • Dual Y-axes charts
      • Error bar charts
      • Evaluation charts
      • Heat map charts
      • Histogram charts
      • Line charts
      • Map charts
      • Math curve charts
      • Multi-chart charts
      • Multiple series charts
      • Parallel charts
      • Pareto charts
      • Pie charts
      • Population pyramid charts
      • Q-Q plots
      • Radar charts
      • Relationship charts
      • Scatter plots and dot plots
      • Scatter matrix charts
      • Series array charts
      • Sunburst charts
      • t-SNE charts
      • Time plots
      • Theme River charts
      • Tree charts
      • Treemap charts
      • Word cloud charts
    • Global visualization preferences
  • Apache Hadoop
    • Machine learning models in Jupyter Python
    • Using Hadoop utilities
    • Using delegation token endpoints
  • Apache Spark
    • Getting started with Spark applications
      • Submitting Spark jobs
        • Spark jobs API syntax, parameters and return codes
        • spark.local.dir configuration parameter
        • Supported application languages and versions
      • Run Spark applications interactively
        • Adding custom configurations
        • Configuring Spark Environment Variables
      • Configuring Spark log level information
      • Persisting applications
    • Spark labs - Development environment
    • Enabling application autoscaling
    • Disaggregated Shuffle
    • Run Spark Streaming applications
    • Accessing data from applications
    • Monitoring Spark jobs through the Spark user interface
    • Accessing and customizing the Spark history server
    • Accessing Spark job driver logs
    • Using custom packages
      • Customizing Spark applications and notebooks using the analyticsengine CR configuration
      • Customizing using service volume instances
    • Using an external metastore for Spark SQL
      • Using IBM Cloud Data Engine
    • Spark libraries
      • Data skipping for Spark SQL
      • Time series analysis
        • Using the time series library
        • Time series key functionality
        • Time series functions
        • Time series lazy evaluation
        • Time reference system
      • Geospatial data analysis
        • Topological relations
        • Geohashing functions
        • Geospatial indexing
        • Ellipsoidal metrics
        • Routing functions
      • Parquet modular encryption
        • Key management by application
        • Key management by KMS
    • Spark service CLI
    • Configuring an Analytics Engine powered by Apache Spark instance for watsonx.data
      • Integrating with watsonx.data on Cloud Pak for Data
      • Integrating with watsonx.data on IBM Cloud
      • Integrating with watsonx.data stand-alone
      • Getting started with watsonx.data Spark use cases
• Deploying and managing AI assets
  • Deploying AI assets
    • Considerations for upgrading Watson Machine Learning and watsonx.ai
    • Deploying generative AI assets
      • Deploying a prompt template
        • Deploying prompt templates that uses custom foundation models
      • Deploying a detached prompt template
      • Deploying custom foundation models
        • Deploying custom foundation models on dedicated GPUs
          • Requirements for deploying custom foundation models on dedicated GPUs
          • Creating custom hardware specifications on dedicated GPUs
          • Creating a deployment for a custom foundation model on dedicated GPUs
        • Deploying a custom foundation model on MIG-enabled cluster
          • Requirements for deploying custom foundation models on MIG cluster
          • Creating custom hardware specifications on MIG cluster
          • Create custom foundation model asset on MIG cluster
          • Creating a deployment for custom foundation models on MIG cluster
        • Properties and parameters for deploying custom foundation models
        • Resource utilization guidelines for custom foundation models
        • Inferencing deployed custom foundation models
        • Managing custom foundation model deployment
      • Deploying tuned models
        • Requirements for deploying LoRA or QLoRA tuned models
        • Deploying tuned models with UI
        • Deploying tuned models with code
      • Deploying AI services
        • Deploying AI services with visual tools
        • Deploying AI services with code
          • Coding and deploying AI services manually
          • Coding and deploying AI services with templates
          • Coding and deploying AI services with ibm-watsonx-ai-cli
        • Testing AI service deployments
        • Managing AI service deployments
    • Deploying machine learning assets
      • Deploying machine learning assets programmatically
        • Python client example notebooks
      • Creating online deployments
      • Creating batch deployments
        • Compute requirements for batch deployment jobs
        • Data sources for scoring batch deployments
        • Batch deployment input details by framework
          • AutoAI models
          • Decision Optimization models
          • Python functions
          • Python scripts
          • Pytorch models
          • R scripts
          • Scikit-learn and XGBoost models
          • Spark and PMML models
          • SPSS models
          • Tensorflow models
          • Using multiple inputs for an SPSS job
      • Deploying Python functions
        • Writing deployable Python functions
      • Deploying Shiny apps
        • Deploying Shiny Apps from the GUI
        • Deploying Shiny Apps with curl and REST API
        • Deploying Shiny Apps with cpdctl
        • Connecting your Shiny application to a persistent storage volume
        • Example script to deploy a Shiny app
      • Deploying code packages
      • Deploying NLP models
      • Deploying scripts
      • Creating deployments with GPU hardware specifications
      • Creating a deployment job
      • Creating jobs in deployment spaces for SPSS Modeler flows
      • Managing deployment jobs
      • Creating deployment jobs to run code packages
      • Managing hardware specifications for deployments
      • Updating a deployment
      • Scaling a deployment
      • Evaluating deployments
      • Deleting a deployment
    • Deploying Decision Optimization models
      • Deploying a model by using the user interface
      • Deploying a model programmatically
        • Deploying a model
        • Model input and output data file formats
          • OPL model input and output data file formats
          • Python model input and output data file formats
        • Batch deployment and model execution
          • Input data for deployment jobs
          • Output data for deployment jobs
          • Run parameters
          • Running jobs
        • REST API example
      • Deploying Java models
      • Changing Python version in a deployed model with REST API
      • Python client examples
      • Delegating the CPLEX engine solve to Watson Machine Learning
    • Enabling model tracking with AI Factsheets
    • Frameworks and software specifications
      • Supported software specifications
      • Managing outdated software specifications or frameworks
    • Customizing deployment runtimes
      • Customizing runtimes with external libraries and packages
        • Requirements for using custom components with ML models
        • Customizing environment templates with pip
  • Managing AI Lifecycle with ModelOps
    • Deployments dashboard
    • Managing AI lifecycle with CPDCTL
      • Example: Creating a script asset with CPDCTL
  • Orchestrating tasks with Pipelines
    • Getting started with Pipelines
      • Planning a pipeline
    • Run the built-in sample pipeline
    • Creating a pipeline
      • Configuring pipeline nodes
      • Managing pipeline settings
      • Configuring global objects
      • Adding conditions to a pipeline
        • Functions used in pipelines Expression Builder
      • Handling pipeline errors
      • Programming a pipeline
        • Samples for programming a pipeline
      • Creating custom components
    • Running and saving pipelines
    • Configuration management for Orchestration Pipelines
• Governing AI
  • AI risk atlas
    • Toxic output
    • Unexplainable and untraceable actions
    • Data poisoning
    • Unreliable source attribution
    • Mitigation and maintenance
    • AI agent compliance
    • Function calling hallucination
    • Harmful output
    • Indirect instructions attack
    • Confidential information in data
    • Lack of model transparency
    • Exploit trust mismatch
    • Unrepresentative data
    • AI agents' Impact on human agency
    • AI agents' impact on human agency
    • Personal information in prompt
    • Sharing IP/PI/confidential information with user
    • Temporal gap
    • Nonconsensual use
    • Decision bias
    • Lack of testing diversity
    • Exposing personal information
    • Confidential data in prompt
    • Data privacy rights alignment
    • Discriminatory actions
    • IP information in prompt
    • Prompt leaking
    • Hallucination
    • Legal accountability
    • Social hacking attack
    • Non-disclosure
    • Lack of training data transparency
    • Model usage rights restrictions
    • Reproducibility
    • Specialized tokens attack
    • Prompt injection attack
    • Incomplete advice
    • Lack of system transparency
    • Data usage restrictions
    • Impact on cultural diversity
    • Impact on education: plagiarism
    • Personal information in data
    • Direct instructions attack
    • Improper usage
    • Exclusion
    • Extraction attack
    • Impact on Jobs
    • Jailbreaking
    • Data acquisition restrictions
    • Sharing IP/PI/confidential information with tools
    • Prompt priming
    • Reidentification
    • Overfitting
    • Attribute inference attack
    • Poor model accuracy
    • Data transfer restrictions
    • Generated content ownership and IP
    • Encoded interactions attack
    • Impact on human dignity
    • Output bias
    • Dangerous use
    • Lack of AI agent transparency
    • Unexplainable output
    • Human exploitation
    • AI agents' impact on jobs
    • Improper data curation
    • Over- or under-reliance on AI agents
    • Attack on AI agents’ external resources
    • Revealing confidential information
    • Lack of domain expertise
    • Spreading disinformation
    • Data bias
    • Uncertain data provenance
    • Unrepresentative risk testing
    • Redundant actions
    • Data usage rights restrictions
    • Unauthorized use
    • Misaligned actions
    • AI agents' impact on environment
    • Harmful code generation
    • Data contamination
    • Incomplete usage definition
    • Copyright infringement
    • Context overload attack
    • Lack of data transparency
    • Impact on affected communities
    • Improper retraining
    • Spreading toxicity
    • Inaccessible training data
    • Impact on education: bypassing learning
    • Introduce data bias
    • Accountability of AI agent actions
    • Incomplete AI agent evaluation
    • Untraceable attribution
    • Evasion attack
    • Impact on the environment
    • Over- or under-reliance
    • Incorrect risk testing
    • Membership inference attack
  • Governing assets with watsonx.governance
  • Planning for AI governance
  • Scenario for governing an AI RAG solution
  • Evaluating AI models
    • Setup options for model evaluations
      • Configuring model evaluations with automatic setup
      • Configuring model evaluations with manual setup
      • Configuring model evaluations with advanced setup
    • Terms for evaluations
    • FAQs
    • Supported machine learning engines, frameworks, and models
      • IBM Watson Machine Learning
      • Microsoft Azure ML Studio frameworks
      • Microsoft Azure ML Service frameworks
      • Amazon SageMaker frameworks
      • Custom ML frameworks
      • IBM SPSS C&DS frameworks
      • Integrating 3rd-party ML engines for model evaluations
    • Preparing to evaluate models
      • Adding deployments for evaluations
      • Providing model details
      • Configure asset deployments using JSON configuration files
    • Managing data for model evaluations
      • Managing payload data
        • Payload logging
      • Managing training data
      • Managing feedback data
      • Sending model transactions
    • Evaluation Studio
      • Evaluation Studio for Agentic AI applications
    • Evaluating prompt templates in projects
    • Evaluating prompt templates in deployment spaces
    • Evaluating detached prompt templates in projects
    • Evaluating detached prompt templates in spaces
    • Evaluating prompt templates for non-IBM foundation models with notebooks
    • Evaluating generative AI output in multiple languages
    • Evaluation metrics
      • Abstractness
      • Accuracy
      • Adversarial robustness
      • Answer relevance
      • Answer similarity
      • API latency
      • API throughput
      • Area under PR
      • Area under ROC
      • Average absolute odds difference
      • Average odds difference
      • Average precision
      • BLEU
      • Compression
      • Contains all
      • Contains any
      • Contains email
      • Contains_JSON
      • Contains link
      • Contains none
      • Contains string
      • Contains valid link
      • Context relevance
      • Coverage
      • Density
      • Disparate impact
      • Embedding drift
      • Ends with
      • Equals to
      • Error rate difference
      • Exact match
      • Faithfulness
      • False discovery rate difference
      • False negative rate difference
      • False omission rate difference
      • False positive rate difference
      • Feature drift
      • Fuzzy match
      • HAP metric
      • Hit rate
      • Impact score
      • Input metadata drift
      • Input token count
      • Is email
      • Is JSON
      • Keyword inclusion
      • Length greater than
      • Length less than
      • Macro F1 score
      • Macro precision
      • Macro recall
      • METEOR
      • Micro F1 score
      • Micro precision
      • Micro recall
      • Model quality drift
      • Natural robustness
      • No invalid links
      • Normalized discounted cumulative gain
      • Output drift
      • Output metadata drift
      • Output token count
      • Payload size
      • PII
      • Prediction drift
      • Prompt leakage risk
      • Question robustness
      • Readability
      • Reciprocal rank
      • Record latency
      • Record throughput
      • Records
      • Regex
      • Repetitiveness
      • Retrieval precision
      • ROUGE
      • SARI
      • Scoring requests
      • Sentence similarity
      • Starts with
      • Statistical parity difference
      • Text quality
      • Unsuccessful requests
      • Users
    • Configuring quality evaluations
    • Configuring fairness evaluations
      • Calculating fairness
      • Configuring for indirect bias
    • Configuring drift v2 evaluations
    • Configuring generative AI quality evaluations
    • Configuring model health monitor evaluations
    • Creating custom evaluations and metrics
    • Creating custom evaluations with LLM as a judge
    • Configuring explainability
    • Setting up alerts
    • Reviewing model insights
      • Reviewing evaluation results
      • Reviewing model transactions
        • Explaining model transactions
        • Reviewing debiased transactions
    • Batch processing
      • Preparing the batch processing environment in IBM Analytics Engine
      • Preparing the batch processing environment in IBM Analytics Engine on Linux on IBM Z and LinuxONE
      • Preparing the batch processing environment on the Hadoop Ecosystem
      • Configure batch processing
    • Model risk management and model governance
      • Manage model risk
    • Metrics computation for generative AI using the Python SDK
      • Agentic AI evaluation
      • Evaluate metrics
      • Model risk evaluation engine
      • Model insights
    • Metrics computation for ML models using the Python SDK
    • Python client samples for model evaluations
  • Governing assets in AI use cases
    • Setting up AI governance
      • Creating and managing inventories
      • Managing report templates
      • Managing attachments
    • Setting up an AI use case
      • Associating workspaces with an AI use case
    • Tracking assets in an AI use case
      • Tracking prompt templates
      • Tracking machine learning models
      • Tracking Python functions
      • Tracking external models
      • Tracking assets in a notebook
    • Viewing factsheets
    • Collaboration roles for governance
    • Adding custom facts to a factsheet
    • Coding custom facts
  • Managing risk and compliance with Governance console
    • Getting started
    • Configuring Governance console
    • Creating use cases and viewing models
    • Completing a risk assessment
    • Completing an applicability assessment
    • Solution components
      • Model Risk Governance solution
        • Model Risk Governance workflows
      • Operational Risk Management solution
        • Operational Risk Management workflows
      • Regulatory Compliance Management solution
        • Regulatory Compliance Management workflows
      • Object types
  • Using governed agentic catalog
    • Finding and using governed agents and tools
    • Managing governed agents and tools
  • Configuring guardrails
    • Setting up an inventory for guardrail policies
    • Creating guardrail policies
    • Available filters for guardrails
    • Creating custom detectors for guardrail policies
    • Programmatic configuration for real-time detection
  • Migrating assets to inventories in watsonx.governance
  • Migrating use cases in watsonx.governance
• Troubleshooting
  • Common core services
  • Connections
  • Analytics Engine powered by Apache Spark
  • Data Refinery
  • Hadoop environments
    • Troubleshooting with the Execution Engine for Apache Hadoop diagnostics tool
  • SPSS Modeler
  • Synthetic Data Generator
  • Watson OpenScale
  • Watson Machine Learning
  • Watson Studio
  • watsonx.governance
• Glossary