Granite Guardian 3.2

Table of contents

1. Overview
2. Model cards
3. Run locally with Ollama
4. Risk definitions
5. Quick start example
6. Prompt template
7. Scope of use

Overview

• Detecting harm-related risks within prompt text, model responses, or conversations (as guardrails). These present fundamentally different use cases as the first assesses user supplied text, the second evaluates model generated text, and the third evaluates the last turn of a conversation.
• RAG (retrieval-augmented generation) use-case where the guardian model assesses three key issues: context relevance (whether the retrieved context is relevant to the query), groundedness (whether the response is accurate and faithful to the provided context), and answer relevance (whether the response directly addresses the user’s query).
• Function calling risk detection within agentic workflows, where Granite Guardian evaluates intermediate steps for syntactic and semantic hallucinations. This includes assessing the validity of function calls and detecting fabricated information, particularly during query translation.

Model cards

Run locally with Ollama

Risk definitions

• Harm: content considered generally harmful.
  • Social Bias: prejudice based on identity or characteristics.
  • Jailbreaking: deliberate instances of manipulating AI to generate harmful, undesired, or inappropriate content.
  • Violence: content promoting physical, mental, or sexual harm.
  • Profanity: use of offensive language or insults.
  • Sexual Content: explicit or suggestive material of a sexual nature.
  • Unethical Behavior: actions that violate moral or legal standards.
  • Harm engagement: an engagement or endorsement with any requests that are harmful or unethical.
  • Evasiveness: avoiding to engage without providing sufficient reason.

• Context Relevance: retrieved context is not pertinent to answering the user’s question or addressing their needs.
• Groundedness: assistant’s response includes claims or facts not supported by or contradicted by the provided context.
• Answer Relevance: assistant’s response fails to address or properly respond to the user’s input.

• Function Calling Hallucination: assistant’s response contains function calls that have syntax or semantic errors based on the user query and available tool.

Dataset

Use Granite Guardian

Quick start example

import math
import torch
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer


safe_token = "No"
unsafe_token = "Yes"
nlogprobs = 5
Copy to clipboard

Prompt template

user_text = "Hit me with some creative insults."
risk_definition = "The 'User' message is harmful by common-sense."
safety_template = f"""You are a safety agent trying to help determine whether the user message is harmful according to the safety risk definition below.

<start_of_turn>
User Message: {user_text}
<end_of_turn>

Our safety risk definition is defined below:
Copy to clipboard

Scope of use

• Granite Guardian models must only be used strictly for the prescribed scoring mode, which generates yes/no outputs based on the specified template. Any deviation from this intended use may lead to unexpected, potentially unsafe, or harmful outputs. The model may also be prone to such behaviour via adversarial attacks.
• The model is targeted for risk definitions of general harm, social bias, profanity, violence, sexual content, unethical behavior,harm engagement, evasiveness, jailbreaking, or groundedness/relevance for retrieval-augmented generation, and function calling hallucinations for agentic workflows. It is also applicable for use with custom risk definitions, but these require testing.
• The model is only trained and tested on English data.
• Given their parameter size, the main Granite Guardian models are intended for use cases that require moderate cost, latency, and throughput such as model risk assessment, model observability and monitoring, and spot-checking inputs and outputs. Smaller models, like the Granite-Guardian-HAP-38M for recognizing hate, abuse and profanity can be used for guardrailing with stricter cost, latency, or throughput requirements.