Generative AI systems present a number of unique security challenges. Alongside the typical challenge of securing access to generative AI models, organizations must balance the creative power of large language models (LLMs) and other generative technologies with the risk that the models will generate incorrect or undesirable outputs, disclose sensitive or private information, or execute undesirable or incorrect / disallowed / illegal actions.
The Open Web Application Security Project, OWASP, has published version 1 of the top 10 risks and vulnerabilities for LLMs and generative AI applications. The diagram below illustrates these vulnerabilities in the context of an agentic AI architecture.
The figure below augments the architecture to show the placement of security components to protect against / mitigate the vulnerabilities in the OWASP Top 10.
An Identity and Access Management (IAM) component is added to provide strong user identities and roles; mitigating the risk of model theft by controlling access to application functionality and APIs that could lead to theft or model disclosure.
Agent identify and access control (Agent Access Control), which functions similar to privileged user, is added to match agent access rights to user identities and roles; guarding against excessive agency and abnormal agent actions as a result of hallucinations, or poorly formed or ambiguous prompts.
Generative AI monitoring components (GenAI Monitoring) are added throughout the architecture to guard against prompt injection, insecure output handling, sensitive data disclosure, and overreliance. A combination of GenAI Monitoring and traditional Data Leakage Monitoring is deployed to guard against prompt-/reponse-based attacks, eg. a prompt injected into the results of a SQL query, as well as the disclosure of sensitive information that may appear in the results of API calls, database queries, and the like.
Training data poising attacks are mitigated by the addition of Configuration Management and monitoring tools, as well as a structured Version Control and release process around model training, fine-tuning, and configuration data.
Finally, an Integrated Behavior Monitoring and event correlation component is added to identify potential vulnerabilities and attacks from individual component logs. A Notification and Alerting component is added to notify system operators of potential issues, and a Response Orchestration component is added to automate and/or coordinate system and manual responses to identified issues.
IBM's Generative AI Architecture is the complete IBM Generative AI Architecture in IBM IT Architect Assistant (IIAA), an architecture development and management tool. Using IIAA, architects can elaborate and customize the architecture to create their own generative AI solutions.