Technical preview: Knowledge Transformer
Knowledge Transformer transforms unstructured data files into an AI-optimized taxonomy collection of structured markdown knowledge documents that improves the accuracy and efficiency of AI systems.
Knowledge Transformer works to intelligently analyze, categorize, and transform technical content into organized knowledge documents with automatic taxonomy generation. The aim is to capture, curate, and produce high-quality knowledge documents that enable AI agents to utilize specialized expertise in decision making.
watsonx Assistant for Z and Knowledge Transformer
Knowledge Transformer is available as a technical preview with watsonx Assistant for Z. As a technical preview feature, Knowledge Transformer provides early access to capabilities that enhance how watsonx Assistant for Z leverages your organization's specialized knowledge. While Knowledge Transformer is functional and supported, it might undergo changes based on user feedback and evolving requirements.
Knowledge Transformer can be accessed via the command-line tool zAssist, complementing the existing content ingestion features with document transformation capabilities. By using Knowledge Transformer with watsonx Assistant for Z, you can transform your enterprise content into AI-optimized knowledge that improves an agent's ability to provide better accurate responses based on your organization's specific context.
Key capabilities of Knowledge Transformer
- Support for multiple file formats as input
- Knowledge Transformer supports multiple file formats as input. This includes .vtt, .pdf, .docx, .mp4, .pptx, .md, and .txt.
- Intelligent taxonomy generation
- Knowledge Transformer analyzes the content and produces self-organizing hierarchical directory structures. The taxonomy is efficiently categorized, and is produced to allow for more efficient search and filtering by AI agents.
- Collision detection
- Knowledge Transformer identifies and merges similar content to prevent duplication.
- Schema validation
- Knowledge Transformer ensures that output documents conform to a predefined knowledge document schema. Throughout the transformation process, any resulting output markdown files maintain a similar structure.
Why Knowledge Transformer is useful
Large Language Models (LLMs) are trained on broad datasets that rarely include deep expertise about specialized enterprise platforms or the unique ways your organization has configured its systems. This knowledge gap limits the value and trustworthiness of AI solutions in enterprise contexts.
Your organization's specialized knowledge exists across multiple sources: email threads, meeting transcripts, presentations, videos, and the expertise of experienced engineers. This information might have accumulated over decades, remaining unstructured, scattered, and difficult to leverage effectively. Simply feeding raw, unstructured content into LLMs produces poor results because the information is too noisy and inconsistent.
Knowledge Transformer transforms your raw enterprise content into optimized, AI-ready knowledge. The solution processes diverse content types, then organizes and refines this information to dramatically improve its usefulness for AI agents. This transformation increases knowledge density and removes noise, enabling LLMs to retrieve and digest information much more effectively.
AI agents, now empowered by Knowledge Transformer, can deliver higher quality answers based on your organization's specific expertise, better guidance for automated tasks, more effective planning support, and increased trust through responses grounded in your verified knowledge.
How Knowledge Transformer works
- The system ingests diverse content types including documents, presentations, transcripts, and videos.
- Content is analyzed and organized into focused knowledge documents, each addressing specific topics.
- Information is refined to remove noise, eliminate duplicates, and increase knowledge density.
- The optimized knowledge is structured for efficient retrieval by LLMs.
As you add more content, the system continuously refines and organizes your knowledge base, ensuring your AI solutions have access to accurate, relevant information when they need it. For more information, see How it works: Knowledge Transformer.
Getting started with Knowledge Transformer
Knowledge Transformer can be accessed via zassist in the CLI. To use Knowledge Transformer, you will need to install zassist, see Installing zassist. You will also need to configure authorization and authentication for the tool, and select your LLM. For more information on security and configuration, see Configuring Knowledge Transformer.
Once you have configured the zassist transform command, you can start by generating a taxonomy.