Configuring advanced options
Advanced options provide fine-grained control over how Content Assistant processes queries and retrieves document content. These options are stored in the Gen AI Advanced Options field of the GenAI Configuration object in JSON format. By configuring these options, you can optimize query performance, improve response accuracy, and reduce processing costs for your specific use case.
Understanding advanced options
Advanced options allow you to customize the behavior of Content Assistant beyond the default settings. These options affect how the system processes queries, retrieves document content, and sends information to the large language model (LLM). The options you configure apply to all queries processed by the object store unless overridden by API callers.
When you configure advanced options, consider the following factors:
- The types of documents in your repository (size, complexity, structure)
- The typical queries your users submit (single document vs multiple documents)
- The balance between response accuracy and processing costs
- The performance requirements for your environment
Available advanced options
You can configure the following advanced options:
- Relevancy score threshold
- Filter out document chunks with low relevancy scores to improve query accuracy and reduce processing costs. By setting a minimum relevancy score threshold, you ensure that only the most relevant document chunks are sent to the LLM. This option is particularly useful when you have large document repositories and want to focus on the most pertinent content. The relevancy score is calculated by the embedding model and indicates how closely a document chunk matches the user's query.
- Default chunk numbers
- Control how many document chunks Content Assistant sends to the LLM for different types of queries. You can set separate defaults for single document queries and multiple document queries. Single document queries typically require fewer chunks because the context is limited to one document, while multiple document queries may benefit from more chunks to provide comprehensive answers across multiple sources. The default range is 1 to 32 chunks. Setting appropriate chunk limits helps balance response quality with processing efficiency and cost.
- Complete document content for vector queries
- Enable sending both document chunks and complete document text to the LLM for vector queries. This option, also known as adaptive RAG mode, provides the LLM with both targeted relevant chunks and full document context. This approach can improve answer quality by giving the model access to complete information while still highlighting the most relevant sections. This option is particularly useful for queries that require understanding the full context of documents or when document structure is important for accurate responses.
How advanced options work
When you configure advanced options, the values are stored in JSON format in the GenAI Configuration object for your object store. The system applies these options when processing queries:
- A user submits a query through Content Assistant.
- The system retrieves the advanced options from the GenAI Configuration object.
- The system applies the configured options to determine how to process the query (which chunks to retrieve, how many chunks to send, whether to include complete document content).
- The system sends the processed content to the LLM according to the configured options.
- The LLM generates a response based on the content it receives.
API callers can override these default settings by specifying their own values in API calls. This flexibility allows different applications or use cases to optimize query processing for their specific needs while maintaining consistent defaults for general use.
Best practices for configuring advanced options
Consider these best practices when configuring advanced options:
- Start with default values and adjust based on actual query performance and user feedback.
- Test different configurations with representative queries before deploying to production.
- Monitor query performance and accuracy after making changes to advanced options.
- Document your configuration decisions and the reasoning behind them for future reference.
- Review and adjust settings periodically as your document repository and query patterns evolve.
- Consider the trade-offs between response quality, processing time, and cost when setting chunk limits.
- Use relevancy score thresholds to filter out noise in large document repositories.
- Enable complete document content for queries where full context is critical for accurate responses.