Diving into Search
none
SensibleAI Agents: Search
Search is a Q&A AI Agent whose knowledge base can be dynamically expanded. Search utilizes Retrieval-Augmented Generation (RAG) to provide a seamless experience for OneStream users by offering a personalized assistant that can help with platform navigation, custom workflows, and daily tasks.
This one-pager will uncover how Search can rapidly and accurately answer user queries based on information from thousands of business-specific documents in various formats.
What is RAG?
Retrieval Augmented Generation (RAG) is an advanced AI technique that enhances the accuracy and relevance of AI-generated responses by dynamically retrieving information from a vast set of business-specific documents. Unlike traditional AI models that rely solely on pre-trained knowledge, RAG integrates a real-time retrieval process to pull in the most relevant data before generating answers. This ensures responses are always grounded in the available information, reducing the risk of inaccurate answers.
RAG operates through two key pipelines: the data ingestion pipeline and the retrieval pipeline. The data ingestion pipeline continuously processes and indexes business documents from various formats, ensuring that all relevant information is structured and accessible. When a user submits a query, the retrieval pipeline searches for the most relevant documents, extracts key information, and feeds it into the Large Language Model (LLM) to generate a well-informed response. This seamless integration of retrieval and generation allows Search to provide OneStream users with highly accurate, context-aware answers tailored to their specific query.
The following sections will dive into each of the pipelines introduced above, and the specific process flows for Search.
Data Ingestion Process Flow
loading...Given the diverse nature of business documentation, the data ingestion pipeline supports a range of file formats, including PDF, DOCX, PPTX, and TXT. Through intelligent extraction of information, it standardizes these varied data sources into a structured format. This transformation is achieved through four essential steps—Parsing, Chunking, Embedding, and Storing—which convert complex, unstructured documentation into clear, accessible information that is stored in a manner optimized for retrieval, enabling swift and accurate search results for users.
Parsing
Parsing refers to the extraction of content from documentation. It is the critical entry point to the data ingestion pipeline, and it requires a high degree of accuracy to ensure reliable processing. Effective parsing captures all relevant information, preventing data loss and enabling precise downstream retrieval. Without accurate content extraction, essential information would be missing from storage, compromising the retrieval process. To address this, the data ingestion pipeline employs intelligent parsing strategies to faithfully extract information from not only text but also images and tables within these complex file formats. The extracted content is then stored in a standardized structure, ensuring that subsequent processing steps can effectively optimize it for multimodal search retrieval.
Chunking
Chunking is the process of breaking down documents into smaller, more searchable segments or “chunks.” These smaller segments are structured to remain coherent and logical, with various strategies employed to ensure that each one retains its connection to the overall document and remains properly contextualized. As the first step after parsing, chunking plays a crucial role in optimizing extracted information for efficient retrieval. This method allows users to upload massive files, potentially thousands of pages long, and still pinpoint a single, buried sentence that might otherwise go unnoticed in a full document review. By structuring information into more digestible pieces, chunking dramatically reduces the time required to locate relevant details, delivering significant value by making knowledge more accessible and search results more precise.
Chunking Strategies
The SensibleAI Agents Media Processing Engine employs advanced, research-backed chunking strategies inspired by cutting-edge techniques from leading AI research laboratories. These strategies are designed to preserve document structure, context, and multimodal content—ensuring that every piece of information is both retrievable and semantically meaningful during search. Some of these techniques include:
- Hierarchical Chunking: This approach preserves relationships between different sections and granularities of a document by organizing chunks in a hierarchical, tree-like structure. Smaller chunks inherit context from their parent sections, allowing the retrieval pipeline to navigate a document at varying levels of depth. This structure enables the system to surface both high-level summaries and fine-grained details, depending on the user’s query.
- Contextual Chunking: Building on techniques pioneered by Anthropic and other AI research leaders, contextual chunking enhances each chunk with additional explanatory metadata and context before it is embedded. This prepended context acts as a mini “summary” or “scope note,” helping the embedding model better capture the semantic role of the chunk within the larger document. The result is improved retrieval accuracy and contextual relevance, especially when user queries are nuanced or refer indirectly to document content.
- Multimodal Chunking: In enterprise finance, valuable insights often reside in non-textual artifacts such as tables, figures, and charts. Multimodal chunking extends the chunking process to include these elements by generating specialized vector representations for images and tables alongside text-based chunks. This enables the retrieval system to assess the relevance of both textual and visual content, ensuring that key quantitative or visual insights—such as financial tables or variance charts—are discoverable through the same search interface.
Embedding
Embedding converts text into dense vector representations that capture semantic meaning, enabling efficient similarity-based retrieval. In the data ingestion pipeline, chunks are transformed into numerical vectors using OpenAI embedding models. This allows information to be stored in a format optimized for fast lookups, even when user queries lack exact keyword matches. By leveraging embeddings, the retrieval pipeline can compare user queries against stored representations to identify the most semantically relevant information, rather than relying solely on keyword matching or lexical similarity.
Document Storage
The storage layer manages all components of a document in a structure optimized for retrieval. Parsed text and metadata enable precise keyword search, while embedded vectors capture semantic meaning for similarity matching. Non-text artifacts such as tables and images are also stored, allowing the system to surface visual and tabular insights when relevant.
This combination of structured text, artifacts, and vector embeddings allows the retrieval engine to perform hybrid search, which will be discussed in the subsequent section.
Retrieval Process Flow
loading...Query Expansion
A user’s initial query may be too broad, ambiguous, or lacking context. To ensure the best possible retrieval of relevant information, Query Expansion enhances the original query by generating variations of it that can improve the relevance of the information that is pulled back during retrieval. Some of the additional queries that are generated are referred to as:
- Step-back queries – Reformulations that generalize the query, allowing the system to retrieve broader background context that may be needed to answer the original query.
- Decomposed queries – Breaking complex, multi-part questions into smaller, more precise sub-queries.
Once expanded, these queries are converted into vector representations using an embedding model. This intelligent expansion ensures that even vague or partial user inputs can retrieve the most relevant data.
**Example of Query Expansion** **Original Query:** * *How do I connect data into a BiViewer when building dashboards?* **Some Outputs of Query Expansion:** * *What is BiViewer?* * *How do I connect data into a BiViewer?* * *What types of data sources are compatible with BiViewer?*
Multi-hop Retrieval
Multi-hop retrieval enables the system to iteratively search for and connect relevant information across multiple sources, ensuring complex queries receive complete and well-supported answers. This process is powered by hybrid search, which combines semantic vector similarity with traditional keyword-based ranking to achieve optimal precision and recall.
Hybrid Search
Hybrid search in Azure AI Search fuses two complementary retrieval techniques—vector search and lexical search—to ensure the most relevant results are retrieved, even when user queries vary in structure or terminology.
- Vector Search: Once embedded, each document chunk and query is represented as a high-dimensional vector. During retrieval, Azure AI Search computes the cosine similarity between the query vector and stored vectors, surfacing results that are semantically related—even if they don’t share the same keywords.
- Lexical Search:
In parallel, Azure AI Search performs a traditional keyword-based search using the BM25 ranking function, a probabilistic model that measures the relevance of a document based on term frequency, inverse document frequency, and document length normalization. This ensures that documents explicitly matching key terms in the user’s query are not overlooked, preserving lexical precision. - Reciprocal Rank Fusion (RRF):
To combine these two result sets—semantic and lexical—Azure AI Search applies Reciprocal Rank Fusion, a state-of-the-art re-ranking algorithm. RRF assigns a fused score to each document based on its relative position in both the BM25 and vector ranking lists. This approach rewards documents that perform well across both retrieval modes, ensuring the final ranked list reflects both contextual understanding and keyword accuracy.
Retrieval Stages
The retrieval process occurs in distinct stages:
- Initial Retrieval: The system converts each query from the query expansion step into an embedding vector and retrieves the most relevant documents from the inverted index, based on both vector similarity and keyword matching.
- Retrieved Information Analysis: After the information is retrieved for the initial set of queries, the document chunks are analyzed. If a complete answer is found, no further retrieval is needed.
- Additional Query Generation: If the initial results lack sufficient detail, the system generates follow-up queries to retrieve missing information. These queries are informed by gaps in the “first-hop” results and relationships between retrieved documents.
- Additional Retrieval Hop(s): The system continues retrieving additional context, analyzing, and generating new queries until it determines that it has gathered enough relevant information to generate a well-rounded and accurate response, or until it reaches its maximum hop limit.
By leveraging hybrid search, iterative information analysis, and query rewriting, multi-hop retrieval ensures that even complex, multi-part queries are answered with precision and depth.
Response Generation
Once relevant information is retrieved, the response generation phase refines and structures the data before passing it to the LLM for synthesis. This ensures responses are accurate, well-structured, and backed by sources.
- Re-Ranking & Context Optimization: Not all retrieved documents are equally relevant. The system applies re-ranking algorithms to prioritize the most useful content based on semantic relevance, reliability, and alignment with the query. After this re-ranking, the content of the chunks is formatted in a specific manner with the name of the original source it came from.
- Grounded Response Generation: The refined and formatted context is then passed to the LLM, which generates a fact-based response strictly grounded in retrieved data, reducing the possibility of hallucination.
- Answer with Transparency and Explainability: To ensure transparency, the final response includes citations, linking key statements back to their original sources for easy verification. Intermediate reasoning steps are also displayed to the users for visibility into the steps Search took to arrive at its final generation. By re-ranking, structuring, and retrieval-grounding, this process delivers accurate, explainable, and verifiable answers.
FAQ
How is access to data provisioned in the inverted index?
All files are grouped in knowledge clusters. Users of the system are provisioned access to knowledge clusters by Sensible AI Agents Administrators. During retrieval, the system dynamically filters on the knowledge clusters a user is provisioned access to, ensuring that only information they are allowed to access is retrieved.
How is the possibility of model hallucination dealt with?
Some of the core features of Search are the transparency and explainability baked into the product. Every answer provided is cited with the exact sources that were used to generate the response, and the user is able to view the exact content from those files in the UI. Additionally, all of the intermediate reasoning steps such as the query expansion and retrieved information analysis is also rendered, so users can trace exactly how Search arrived at its answer.
Furthermore, the system internally uses several state-of-the-art hallucination mitigation techniques such as advanced prompt engineering and multi-model orchestration.